Recent hardware advances are creating multi-core systems with heterogeneous functionality. This paper explores how applications and middleware can utilize systems comprised of processors specialized for communication vs. computational tasks. The C-CORE execution environment enables applications, through middleware and underlying system functionality, to utilize both the computational capabilities of general purpose CPUs and the high performance communication hardware provided by specialized communication processors. Such future heterogeneous multi-core hardware is emulated by attaching a representative network processor - Intel?s IXP2400 processor - to a general purpose CPU via a dedicated interconnect. For this platform, C-CORE provides abstractions to represent an application?s communication actions, to efficiently couple such actions with application-level computations, and to dynamically create and configure the platform-resident ?chains? of computational and communication actions used by applications. C-CORE?s functionality is evaluated with representative, communication-intensive applications. Measurements on our experimental platform establish the performance advantages afforded to applications by C-CORE.