Ease of programming is one of the main requirements for the broad acceptance of multi-core systems without hardware support for transparent data transfer between local and global memories. Software cache is a robust approach to provide the user with a transparent view of the memory architecture; but this software approach can suffer from poor performance. In this paper, we propose a hierarchical, hybrid software-cache architecture that targets enabling prefetch techniques. Memory accesses are classified at compile time into two classes: high-locality and irregular. Our approach then steers the memory references toward one of two specific cache structures optimized for their respective access pattern. The specific cache structures are optimized to enable high-level compiler optimizations to aggressively unroll loops, reorder cache references, and/or transform surrounding loops so as to practically eliminate the software cache overhead in the innermost loop. The cache design enables automatic prefetch and modulo scheduling transformations. Performance evaluation indicates that optimized software cache structures combined with the proposed prefetch techniques translate into speed-up between 10% and 20%. As a result of the proposed technique, we can achieve similar performance on the Cell BE processor as on a modern server-class multi-core such as the IBM PowerPC 970MP processor for a set of parallel NAS applications.