In this paper, we present an evolving system for the analysis and visualization of parallel application performance on shared memory multiprocessors. Our system couples SimOS, a complete machine simulator, with Rivet, a powerful visualization environment. This system demonstrates how visualization is necessary to realize the full power of simulation for performance analysis. We identify several features required of the visualization system, including flexibility, exploratory interaction techniques, and data aggregation schemes.We demonstrate the effectiveness of this parallel analysis and visualization system with a case study. We developed two visualizations within Rivet to study the Argus parallel rendering library, focusing on the memory system and process scheduling activity of Argus respectively. Using these visualizations, we uncovered several unexpected interactions between Argus and the underlying operating system. The results of the analysis led to changes that greatly improved its performance and scalability. Argus had previously been unable to scale beyond 26 processors; after analysis and modification, it achieved linear speedup up to 45 processors.