In this paper we show the difficulties encountered when performing full system simulation of a distributed memory parallel system. To illustrate the problem, we have chosen a workbench that evaluates the impact on application performance of some simple congestion-control mechanism that can be implemented in the interconnection network. Applications of choice are some of those included in the NAS Parallel Benchmarks. Running a full-system, execution-driven simulation that combines Simics with an interconnection network simulator, we observe some unexpected, negative interactions of overlapping congestion control techniques implemented at the network level and at the host level.

Each MPI implementation uses a different protocol stack, and some of them work without TCP. We compare and contrast results obtained with MPICH, LA-MPI and LAM..