Diagnosis of Performance Faults in LargeScale MPI Applications via Probabilistic Progress-Dependence Inference
Issue No. 05 - May (2015 vol. 26)
Ignacio Laguna , Lawrence Livermore National Laboratory, Livermore, CA 94550
Dong H. Ahn , Lawrence Livermore National Laboratory, Livermore, CA 94550
Bronis R. de Supinski , Lawrence Livermore National Laboratory, Livermore, CA 94550
Saurabh Bagchi , Department of Electrical and Computer Engineering, Purdue University, 465 Northwestern Avenue, West Lafayette, IN 47907
Todd Gamblin , Lawrence Livermore National Laboratory, Livermore, CA 94550
Debugging large-scale parallel applications is challenging. Most existing techniques provide little information about failure root causes. Further, most debuggers significantly slow down program execution, and run sluggishly with massively parallel applications. This paper presents a novel technique that scalably infers the tasks in a parallel program on which a failure occurred, as well as the code in which it originated. Our technique combines scalable runtime analysis with static analysis to determine the least-progressed task(s) and to identify the code lines at which the failure arose. We present a novel algorithm that infers probabilistically progress dependence among MPI tasks using a globally constructed Markov model that represents tasks’ control-flow behavior. In comparison to previous work, our algorithm infers more precisely the least-progressed task. We combine this technique with static backward slicing analysis, further isolating the code responsible for the current state. A blind study demonstrates that our technique isolates the root cause of a concurrency bug in a molecular dynamics simulation, which only manifests itself at 7,996 tasks or more. We extensively evaluate fault coverage of our technique via fault injections in 10 HPC benchmarks and show that our analysis takes less than a few seconds on thousands of parallel tasks.
Handheld computers, Debugging, Probabilistic logic, Algorithm design and analysis, Markov processes, Benchmark testing, Computational modeling
I. Laguna, D. H. Ahn, B. R. Supinski, S. Bagchi and T. Gamblin, "Diagnosis of Performance Faults in LargeScale MPI Applications via Probabilistic Progress-Dependence Inference," in IEEE Transactions on Parallel & Distributed Systems, vol. 26, no. 5, pp. 1280-1289, 2015.