This Article 
 Bibliographic References 
 Add to: 
Fast Parallel Sorting Under LogP: Experience with the CM-5
August 1996 (vol. 7 no. 8)
pp. 791-805

Abstract—In this paper, we analyze four parallel sorting algorithms (bitonic, column, radix, and sample sort) with the LogP model. LogP characterizes the performance of modern parallel machines with a small set of parameters: the communication latency (L), overhead (o), bandwidth (g), and the number of processors (P). We develop implementations of these algorithms in Split-C, a parallel extension to C, and compare the performance predicted by LogP to actual performance on a CM-5 of 32 to 512 processors for a range of problem sizes. We evaluate the robustness of the algorithms by varying the distribution and ordering of the key values. We also briefly examine the sensitivity of the algorithms to the communication parameters.

We show that the LogP model is a valuable guide in the development of parallel algorithms and a good predictor of implementation performance. The model encourages the use of data layouts which minimize communication and balanced communication schedules which avoid contention. With an empirical model of local processor performance, LogP predictions closely match observed execution times on uniformly distributed keys across a broad range of problem and machine sizes. We find that communication performance is oblivious to the distribution of the key values, whereas the local processor performance is not; some communication phases are sensitive to the ordering of keys due to contention. Finally, our analysis shows that overhead is the most critical communication parameter in the sorting algorithms.

[1] S. Fortune and J. Wyllie, "Parallelism in Random Access Machines," Proc. 10th Ann Symp. Theory of Computing, pp. 114-118, 1978.
[2] D. Culler,R. Karp,D. Patterson,A. Sahay,K.E. Schauser,E. Santos,R. Subramonian,, and T. von Eicken,“LogP: Towards a realistic model of parallel computation,” Fourth Symp. Principles and Practices Parallel Programming, SIGPLAN’93, ACM, May 1993.
[3] D.E. Culler, A. Dusseau, S.C. Goldstein, A. Krishnamurthy, S. Lumetta, T. von Eicken, and K. Yelick, "Parallel Programming in Split-C," Supercomputing, 1993.
[4] G.E. Blelloch, C.E. Leiserson, B.M. Maggs, C.G. Plaxton, S. Smith, and M. Zagha, "A Comparison of Sorting Algorithms for the Connection Machine CM-2," Proc. Third Ann. ACM Symp. Parallel Algorithms and Architectures, pp. 3-16, July 1991.
[5] K. Batcher, "Sorting Networks and their Applications" Proc. AFIPS Spring Joint Computing Conf., 1986.
[6] T. Leighton, "Tight Bounds on the Complexity of Parallel Sorting," IEEE Trans. Computers, vol. 34, no. 4, pp. 344-354, Apr. 1985.
[7] M. Zagha and G. Blelloch, "Radix Sort for Vector Multiprocessors," Supercomputing, 1991.
[8] D. Lenoski et al., “The Stanford DASH Multiprocessor,” Computer, pp. 63-79, Mar. 1992.
[9] R.M. Karp, M. Luby, and F. Meyer auf der Heide, Efficient PRAM Simulation on a Distributed Memory Machine Proc. 24th ACM Symp. Theory of Computing, pp. 318-326, May 1992.
[10] K. Mehlhorn and U. Vishkin, "Randomized and Deterministic Simulations of PRAMs by Parallel Machines with Restricted Granularity of Parallel Memories," Acta Informatica, vol. 21, pp. 339-374, 1984.
[11] R. Cole and O. Zajicek, "The APRAM: Incorporating Asynchrony into the PRAM Model," Proc. Symp. Parallel Architectures and Algorithms, pp. 169-178, 1989.
[12] P. Kanellakis and A. Shvartsman, "Efficient Parallel Algorithms Can Be Made Robust," Proc. Eighth Symp. Principles of Distributed Computing, pp. 211-221, 1989.
[13] Z.M. Kedem, K.V. Palem, and P.G. Spirakis, "Efficient Robust Parallel Computations," Proc. 22nd Ann. Symp. Theory of Computing, pp. 138-148, 1990.
[14] P.B. Gibbons, "A More Practical PRAM Model," Proc ACM Symp. Parallel Algorithms and Architectures, pp. 158-168, ACM, 1989.
[15] C.H. Papadimitriou and M. Yannakakis, "Towards an Architecture-Independent Analysis of Parallel Algorithms," Proc. 20th Ann. ACM Symp. Theory of Computing, pp. 510-513, ACM, 1988.
[16] A. Bar-Noy and S. Kipnis, "Designing Broadcasting Algorithms in the Postal Model for Message-Passing Systems," Proc. ACM Symp. Parallel Algorithms and Architectures, pp. 11-22, June 1992.
[17] A. Aggarwal, A.K. Chandra, and M. Snir, "On Communication Latency in PRAM Computation," Proc. ACM Symp. Parallel Algorithms and Architectures, pp. 11-21, ACM, June 1989.
[18] A. Aggarwal, A.K. Chandra, and M. Snir, "Comm. Complexity of PRAMs," Theoretical Computer Science, pp. 3-28, Mar. 1990.
[19] C.U. Martel and A. Raghunathan, "Asynchronous PRAMs with Memory Latency," Technical Report, Univ. of California, Davis, 1991.
[20] L.G. Valiant, “A Bridging Model for Parallel Computation,” Comm. ACM, vol. 33, no. 8, pp. 103-111, Aug. 1990.
[21] T. von Eicken et al., "Active Messages: a Mechanism for Integrated Communications and Computation," Computer Architecture News, Vol. 20, No. 2, May 1992, pp. 256-266.
[22] C. Shannon and W. Weaver, The Mathematical Theory of Comm., Univ. of Illinois Press: Urbana, 1949.
[23] K. Thearling and S. Smith, "An Improved Supercomputer Sorting Benchmark," Technical Report, Thinking Machines Corp., 1991.
[24] P. Liu, W. Aiello, and S. Bhatt, "An Atomic Model for Message-Passing," Proc. ACM Symp. Parallel Algorithms and Architectures, 1993.
[25] R. Karp,A. Sahay,E. Santos,, and K.E. Schauser,“Optimal broadcast and summation in the LogP model,” Proc. Fifth Ann. Symp. Parallel Algorithms and Architectures, ACM, June 1993.
[26] J.H. Reif and L.G. Valiant, "A Logarithmic Time Sort for Linear Size Networks," J. ACM, vol. 34, pp. 60-76, Jan. 1987.

Index Terms:
Massively parallel processing, models of parallel computation, sorting, algorithm implementation, communication performance, communication schedules, performance prediction, performance analysis, data layout.
Andrea C. Dusseau, David E. Culler, Klaus Erik Schauser, Richard P. Martin, "Fast Parallel Sorting Under LogP: Experience with the CM-5," IEEE Transactions on Parallel and Distributed Systems, vol. 7, no. 8, pp. 791-805, Aug. 1996, doi:10.1109/71.532111
Usage of this product signifies your acceptance of the Terms of Use.