This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Effectively Utilizing Global Cluster Memory for Large Data-Intensive Parallel Programs
January 2006 (vol. 17 no. 1)
pp. 66-77

Abstract—Large scientific parallel applications demand large amounts of memory space. Current parallel computing platforms schedule jobs without fully knowing their memory requirements. This leads to uneven memory allocation in which some nodes are overloaded. This, in turn, leads to disk paging, which is extremely expensive in the context of scientific parallel computing. To solve this problem, we propose a new peer-to-peer solution called Parallel Network RAM. This approach avoids the use of disk, better utilizes available RAM resources, and will allow larger problems to be solved while reducing the computational, communication, and synchronization overhead typically involved in parallel applications. We proposed several different Parallel Network RAM designs and evaluated the performance of each under different conditions. We discovered that different designs are appropriate in different situations.

[1] D.G. Feitelson, L. Rudolph, U. Schwiegelshohn, K.C. Sevcik, and P. Wong, “Theory and Practice In Parallel Job Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 1-34, 1997.
[2] A. Batat and D.G. Feitelson, “Gang Scheduling with Memory Considerations,” Proc. IEEE 2000 Int'l Parallel and Distributed Processing Symp., pp. 109-114, 2000.
[3] D.C. Burger, R.S. Hyder, B.P. Miller, and D.A. Wood, “Paging Tradeoffs in Distributed-Shared-Memory Multiprocessors,” The J. Supercomputing, vol. 10, no. 1, pp. 87-104, 1996.
[4] E.W. Parsons and K.C. Sevcik, “Benefits of Speedup Knowledge in Memory-Constrained Multiprocessor Scheduling,” Performance Evaluation, vols. 27/28, no. 4, pp. 253-272, 1996.
[5] A. Acharya and S. Setia, “Availability and Utility of Idle Memory in Clusters of Workstations,” Proc. ACM SIGMETRICS Conf. Measurement and Modeling of Computer Systems, 1999.
[6] M.D. Flouris and E.P. Markatos, High Performance Cluster Computing. Prentice Hall, pp. 383-408, 1999.
[7] M.J. Feeley, W.E. Morgan, F.H. Pighin, A.R. Karlin, H.M. Levy, and C.A. Thekkath, “Implementing Global Memory Management In A Workstation Cluster,” Proc. Symp. Operating Systems Principles, pp. 201-212, 1995.
[8] E.P. Markatos and G. Dramitinos, “Implementation of a Reliable Remote Memory Pager,” Proc. USENIX Ann. Technical Conf., pp. 177-190, 1996.
[9] L. Xiao, X. Zhang, and S.A. Kubricht, “Incorporating Job Migration and Network Ram to Share Cluster Memory Resources,” Proc. Ninth IEEE Int'l Symp. High Performance Distributed Computing, pp. 71-78, 2000.
[10] D.G. Feitelson, “Packing Schemes for Gang Scheduling,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 89-110, 1996.
[11] B.B. Zhou, D. Welsh, and R.P. Brent, “Resource Allocation Schemes for Gang Scheduling,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 74-86, 2000.
[12] A.B. Downey, “Lachesis: A Job Scheduler for the Cray T3E,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 47-61, 1998.
[13] A. Hori, H. Tezuka, and Y. Ishikawa, “Overhead Analysis of Preemptive Gang Scheduling,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 217-230, 1998.
[14] Y. Wiseman and D. Feitelson, “Paired Gang Scheduling,” Technical Report 2001-10, Hebrew Univ., 2001.
[15] S.K. Setia, “The Interaction between Memory Allocation and Adaptive Partitioning in Message-Passing Multicomputers,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 146-164, 1995.
[16] Y. Zhang, A. Sivasubramaniam, H. Franke, and J.E. Moreira, “Improving Parallel Job Scheduling by Combining Gang Scheduling and Backfilling Techniques. ” Proc. IEEE 2000 Int'l Parallel and Distributed Processing Symp., pp. 133-142, 2000.
[17] S. Srinivasan, R. Kettimuthu, V. Subramani, and P. Sadayappan, “Characterization of Backfilling Strategies for Parallel Job Scheduling,” Proc. 2002 Int'l Workshops Parallel Processing, 2002.
[18] E.W. Parsons and K.C. Sevcik, “Coordinated Allocation of Memory and Processors in Multiprocessors,” Proc. 1996 ACM SIGMETRICS Conf. Measurement and Modeling of Computer Systems, pp. 57-67, 1996.
[19] F. Wang, M. Papaefthymiou, and M. Squillante, “Performance Evaluation of Gang Scheduling for Parallel and Distributed Multiprogramming,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 277-298, 1997.
[20] A. Barak and A. Braverman, “Memory Ushering in a Scalable Computing Cluster,” J. Microprocessors and Microsystems, vol. 22, nos. 3-4, pp. 175-182, Aug. 1998.
[21] L. Xiao, S. Chen, and X. Zhang, “Dynamic Cluster Resource Allocations for Jobs With Known and Unknown Memory Demands,” IEEE Trans. Parallel and Distributed Systems, vol. 13, no. 3, pp. 223-240, Mar. 2002.
[22] L. Xiao, S. Chen, and X. Zhang, “Adaptive Memory Allocations in Clusters to Handle Unexpectedly Large Data-Intensive Jobs,” IEEE Trans. Parallel and Distributed Systems, vol. 15, no. 6, June 2004.
[23] J. Oleszkiewicz and L. Xiao, “Pnrsim: A Parallel Network Ram Simulator,” Technical Report MSU-CSE-04-13, Computer Science and Eng., Michigan State Univ., East Lansing, Apr. 2004.
[24] S.J. Chapin, W. Cirne, D.G. Feitelson, J.P. Jones, S.T. Leutenegger, U. Schwiegelshohn, W. Smith, and D. Talby, “Benchmarks and Standards for the Evaluation of Parallel Job Schedulers,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 67-90, 1999.
[25] D.G. Feitelson, “Memory Usage in the LANL CM-5 Workload,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 78-94, 1997.
[26] V. Lo, J. Mache, and K. Windisch, “A Comparative Study of Real Workload Traces and Synthetic Workload Models for Parallel Job Scheduling,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 25-46, 1998.
[27] D.G. Feitelson and L. Rudolph, “Metrics and Benchmarking for Parallel Job Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 1-24, 1998.
[28] U. Schwiegelshohn and R. Yahyapour, “Improving First-Come-First-Serve Job Scheduling by Gang Scheduling,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 180-198, 1998.
[29] F. Darema-Rogers, G.F. Pfister, and K. So, “Memory Access Patterns of Parallel Scientific Programs,” Performance Evaluation Rev., vol. 15, no. 1, pp. 46-57, 1987.
[30] K.C. Sevcik and S. Zhou, “Performance Benefits and Limitations of Large NUMA Multiprocessors,” Performance Evaluation, vol. 20, pp. 185-205, 1994.
[31] D.G. Feitelson, “Metrics for Parallel Job Scheduling and Their Convergence,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 188-205, 2001.
[32] J.P. Jones and B. Nitzberg, “Scheduling for Parallel Supercomputing: A Historical Perspective Of Achievable Utilization,” Job Scheduling Strategies for Parallel Processing, D.G. Feitelson and L. Rudolph, eds., pp. 1-16, 1999.
[33] J. Oleszkiewicz, L. Xiao, and Y. Liu, “Parallel Network Ram: Effectively Utilizing Global Cluster Memory for Large Data-Intensive Parallel Programs,” Proc. 2004 Int'l Conf. Parallel Processing, pp. 353-360, 2004.

Index Terms:
Parallel programs, peer-to-peer, cluster, network RAM, scheduling, simulation.
Citation:
John Oleszkiewicz, Li Xiao, Yunhao Liu, "Effectively Utilizing Global Cluster Memory for Large Data-Intensive Parallel Programs," IEEE Transactions on Parallel and Distributed Systems, vol. 17, no. 1, pp. 66-77, Jan. 2006, doi:10.1109/TPDS.2006.10
Usage of this product signifies your acceptance of the Terms of Use.