This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Impact of Workload and System Parameters on Next Generation Cluster Scheduling Mechanisms
September 2001 (vol. 12 no. 9)
pp. 967-985

Abstract—Scheduling of processes onto processors of a parallel machine has always been an important and challenging area of research. The issue becomes even more crucial and difficult as we gradually progress to the use of off-the-shelf workstations, operating systems, and high bandwidth networks to build cost-effective clusters for demanding applications. Clusters are gaining acceptance not just in scientific applications that need supercomputing power, but also in domains such as databases, web service, and multimedia which place diverse Quality-of-Service (QoS) demands on the underlying system. Further, these applications have diverse characteristics in terms of their computation, communication, and I/O requirements, making conventional parallel scheduling solutions, such as space sharing or gang scheduling, unattractive. At the same time, leaving it to the native operating system of each node to make decisions independently can lead to ineffective use of system resources whenever there is communication. Instead, an emerging class of dynamic coscheduling mechanisms that attempt to take remedial actions to guide the system toward coscheduled execution without requiring explicit synchronization offers a lot of promise for cluster scheduling. Using a detailed simulator, this paper evaluates the pros and cons of different dynamic coscheduling alternatives while comparing their advantages over traditional gang scheduling (and not performing any coordinated scheduling at all). The impact of dynamic job arrivals, job characteristics, and different system parameters on these alternatives is evaluated in terms of several performance criteria. In addition, heuristics to enhance one of the alternatives even further are identified, classified, and evaluated. It is shown that these heuristics can significantly outperform the other alternatives over a spectrum of workload and system parameters and is thus a much better option for clusters than conventional gang scheduling.

[1] A.C. Arpaci-Dusseau, D.E. Culler, and A.M. Mainwaring, “Scheduling with Implicit Information in Distributed Systems,” Proc. ACM SIGMETRICS 1998 Conf. Measurement and Modeling of Computer Systems, 1998.
[2] M. Banikazemi, V. Moorthy, D.K. Panda, L. Herger, and B. Abali, “Efficient Virtual Interface Architecture Support for the IBM SP Switch-Connected NT Clusters,” Proc. Int'l Symp. Parallel and Distributed Processing Systems, pp. 33-42, May 2000.
[3] E. Barton, J. Cownie, and M. McLaren, “Message Passing on the Meiko CS-2,” Parallel Computing, vol. 20, no. 4, Apr. 1994.
[4] P. Brinch-Hansen, “An Analysis of Response Ratio Scheduling,” Int'l Federation Information Processing Congress, vol. TA-3, pp. 150-154, Aug. 1971.
[5] M. Buchanan and A. Chien, “Coordinated Thread Scheduling for Workstation Clusters under Windows NT,” Proc. USENIX Windows NT Workshop, Aug. 1997.
[6] M. Crovella, M. Harchol-Balter, and C. Murta, “Task Assignment in a Distributed System: Improving Performance by Unbalancing Load,” Proc. ACM Sigmetrics Conf. Measurement and Modeling of Computer Systems, pp. 268-269, June 1998
[7] A. Dusseau, R. Arpaci, and D. Culler, "Effective Distributed Scheduling of Parallel Workloads," Proc. 1996 ACM Sigmetrics Int'l Conf. Measurement and Modeling of Computer Systems, Assoc. of Computing Machinery, N.Y., May 1996.
[8] D.G. Feitelson, “A Survey of Scheduling in Multiprogrammed Parallel Systems,” Technical Report Research Report RC 19790(87657), IBM T.J. Watson Research Center, Oct. 1994.
[9] D.G. Feitelson and L. Rudolph, “Coscheduling Based on Run-Time Identification of Activity Working Sets,” Technical Report Research Report RC 18416(80519), IBM T.J. Watson Research Center, Oct. 1992.
[10] D.G. Feitelson and L. Rudolph, “Gang Scheduling Performance Benefits for Fine-Grained Synchronization,” J. Parallel and Distributed Computing, vol. 16, no. 4, pp. 306-318, Dec. 1992.
[11] H. Franke, J. Jann, J.E. Moreira, P. Pattnaik, and M.A. Jette, “Evaluation of Parallel Job Scheduling for ASCI Blue-Pacific,” Proc. Supercomputing, Nov. 1999.
[12] G. Ghare and S. Leutenegger, “The Effect of Correlating Quantum Allocation and Job Size for Gang Scheduling,” Proc. Fifth Ann. Workshop Job Scheduling Strategies for Parallel Processing, Apr. 1999.
[13] A. Hori, H. Tezuka, and Y. Ishikawa, “Global State Detection Using Network Preemption,” Proc. Int'l Parallel Processing Symp. Workshop Job Scheduling Strategies for Parallel Processing, pp. 262-276, Apr. 1997.
[14] Intel Corporation, Paragon User's Guide, 1993.
[15] N. Islam, A.L. Prodromidis, M.S. Squillante, L.L. Fong, and A.S. Gopal, “Extensible Resource Management for Cluster Computing,” Proc. 17th Int'l Conf. Distributed Computing Systems, pp. 561-568, 1997.
[16] H. Kaneko, J.A. Stankovic, S. Sen, and K. Ramamritham, "Integrated Scheduling of Multimedia and Hard Real-Time Tasks," Proc. 17th Real-Time Systems Symp. pp. 206-217, Dec. 1996.
[17] D. Lifka, “The ANL/IBM SP Scheduling System,” Proc. Int'l Parallel and Distributed Processing Symp. Workshop Job Scheduling Strategies for Parallel Processing vol. 949, pp. 295-303, Apr. 1995.
[18] J. M. Mellor-Crummey and M. L. Scott,“Algorithms for scalable synchronization on shared-memory multiprocessors,”ACM Trans. Comput. Syst., vol, 9, no. 1, pp. 21–65, Feb. 1991.
[19] J.E. Moreira, H. Franke, W. Chan, L.L. Fong, M.A. Jette, and A. Yoo, “A Gang-Scheduling System for ASCI Blue-Pacific,” High-Performance Computing and Networking, Seventh Int'l Conf., vol. 1593, pp. 831-840, Apr. 1999.
[20] S. Nagar, A. Banerjee, A. Sivasubramaniam, and C. Das, “An Experimental Evaluation of Scheduling Strategies for a Network of Workstations,” Proc. ACM Symp. Parallel Algorithms and Architectures (SPAA), pp. 96-105, June 1999.
[21] S. Nagar, A. Banerjee, A. Sivasubramaniam, and C.R. Das, “Alternatives to Coscheduling a Network of Workstations,” J. Parallel and Distributed Computing, vol. 59, no. 2, pp. 302-327, Nov. 1999.
[22] J.K. Ousterhout, “Scheduling Techniques for Concurrent Systems” Proc. Third Int'l Conf. Distributed Computing Systems, pp. 22-30, May 1982.
[23] S. Pakin, M. Lauria, and A. Chien, "High Performance Messaging on Workstations: Illinois Fast Messages (FM) for Myrinet," Proc. Supercomputing 95, IEEE Computer Society, Los Alamitos, Calif., Dec. 1995.
[24] V.G.J. Peris, M.S. Squillante, and V.K. Naik, “Analysis of the Impact of Memory in Distributed Parallel Processing Systems,” Proc. ACM SIGMETRICS Conf. Measurement and Modeling of Computer Systems, pp. 5-18, May 1994.
[25] F. Petrini and W. Feng, “Buffered Coscheduling: A New Method for Multitasking Parallel Jobs on Distributed Systems,” Proc. Int'l Parallel and Distributed Processing Symp., pp. 439-444, May 2000.
[26] F. Petrini and W. Feng, “Time-Sharing Parallel Jobs in the Presence of Multiple Resource Requirements,” Proc. Sixth Ann. Workshop Job Scheduling Strategies for Parallel Processing, pp. 71-92, May 2000.
[27] R. Poovendran, P. Keleher, and J.S. Baras, “A Decision-Process Analysis of Implicit Coscheduling,” Proc. Int'l Parallel and Distributed Processing Symp., pp. 1115-120, May 2000.
[28] K. Ramamritham, C. Shen, O. Gonzalez, S. Sen, and S.B Shirgurkar, “Using Windows NT for Real-Time Applications: Experimental Observations and Recommendations,” Proc. Fourth IEEE Real-Time Technology and Applications, June 1998.
[29] K. Ramamritham and J.A. Stankovic, “Scheduling Algorithms and Operating System Support for Real Time Systems,” Proc. IEEE, vol. 82, no. 1, Jan. 1994.
[30] C. Shen, O. Gonzalez, K. Ramamritham, and I. Mizunuma, “User Level Scheduling of Communicating Real-Time Tasks,” Proc. IEEE Real-Time Technology and Applications Symp., June 1999.
[31] P.G. Sobalvarro, “Demand-Based Coscheduling of Parallel Jobs on Multiprogrammed Multiprocessors,” PhD thesis, Dept. of Electrical Eng. and Computer Science, Massachusetts. Inst. of Tech nology, Jan. 1997.
[32] M.S. Squillante, Y. Zhang, A. Sivasubramaniam, N. Gautam, H. Franke, and J. Moreira, “Analytic Modeling and Analysis of Dynamic Coscheduling for a Wide Spectrum of Parallel and Distributed Environments,” Technical Report CSE-01-004, Computer Science and Eng. Dept., Pennsylvania State Univ., Feb. 2001.
[33] R. Subrahmaniam, “Implementing Coscheduling Heuristics for Windows NT Clusters,” master's thesis, Dept. of Computer Science and Eng., Pennsylvania State Univ., October 1999.
[34] K. Suzaki and D. Walsh, “Implementation of the Combination of Time Sharing and Space Sharing on AP/Linux,” Int'l Parallel and Distributed Processing Symp. Workshop Job Scheduling Strategies for Parallel Processing, Mar. 1998.
[35] Thinking Machines Corp., The Connection Machine CM-5 Technical Summary, Oct. 1991.
[36] L.W. Tucker and G.G. Robertson, Architecture and Applications of the Connection Machine Computer, vol. 21, no. 8, pp. 26-38, Aug. 1988.
[37] Specification for the Virtual Interface Architecture, http:/www.viarch.org. 2001.
[38] T. von Eicken et al., "U-Net: A User-Level Network Interface for Parallel and Distributed Computing," Proc. 15th ACM Symp. OS Principles, ACM Press, New York, 1995, pp. 40-53.
[39] F. Wang, M. Papaefthymiou, and M. Squillante, “Performance Evaluation of Gang Scheduling for Parallel and Distributed Multiprogramming,” Proc. Int'l Parallel Processing Symp. Workshop Job Scheduling Strategies for Parallel Processing, pp. 277-298, Apr. 1997.
[40] Y. Zhang, H. Franke, J. Moreira, and A. Sivasubramaniam, Improving Parallel Job Scheduling by Combining Gang Scheduling and Backfilling Techniques Proc. Int'l Parallel and Distributed Processing Symp., pp. 133-142, May 2000.
[41] B. Zhou, R. Brent, C. Johnson, and D. Walsh, “Job Re-Packing for Enhancing the Fine and Coarse Grain Parallel Processes,” Proc. Fifth Ann. Workshop Job Scheduling Strategies for Parallel Processing, Apr. 1999.

Index Terms:
Parallel scheduling, gang scheduling, dynamic coscheduling, clusters, simulation.
Citation:
Yanyong Zhang, Anand Sivasubramaniam, José Moreira, Hubertus Franke, "Impact of Workload and System Parameters on Next Generation Cluster Scheduling Mechanisms," IEEE Transactions on Parallel and Distributed Systems, vol. 12, no. 9, pp. 967-985, Sept. 2001, doi:10.1109/71.954632
Usage of this product signifies your acceptance of the Terms of Use.