This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Experimental Analysis of the Root Causes of Performance Evaluation Results: A Backfilling Case Study
February 2005 (vol. 16 no. 2)
pp. 175-182

Abstract—The complexity of modern computer systems may enable minor variations in performance evaluation procedures to actually determine the outcome. Our case study concerns the comparison of two parallel job schedulers, using different workloads and metrics. It shows that metrics may be sensitive to different job classes, and not measure the performance of the whole workload in an impartial manner. Workload models may implicitly assume that some workload attribute is unimportant and does not warrant modeling; this too can turn out to be wrong. As such effects are hard to predict, a careful experimental methodology is needed in order to find and verify them.

[1] S-H. Chiang, A. Arpaci-Dusseau, and M.K. Vernon, “The Impact of More Accurate Requested Runtimes on Production Job Scheduling Performance,” Job Scheduling Strategies for Parallel Processing, pp. 103-127, Springer Verlag, 2002.
[2] W. Cirne and F. Berman, “A Model for Moldable Supercomputer Jobs,” Proc. 15th Int'l Parallel and Distributed Processing Symp., Apr. 2001.
[3] D.G. Feitelson, “Metric and Workload Effects on Computer Systems Evaluation,” Computer, vol. 36, no. 9, pp. 18-25, Sept. 2003.
[4] D.G. Feitelson, “Packing Schemes for Gang Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 89-110, Springer-Verlag, 1996.
[5] D.G. Feitelson, L. Rudolph, U. Schwiegelshohn, K.C. Sevcik, and P. Wong, “Theory and Practice in Parallel Job Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 1-34, Springer Verlag, 1997.
[6] R. Jain, The Art of Computer Systems Performance Analysis. John Wiley and Sons, 1991.
[7] J. Jann, P. Pattnaik, H. Franke, F. Wang, J. Skovira, and J. Riodan, “Modeling of Workload in MPPs,” Job Scheduling Strategies for Parallel Processing, pp. 95-116, Springer Verlag, 1997.
[8] A.M. Law and W.D. Kelton, Simulation Modeling and Analysis, third ed. McGraw Hill, 2000.
[9] C.B. Lee, Y. Schwartzman, J. Hardy, and A. Snavely, “Are User Runtime Estimates Inherently Inaccurate?” Job Scheduling Strategies for Parallel Processing, Springer-Verlag, 2004.
[10] D. Lifka, “The ANL/IBM SP Scheduling System,” Job Scheduling Strategies for Parallel Processing, pp. 295-303, Springer-Verlag, 1995.
[11] V. Lo, J. Mache, and K. Windisch, “A Comparative Study of Real Workload Traces and Synthetic Workload Models for Parallel Job Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 25-46, Springer Verlag, 1998.
[12] M.H. MacDougall, Simulating Computer Systems: Techniques and Tools. MIT Press, 1987.
[13] A.W. Mu'alem and D.G. Feitelson, “Utilization, Predictability, Workloads, and User Runtime Estimates in Scheduling the IBM SP2 with Backfilling,” IEEE Trans. Parallel and Distributed Systems, vol. 12, no. 6, pp. 529-543, June 2001.
[14] S. Srinivasan, R. Kettimuthu, V. Subramani, and P. Sadayappan, “Selective Reservation Strategies for Backfill Job Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 55-71, Springer-Verlag, 2002.
[15] D. Tsafrir and D.G. Feitelson, “Workload Flurries,” Technical Report 2003-85, Hebrew Univ., Nov. 2003.
[16] D. Zotkin and P.J. Keleher, “Job-Length Estimation and Performance in Backfilling Schedulers,” Proc. Eighth Int'l Symp. High Performance Distributed Computing, Aug. 1999.

Index Terms:
Performance evaluation, sensitivity of results, experimental verification, simulation, parallel job scheduling, backfilling.
Citation:
Dror G. Feitelson, "Experimental Analysis of the Root Causes of Performance Evaluation Results: A Backfilling Case Study," IEEE Transactions on Parallel and Distributed Systems, vol. 16, no. 2, pp. 175-182, Feb. 2005, doi:10.1109/TPDS.2005.18
Usage of this product signifies your acceptance of the Terms of Use.