|
| This Article | ||
| ||
| Share | ||
| Bibliographic References | ||
| Add to: | ||
 Digg  Furl  Spurl  Blink  Simpy  Google  Del.icio.us  Y!MyWeb | ||
| Search | ||
| ||
| ASCII Text | x | ||
| Dror G. Feitelson, "Experimental Analysis of the Root Causes of Performance Evaluation Results: A Backfilling Case Study," IEEE Transactions on Parallel and Distributed Systems, vol. 16, no. 2, pp. 175-182, February, 2005. | |||
| BibTex | x | ||
| @article{ 10.1109/TPDS.2005.18, author = {Dror G. Feitelson}, title = {Experimental Analysis of the Root Causes of Performance Evaluation Results: A Backfilling Case Study}, journal ={IEEE Transactions on Parallel and Distributed Systems}, volume = {16}, number = {2}, issn = {1045-9219}, year = {2005}, pages = {175-182}, doi = {http://doi.ieeecomputersociety.org/10.1109/TPDS.2005.18}, publisher = {IEEE Computer Society}, address = {Los Alamitos, CA, USA}, } | |||
| RefWorks Procite/RefMan/Endnote | x | ||
| TY - JOUR JO - IEEE Transactions on Parallel and Distributed Systems TI - Experimental Analysis of the Root Causes of Performance Evaluation Results: A Backfilling Case Study IS - 2 SN - 1045-9219 SP175 EP182 EPD - 175-182 A1 - Dror G. Feitelson, PY - 2005 KW - Performance evaluation KW - sensitivity of results KW - experimental verification KW - simulation KW - parallel job scheduling KW - backfilling. VL - 16 JA - IEEE Transactions on Parallel and Distributed Systems ER - | |||
Abstract—The complexity of modern computer systems may enable minor variations in performance evaluation procedures to actually determine the outcome. Our case study concerns the comparison of two parallel job schedulers, using different workloads and metrics. It shows that metrics may be sensitive to different job classes, and not measure the performance of the whole workload in an impartial manner. Workload models may implicitly assume that some workload attribute is unimportant and does not warrant modeling; this too can turn out to be wrong. As such effects are hard to predict, a careful experimental methodology is needed in order to find and verify them.
[1] S-H. Chiang, A. Arpaci-Dusseau, and M.K. Vernon, “The Impact of More Accurate Requested Runtimes on Production Job Scheduling Performance,” Job Scheduling Strategies for Parallel Processing, pp. 103-127, Springer Verlag, 2002.
[2] W. Cirne and F. Berman, “A Model for Moldable Supercomputer Jobs,” Proc. 15th Int'l Parallel and Distributed Processing Symp., Apr. 2001.
[3] D.G. Feitelson, “Metric and Workload Effects on Computer Systems Evaluation,” Computer, vol. 36, no. 9, pp. 18-25, Sept. 2003.
[4] D.G. Feitelson, “Packing Schemes for Gang Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 89-110, Springer-Verlag, 1996.
[5] D.G. Feitelson, L. Rudolph, U. Schwiegelshohn, K.C. Sevcik, and P. Wong, “Theory and Practice in Parallel Job Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 1-34, Springer Verlag, 1997.
[6] R. Jain, The Art of Computer Systems Performance Analysis. John Wiley and Sons, 1991.
[7] J. Jann, P. Pattnaik, H. Franke, F. Wang, J. Skovira, and J. Riodan, “Modeling of Workload in MPPs,” Job Scheduling Strategies for Parallel Processing, pp. 95-116, Springer Verlag, 1997.
[8] A.M. Law and W.D. Kelton, Simulation Modeling and Analysis, third ed. McGraw Hill, 2000.
[9] C.B. Lee, Y. Schwartzman, J. Hardy, and A. Snavely, “Are User Runtime Estimates Inherently Inaccurate?” Job Scheduling Strategies for Parallel Processing, Springer-Verlag, 2004.
[10] D. Lifka, “The ANL/IBM SP Scheduling System,” Job Scheduling Strategies for Parallel Processing, pp. 295-303, Springer-Verlag, 1995.
[11] V. Lo, J. Mache, and K. Windisch, “A Comparative Study of Real Workload Traces and Synthetic Workload Models for Parallel Job Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 25-46, Springer Verlag, 1998.
[12] M.H. MacDougall, Simulating Computer Systems: Techniques and Tools. MIT Press, 1987.
[13] A.W. Mu'alem and D.G. Feitelson, “Utilization, Predictability, Workloads, and User Runtime Estimates in Scheduling the IBM SP2 with Backfilling,” IEEE Trans. Parallel and Distributed Systems, vol. 12, no. 6, pp. 529-543, June 2001.
[14] S. Srinivasan, R. Kettimuthu, V. Subramani, and P. Sadayappan, “Selective Reservation Strategies for Backfill Job Scheduling,” Job Scheduling Strategies for Parallel Processing, pp. 55-71, Springer-Verlag, 2002.
[15] D. Tsafrir and D.G. Feitelson, “Workload Flurries,” Technical Report 2003-85, Hebrew Univ., Nov. 2003.
[16] D. Zotkin and P.J. Keleher, “Job-Length Estimation and Performance in Backfilling Schedulers,” Proc. Eighth Int'l Symp. High Performance Distributed Computing, Aug. 1999.