This Article 
 Bibliographic References 
 Add to: 
Computer Benchmark Evaluation and Design of Experiments: A Case Study
October 1992 (vol. 41 no. 10)
pp. 1279-1289

The author defines installation tuning and promotes it as an important area of concern for computer configurations. Examples include deciding on the paging configuration for a particular workload and partitioning available memory into system and user areas. Available tuning options are often difficult to select due to poor understanding of their effects, so analytic models rarely exist for these areas. The author illustrates the use of various models to assess the significance of installation tuning factors. A case study is presented in which a statistical model for response time is developed incorporating the key factors of user load, paging, and main storage configuration. A case study is presented in which a statistical design of experiments to computer performance analysis is presented. General statistical issues pertaining to the design and analysis of computer performance experiments are identified and illustrated.

[1] Y. Bard, "Experimental evaluation of system performance,"IBM Syst. J., vol. 12, no. 3, pp. 302-314, IBM Corp. 1973.
[2] Y. Bard, "An experimental approach to system tuning," inProc. Int. Symp. Comput. Perform. Modeling, Measurement and Eval., Mar. 1976.
[3] Y. Bard and K. V. Suryanarayana, "On the structure of CP-67 overhead," inStatistical Computer Performance Evaluation, W. Freiberger, Ed., 1972, pp. 329-346.
[4] R. Berry "Computer benchmark evaluation and design of experiments: A case study," IBM Corp., May 1990 (RC15762).
[5] BGS Corporation, Best/1-MVS Product Description, BGS Corp., 1988 (85066-01).
[6] Boole&Babbage, Inc., Model 300 Product Description, Boole&Babbage, Inc., June 1988 (TD2043-2).
[7] G. E. P. Box, W. G. Hunter, and J. Stuart Hunter,Statistics for Experimenters, New York: Wiley, 1978.
[8] G. P. Box, L. R. Connor, W. R. Cousins, O. L. Davies, F. R. Himsworth, and G. P. Sillitto,Design and Analysis of Industrial Experiments, second ed., O.L. Davies, Ed. Longman Group Ltd., 1978.
[9] P. Bratley, B. L. Fox, and L. E. Schrage,A Guide to Simulation, New York: Springer-Verlag, 1983.
[10] G. Burkland, P. Heidelberger, M. Schatzoff, P. Welch, and L. Wu "An APL system for interactive scientific-engineering graphics and data analysis,"APL84 Proc., pp. 95-102, June 1984.
[11] W. G. Cochran and G. M. Cox,Experimental Designs, second ed. New York: Wiley, 1957.
[12] R. W. Conway, "Some tactical problems in digital simulation,"Management Sci., vol. 10, pp. 47-61, 1963.
[13] C. Daniel and F. S. Wood,Fitting Equations to Data: Computer Analysis of Multifactor Data, second ed. New York: Wiley, 1980.
[14] N. R. Draper and H. Smith,Applied Regression Analysis, second ed. New York Wiley, 1981.
[15] D. J. Finney, "The fractional replication of factorial arrangements,"Ann. Eugenics, vol. 12, no. 4, pp. 291-301, 1945.
[16] U. Grenander and R.F. Tsao, "Quantitative methods for evaluating computer system performance: A review and proposals," inStatistical Computer Performance Evaluation, W. Freiberger, Ed., 1972, pp. 3-24.
[17] P. Heidelberger and S. S. Lavenberg, "Computer performance evaluation methodology,"IEEE Trans. Comput., vol. C-33, no. 12, pp. 1195-1220, Dec. 1984.
[18] IBM Corp. Managing VM/SP HPO Real Storage Constraint Below the 16 megabyte Line, Oct. 1986 (GG66-0261).
[19] IBM Corp. Virtual Machine Monitor Analysis Program: User's Guide and Reference, 1985 (SC34-2166).
[20] IBM Corp., VM/SP performance Tuning Guide, Nov., 1986 (GG24-3112).
[21] IBM Corp., General Information: VM/Integrated System Release 5, 1987 (GH24-5119).
[22] T. Lane and P. Welch, "The integration of a menu-oriented graphical statistical system with its underlying general purpose language,"Comput. Sci. and Statist. Proc. 19th Symp. Interface, Feb. 1987, pp. 267-273.
[23] S. S. Lavenberg and C. H. Sauer, "Analytical results for queueing models," inComputer Performance Modeling Handbook, S. S. Lavenberg, Ed. New York: Academic, 1983.
[24] B. H. Margolin, R. P. Parmelee, and M. Schatzoff, "Analysis of free storage algorithms,"IBM Syst. J., vol. 10, no. 4, pp. 283-304, IBM Corp., 1971.
[25] M. Schatzoff, "Design of experiments in computer performance evaluation,"IBM J. Res. Develop., vol. 25, no. 6, pp. 848-859, IBM Corp., 1981.
[26] M. Schatzoff and C.C. Tillman, "Design of experiments in simulator validation,"IBM Syst. J., vol. 19, no. 3, pp. 252-262, IBM Corp., 1975.
[27] R. F. Tsao, L. W. Comeau, and B. H. Margolin, "A multi-factor paging experiment: I. The experiment and conclusions," inStatistical Computer Performance Evaluation, W. Freiberger, Ed., 1972, pp. 103-134.
[28] R. F. Tsao and B. H. Margolin, "A multi-factor paging experiment: II, Statistical methodology," inStatistical Computer Performance Evaluation, W. Freiberger, Ed., 1972, pp. 135-158.
[29] P. D. Welch "The statistical analysis of simulation results," inComputer Performance Modeling Handbook, S.S. Lavenberg, Ed. New York: Academic, 1983.
[30] J. L. Wolf, "The placement optimization program: A practical solution to the disk file assignment problem," inProc. Sigmetrics Conf., Berkeley, CA, 1989.

Index Terms:
computer benchmark evaluation; design of experiments; installation tuning; computer configurations; paging configuration; models; statistical model; response time; user load; main storage configuration; computer performance analysis; computer installation; configuration management; performance evaluation.
R. Berry, "Computer Benchmark Evaluation and Design of Experiments: A Case Study," IEEE Transactions on Computers, vol. 41, no. 10, pp. 1279-1289, Oct. 1992, doi:10.1109/12.166605
Usage of this product signifies your acceptance of the Terms of Use.