This Article 
 Bibliographic References 
 Add to: 
Efficient Evaluation of Multifactor Dependent System Performance Using Fractional Factorial Design
September 2003 (vol. 29 no. 9)
pp. 769-781

Abstract—Performance of computer-based systems may depend on many different factors, internal and external. In order to design a system to have the desired performance or to validate that the system has the required performance, the effect of the influencing factors must be known. Common methods give no or little guidance on how to vary the factors during prototyping or validation. Varying the factors in all possible combinations would be too expensive and too time-consuming. This paper introduces a systematic approach to the prototyping and the validation of a system's performance, by treating the prototyping or validation as an experiment, in which the fractional factorial design methodology is commonly used. To show that this is possible, a case study evaluating the influencing factors of the false and real target rate of a radar system is described. Our findings show that prototyping and validation of system performance become structured and effective when using the fractional factorial design. The methodology enables planning, performance, structured analysis, and gives guidance for appropriate test cases. The methodology yields not only main factors, but also interacting factors. The effort is minimized for finding the results, due to the methodology. The case study shows that after 112 test cases, of 1,024 possible, the knowledge gained was enough to draw conclusions on the effects and interactions of 10 factors. This is a reduction with a factor 5-9 compared to alternative methods.

[1] Software Engineering Product Quality Part 1: Quality model, ISO-9126-1, 2001.
[2] N.E. Fenton and S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach, second ed. Thomson Computer Press, 1998.
[3] T. Berling and P. Runeson, Application of Factorial Design to Validation of System Performance Proc. Seventh IEEE Int'l Conf. and Workshop the Eng. of Computer-Based Systems, pp. 318-326, Apr. 2000.
[4] G.E.P. Box, W.G. Hunter, and J.S. Hunter, Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. John Wiley&Sons, 1978.
[5] R.R. Barton, Designing Simulation Experiments Proc. 2001 Winter Simulation Conf., B.A. Peters, J.S. Smith, D.J. Medeiros, and M.W. Rohrer, eds., pp. 47-52, 2001.
[6] J.M. Donohue, Experimental Design for Simulation Proc. 1994 Winter Simulation Conf., J.D. Tew, S. Manivannan, D.A. Sadowski, and A.F. Seila, eds., pp. 200-206, 1994.
[7] J.P.C. Kleijnen, Statistical Tools for Simulation Practitioners. New York: Marcel Dekker, 1987.
[8] A.M. Law and W.D. Kelton, Simulation Modeling and Analysis. third ed. McGraw-Hill, 2000.
[9] D.M. Cohen, S.R. Dalal, J. Parelius, and G.C. Patton, "The Combinatorial Design Approach to Automatic Test Generation," IEEE Software, vol. 13, no. 5, pp. 83-89, Sept. 1996.
[10] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, The AETG System: An Approach to Testing Based on Combinatorial Design IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437-443, July 1997.
[11] W.K. Ehrlich, I.S. Dunietz, B.D. Szablak, C.L. Mallows, and A. Iannino, "Applying Design of Experiments to Software Testing," Proc. 19th Int'l Conf. Software Eng., IEEE, 1997.
[12] R. Mandl, "Orthogonal Latin Squares: An Application of Experimental Design to Compiler Testing," Comm. ACM, vol. 28, no. 10, pp. 1,054-1,058, Oct. 1985.
[13] R. Brownlie, J. Prowse, and M.S. Phadke, Robust Testing of AT&T PMX/StarMail Using OATS AT&T Technical J., vol. 71, no. 3, pp. 41-47, 1992.
[14] M. Hall Jr., Combinatorial Theory, second ed. Wiley Interscience, 1998.
[15] G. Taguchi, System of Experimental Design, vol. 1. UNIPUB/Kraus Int'l Publications, 1987.
[16] M.S. Phadke, Quality Eng. Using Robust Design.Englewood Cliffs, N.J.: Prentice Hall, 1989.
[17] D.C. Montgomery, Design and Analysis of Experiments, fifth ed. John Wiley and Sons, 2000.
[18] R.R. Barton, Graphical Methods for the Design of Experiments. Springer-Verlag, 1999.
[19] B.E. Ankenman, Design of Experiments with Two- and Four-Level Factors J. Quality Technology, vol. 31, no. 4, pp. 363-375, 1999.
[20] T.C. Bingham, An Approach to Developing Multi-Level Fractional Factorial Designs J. Quality Technology, vol. 29, no. 4, pp. 370-380, 1997.

Index Terms:
Fractional factorial design, performance of systems, performance evaluation, prototyping.
Tomas Berling, Per Runeson, "Efficient Evaluation of Multifactor Dependent System Performance Using Fractional Factorial Design," IEEE Transactions on Software Engineering, vol. 29, no. 9, pp. 769-781, Sept. 2003, doi:10.1109/TSE.2003.1232283
Usage of this product signifies your acceptance of the Terms of Use.