This Article 
 Bibliographic References 
 Add to: 
The AETG System: An Approach to Testing Based on Combinatorial Design
July 1997 (vol. 23 no. 7)
pp. 437-444

Abstract—This paper describes a new approach to testing that uses combinatorial designs to generate tests that cover the pair-wise, triple, or n-way combinations of a system's test parameters. These are the parameters that determine the system's test scenarios. Examples are system configuration parameters, user inputs and other external events. We implemented this new method in the AETG system. The AETG system uses new combinatorial algorithms to generate test sets that cover all valid n-way parameter combinations. The size of an AETG test set grows logarithmically in the number of test parameters. This allows testers to define test models with dozens of parameters. The AETG system is used in a variety of applications for unit, system, and interoperability testing. It has generated both high-level test plans and detailed test cases. In several applications, it greatly reduced the cost of test plan development.

[1] R. Brownlie, J. Prowse, and M. Phadke, "Robust Testing of AT&T PMX/StarMail Using OATS," AT&T Technical J., vol. 71, no. 3, pp. 41-47, Mar. 1992.
[2] K. Burr and W. Young, "Test Acceleration and Automatic Efficient Testcase Generation," NortelTechnical Report, May 1997.
[3] K. Burroughs, A. Jain, and R.L. Erickson, "Improved Quality of Protocol Testing through Techniques of Experimental Design," Proc. Supercomm/IEEE Int'l Conf. Comm. '94, pp. 745-752, 1994.
[4] D.M. Cohen, S.R. Dalal, A. Kajla, and G.C. Patton, "The Automatic Efficient Tests Generator," Proc. Fifth Int'l Symp. Software Reliability Eng., pp. 303-309, IEEE, 1994.
[5] D.M. Cohen, S.R. Dalal, J. Parelius, and G.C. Patton, "The Combinatorial Design Approach to Automatic Test Generation," IEEE Software, vol. 13, no. 5, pp. 83-89, Sept. 1996.
[6] D.M. Cohen, S.R. Dalal, G. Horruitiner, and G.C. Patton, "The AETG System," Fifth Int'l Conf. Software Testing, Analysis and Review, Software Quality Eng., Jacksonville, Fla., 1996.
[7] D.M. Cohen and M.L. Fredman, "New Techniques for Designing Qualitatively Independent Systems," J. of Combinatorial Designs, to appear.
[8] S.R. Dalal and G.C. Patton, "Automatic Efficient Test Generator (AETG): A Test Generation System for Screen Testing, Protocol Verification, and Feature Interactions Testing," Internal Bellcore Technical Memorandum, 1993.
[9] J. Duran and S. Ntafos, "An Evaluation of Random Testing," IEEE Tran. Software Eng., vol. 10, pp. 438-444, July 1984.
[10] W.K. Ehrlich, I.S. Dunietz, B.D. Szablak, C.L. Mallows, and A. Iannino, "Applying Design of Experiments to Software Testing," Proc. 19th Int'l Conf. Software Eng., IEEE, 1997.
[11] L. Gargano, J. Körner, and U. Vaccaro, "Sperner Capacities," Graphs and Combinatorics, vol. 9, pp. 31-46, 1993.
[12] M. Hall Jr., Combinatorial Theory.New York: Wiley Interscience, 1986.
[13] D. Hamlet and R. Taylor, "Partition Testing Does Not Inspire Confidence," IEEE Trans. Software Eng., vol. 16, pp. 1,402-1,412, Dec. 1990.
[14] E. Heller, "Using Design of Experiment Structures to Generate Software Test Cases," 12th Int'l Conf. Testing Computer Software, pp. 33-41, June 1995,.
[15] J.R. Horgan and S. London, "ATAC: A Data Flow Coverage Testing Tool for C," Proc. IEEE Assessment of Quality Software Development Tools, pp. 2-10, IEEE, 1992.
[16] C. Mallows, "Covering Designs in Random Environments," Festschrift for John Tukey, 1997, to appear.
[17] R. Mandl, "Orthogonal Latin Squares: An Application of Experimental Design to Compiler Testing," Comm. ACM, vol. 28, no. 10, pp. 1,054-1,058, Oct. 1985.
[18] M.S. Phadke, Quality Eng. Using Robust Design.Englewood Cliffs, N.J.: Prentice Hall, 1989.
[19] G. Sherwood, "Effective Testing of Factor Combinations," Third Int'l Conf. Software Testing, Analysis and Review, Software Quality Eng., Jacksonville, Fla., 1994.
[20] N.J.A. Sloane, "Covering Arrays and Intersecting Codes," J. Combinatorial Designs, vol. 1, no. 1, pp. 51-63, 1993.
[21] G. Taguchi, System of Experimental Design. Quality Resources, 1987. Translation of Jikken keikakuho, Maurzen Co., Tokyo, 1976.
[22] C.H. West, "Protocol Validation—Principles and Applications," Computer Networks and ISDN Systems, vol. 24, no. 3, pp. 219-242, May 1992.
[23] W.E. Wong, J.R. Horgan, S. London, and A.P. Mathur, "Effect of Test Set Minimization on Fault Detection Effectiveness," Proc. 17th Int'l Conf. Software Eng., pp. 41-50, IEEE, 1995.

Index Terms:
Testing, combinatorial designs, experimental designs, orthogonal arrays.
David M. Cohen, Siddhartha R. Dalal, Michael L. Fredman, Gardner C. Patton, "The AETG System: An Approach to Testing Based on Combinatorial Design," IEEE Transactions on Software Engineering, vol. 23, no. 7, pp. 437-444, July 1997, doi:10.1109/32.605761
Usage of this product signifies your acceptance of the Terms of Use.