This Article 
 Bibliographic References 
 Add to: 
Generating Software Test Data by Evolution
December 2001 (vol. 27 no. 12)
pp. 1085-1110

This paper discusses the use of genetic algorithms (GAs) for automatic software test data generation. This research extends previous work on dynamic test data generation where the problem of test data generation is reduced to one of minimizing a function. In our work, the function is minimized by using one of two genetic algorithms in place of the local minimization techniques used in earlier research. We describe the implementation of our GA-based system and examine the effectiveness of this approach on a number of programs, one of which is significantly larger than those for which results have previously been reported in the literature. We also examine the effect of program complexity on the test data generation problem by executing our system on a number of synthetic programs that have varying complexities.

[1] W. Miller and D.L. Spooner, “Automatic Generation of Floating Point Test Data,” IEEE Trans. Software Eng., vol. 2, no. 3, pp. 223–226, Sept. 1976.
[2] B. Korel, “Automated Software Test Data Generation,” IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870–879, Aug. 1990.
[3] P. Frankl, D. Hamlet, B. Littlewood, and L. Strigini, “Choosing a Testing Method to Deliver Reliability,” Proc. 19th Int'l Conf. Software Eng., pp. 68-78, May 1997.
[4] R. Ferguson and B. Korel, "The Chaining Approach for Software Test Data Generation," ACM Trans. Software Eng. and Methodology, vol. 5, no. 1, pp. 63-86, Jan. 1996.
[5] M.J. Gallagher and V.L. Narasimhan, “Adtest: A Test Data Generation Suite for Ada Software Systems,” IEEE Trans. Software Eng., vol. 23, no. 8, pp. 473–484, Aug. 1997.
[6] J.H. Holland, Adaptation in Natural and Artificial Systems. Univ. of Michigan Press, 1975.
[7] S. Kirkpatrick, C.D. Gellat Jr., and M.P. Vecchi, “Optimization by Simulated Annealing,” Science, vol. 220, no. 4,598, pp. 671–680, May 1983.
[8] F. Glover, “Tabu Search Part I, II,” ORSA J. Computing, vol. 1, no. 3, pp. 190–206, 1989.
[9] J.R. Horgan, S. London, and M. Lyu, “Achieving Software Quality with Testing Coverage Measures,” Computer, Sept. 1994, pp. 60-69.
[10] J. Chilenski and S. Miller, “Applicability of Modified Condition/Decision Coverage to Software Testing,” Software Eng. J., pp. 193–200, Sept. 1994.
[11] R. DeMillo and A. Mathur, “On the Uses of Software Artifacts to Evaluate the Effectiveness of Mutation Analysis for Detecting Errors in Production Software,” Technical Report SERC-TR-92-P, Purdue Univ., 1992.
[12] D. Hamlet and R. Taylor, "Partition Testing Does Not Inspire Confidence," IEEE Trans. Software Eng., vol. 16, pp. 1,402-1,412, Dec. 1990.
[13] E.J. Weyuker and B. Jeng,“Analyzing partition testing strategies,” IEEE Trans. Software Engineering, vol. 17, pp. 703-711, 1991.
[14] E.J. Weyuker, “Axiomatizing Software Test Adequacy,” IEEE Trans. Software Eng., vol. 12, no. 12, pp. 1128–1137, Dec. 1986.
[15] P.G. Frankl and E.J. Weyuker,“A formal analysis of the fault-detecting ability of testing methods,” IEEE Transactions on Software Engineering, vol. 19, no. 3, Mar. 1993, pp. 202-213,
[16] B. Korel, “Automated Test Data Generation for Programs with Procedures,” Proc. Int'l Symp. Software Testing and Analysis, pp. 209–215, 1996.
[17] L.A. Clarke, “A System to Generate Test Data Symbolically and Execute Programs,” IEEE Trans. Software Eng., vol. 2, no. 3, pp. 215–222, Sept. 1976.
[18] C.V. Ramamoorty, S.F. Ho, and W.T. Chen, “On the Automated Generation of Program Test Data,” IEEE Trans. Software Eng., vol. 2, no. 4, pp. 293–300, Dec. 1976.
[19] J. Offutt, “An Integrated Automatic Test Data Generation System,” J. Systems Integration, vol. 1, pp. 391–409, 1991.
[20] W.H. Deason, D.B. Brown, K. Chang, and J.H. Cross, "A Rule-Based Software Test Data Generator," IEEE Trans. Knowledge and Data Eng., vol. 3, no. 1, pp. 108-116, Jan. 1991.
[21] K.H. Chang, J.H. Cross II, W.H. Carlisle, and S.-S. Liao, “A Performance Evaluation of Heuristics-Based Test Case Generation Methods for Software Branch Coverage,” Int'l J. Software Eng. and Knowledge Eng., vol. 6, no. 4, pp. 585–608, 1966.
[22] N. Tracey, J. Clark, and K. Mander, “The Way Forward for Unifying Dynamic Test-Case Generation: The Optimisation-Based Approach,” Proc. Int'l Workshop Dependable Computing and Its Applications (DCIA), pp. 169–180, Jan. 1998.
[23] N. Tracey, J. Clark, and K. Mander, “Automated Program Flaw Finding Using Simulated Annealing,” Proc. Int'l Symp. Software Testing and Analysis, Software Eng. Notes, pp. 73–81, Mar. 1998.
[24] H. Tracey, J. Clark, K. Mander, and J. McDermid, “An Automated Framework for Structural Test-Data Generation,” Proc. Automated Software Eng. '98, pp. 285–288, 1998.
[25] C.C. Michael, G.E. McGraw, and M.A. Schatz, “Genetic Algorithms for Dynamic Test Data Generation,” Proc. Automated Software Eng. '97, pp. 307–308, 1997.
[26] C.C. Michael, G.E. McGraw, and M.A. Schatz, “Opportunism and Diversity in Automated Software Test Data Generation,” Proc. Automated Software Eng. '98, pp. 136–146, 1998.
[27] A.C. Schultz, J.C. Grefenstette, and K.A. DeJong, “Test and Evaluation by Genetic Algorithms,” IEEE Expert, pp. 9–14, Oct. 1993.
[28] W.H. Press, B.P. Flannery, S.A. Teukolsky, and W.T. Vetterling, Numerical Recipes in C. New York: Cambridge Univ. Press, 1991.
[29] J. Skorin-Kapov, “Tabu Search Applied to the Quadratic Assignment Problem,” ORSA J. Computing, vol. 2, no. 1, pp. 33–41, Winter 1990.
[30] D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning. Reading, Mass.: Addison-Wesley, 1989.
[31] M. Mitchell, An Introduction to Genetic Algorithms. MIT Press, 1996.
[32] Foundations of Genetic Algorithms. G. Rawlins, ed. San Mateo, Calif.: Morgan Kaufmann, 1991.
[33] R. Storn, “On the Usage of Differential Evolution for Function Optimization,” Proc. North Am. Fuzzy Information Prossessing Soc., (NAFIPS '96), pp. 519–523, June 1996.
[34] S.K. Park and K.W. Miller, “Random Number Generators: Good Ones are Hard to Find,” Comm. ACM, vol. 31, no. 10, pp. 1192–1201, Oct. 1988.
[35] R.A. DeMillo and A.J. Offutt, “Experimental Results from an Automatic Test Case Generator,” ACM Trans. Software Eng. Methodology, vol. 2, no. 1, pp. 215–222, Jan. 1993.
[36] J.D. Musa, "Operational Profiles in Software Reliability Engineering," IEEE Software, vol. 10, no. 2, pp. 14-32, 1993.

Index Terms:
Software testing, automatic test case generation, code coverage, genetic algorithms, combinatorial optimization
C.C. Michael, G. Mcgraw, M.A. Schatz, "Generating Software Test Data by Evolution," IEEE Transactions on Software Engineering, vol. 27, no. 12, pp. 1085-1110, Dec. 2001, doi:10.1109/32.988709
Usage of this product signifies your acceptance of the Terms of Use.