The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - March/April (2010 vol.36)
pp: 226-247
Mark Harman , King's College London, London
Phil McMinn , University of Sheffield, Sheffield
ABSTRACT
Search-based optimization techniques have been applied to structural software test data generation since 1992, with a recent upsurge in interest and activity within this area. However, despite the large number of recent studies on the applicability of different search-based optimization approaches, there has been very little theoretical analysis of the types of testing problem for which these techniques are well suited. There are also few empirical studies that present results for larger programs. This paper presents a theoretical exploration of the most widely studied approach, the global search technique embodied by Genetic Algorithms. It also presents results from a large empirical study that compares the behavior of both global and local search-based optimization on real-world programs. The results of this study reveal that cases exist of test data generation problem that suit each algorithm, thereby suggesting that a hybrid global-local search (a Memetic Algorithm) may be appropriate. The paper presents a Memetic Algorithm along with further empirical results studying its performance.
INDEX TERMS
Automated test data generation, search-based testing, search-based software engineering, Evolutionary Testing, Genetic Algorithms, Hill Climbing, schema theory, Royal Road, testing and debugging, testing tools, artificial intelligence, problem solving, control methods, and search, heuristic methods, algorithms, experimentation, measurement, performance, theory.
CITATION
Mark Harman, Phil McMinn, "A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search", IEEE Transactions on Software Engineering, vol.36, no. 2, pp. 226-247, March/April 2010, doi:10.1109/TSE.2009.71
REFERENCES
[1] The Software-Artifact Infrastructure Repository, http://sir.unl. edu/portalindex.html, 2009.
[2] J. Aguilar-Ruiz, I. Ramos, J.C. Riquelme, and M. Toro, "An Evolutionary Approach to Estimating Software Development Projects," Information and Software Technology, vol. 43, no. 14, pp. 875-882, 2001.
[3] G. Antoniol, M.D. Penta, and M. Harman, "Search-Based Techniques Applied to Optimization of Project Planning for a Massive Maintenance Project," Proc. IEEE Int'l Conf. Software Maintenance, pp. 240-249, 2005.
[4] A.J. Bagnall, V.J. Rayward-Smith, and I.M. Whittley, "The Next Release Problem," Information and Software Technology, vol. 43, no. 14, pp. 883-890, 2001.
[5] J.E. Baker, "Reducing Bias and Inefficiency in the Selection Algorithm," Proc. Second Int'l Conf. Genetic Algorithms and Their Application, 1987.
[6] A. Baresel and H. Sthamer, "Evolutionary Testing of Flag Conditions," Proc. Genetic and Evolutionary Computation Conf., pp. 2442-2454, 2003.
[7] A. Baresel, H. Sthamer, and M. Schmidt, "Fitness Function Design to Improve Evolutionary Structural Testing," Proc. Genetic and Evolutionary Computation Conf., pp. 1329-1336, 2002.
[8] A. Bertolino, "Software Testing Research: Achievements, Challenges, Dreams," Proc. Int'l Conf. Future of Software Eng. 2007, pp. 85-103, 2007.
[9] L. Bottaci, "Instrumenting Programs with Flag Variables for Test Data Search by Genetic Algorithm," Proc. Genetic and Evolutionary Computation Conf., pp. 1337-1342, 2002.
[10] S. Bouktif, G. Antoniol, E. Merlo, and M. Neteler, "A Novel Approach to Optimize Clone Refactoring Activity," Proc. Genetic and Evolutionary Computation Conf., pp. 1885-1892, 2006.
[11] S. Bouktif, H. Sahraoui, and G. Antoniol, "Simulated Annealing for Improving Software Quality Prediction," Proc. Genetic and Evolutionary Computation Conf., pp. 1893-1900, 2006.
[12] L.C. Briand, Y. Labiche, and M. Shousha, "Stress Testing Real-Time Systems with Genetic Algorithms," Proc. Genetic and Evolutionary Computation Conf., pp. 1021-1028, 2005.
[13] British Standards Inst., "BS 7925-1 Vocabulary of Terms in Software Testing," 1998.
[14] G. Canfora, M.D. Penta, R. Esposito, and M.L. Villani, "An Approach for QoS-Aware Service Composition Based on Genetic Algorithms," Proc. Genetic and Evolutionary Computation Conf., pp. 1069-1075, 2005.
[15] K. Derderian, R. Hierons, M. Harman, and Q. Guo, "Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs," The Computer J., vol. 39, pp. 331-344, 2006.
[16] H. Do, S. Elbaum, and G. Rothermel, "Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact," Empirical Software Eng., vol. 10, no. 4, pp. 405-435, 2005.
[17] J.J. Dolado, "A Validation of the Component-Based Method for Software Size Estimation," IEEE Trans. Software Eng., vol. 26, no. 10, pp. 1006-1021, Oct. 2000.
[18] R. Ferguson and B. Korel, "The Chaining Approach for Software Test Data Generation," ACM Trans. Software Eng. and Methodology, vol. 5, no. 1, pp. 63-86, 1996.
[19] A. Finkelstein, M. Harman, A. Mansouri, J. Ren, and Y. Zhang, "Fairness Analysis in Requirements Assignments," Proc. IEEE Int'l Requirements Eng. Conf., 2008.
[20] R.L. Glass, Facts and Fallacies of Software Engineering. Addison Wesley, 2002.
[21] D. Greer and G. Ruhe, "Software Release Planning: An Evolutionary and Iterative Approach," Information and Software Technology, vol. 46, no. 4, pp. 243-253, 2004.
[22] M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. Int'l Conf. Future of Software Eng. 2007, pp. 342-357, 2007.
[23] M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, and J. Wegener, "The Impact of Input Domain Reduction on Search-Based Test Data Generation," Proc. ACM SIGSOFT Symp. Foundations of Software Eng., pp. 155-164, 2007.
[24] M. Harman, L. Hu, R. Hierons, J. Wegener, H. Sthamer, A. Baresel, and M. Roper, "Testability Transformation," IEEE Trans. Software Eng., vol. 30, no. 1, pp. 3-16, Jan. 2004.
[25] M. Harman and B. Jones, "Search-Based Software Engineering," Information and Software Technology, vol. 43, no. 14, pp. 833-839, 2001.
[26] M. Harman, K. Lakhotia, and P. McMinn, "A Multi-Objective Approach to Search-Based Test Data Generation," Proc. Genetic and Evolutionary Computation Conf., pp. 1098-1105, 2007.
[27] M. Harman and P. McMinn, "A Theoretical & Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation," Proc. Int'l Symp. Software Testing and Analysis, pp. 73-83, 2007.
[28] M. Harman and L. Tratt, "Pareto Optimal Search-Based Refactoring at the Design Level," Proc. Genetic and Evolutionary Computation Conf., pp. 1106-1113, 2007.
[29] J.H. Holland, Adaptation in Natural and Artificial Systems. Univ. of Michigan Press, 1975.
[30] Y. Jia and M. Harman, "Constructing Subtle Faults Using Higher Order Mutation Testing," Proc. Eighth Int'l Working Conf. Source Code Analysis and Manipulation, 2008.
[31] T. Jones, "Evolutionary Algorithms, Fitness Landscapes and Search," PhD thesis, Univ. of New Mexico, 1995.
[32] T.M. Khoshgoftaar, L. Yi, and N. Seliya, "A Multi-Objective Module-Order Model for Software Quality Enhancement," IEEE Trans. Evolutionary Computation, vol. 8, no. 6, pp. 593-608, Dec. 2004.
[33] B. Korel, "Automated Software Test Data Generation," IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.
[34] B. Korel, "Dynamic Method for Software Test Data Generation," Software Testing, Verification and Reliability, vol. 2, no. 4, pp. 203-213, 1992.
[35] K. Lakhotia, M. Harman, and P. McMinn, "Handling Dynamic Data Structures in Search-Based Testing," Proc. Genetic and Evolutionary Computation Conf., pp. 1759-1766, 2008.
[36] D. Leffingwell and D. Widrig, Managing Software Requirements: A Use Case Approach. Addison Wesley, 2003.
[37] K. Mahdavi, M. Harman, and R. Hierons, "A Multiple Hill Climbing Approach to Software Module Clustering," Proc. IEEE Int'l Conf. Software Maintenance, pp. 315-324, 2003.
[38] N. Mansour and M. Salame, "Data Generation for Path Testing," Software Quality J., vol. 12, no. 2, pp. 121-134, 2004.
[39] G. McGraw, C. Michael, and M. Schatz, "Generating Software Test Data by Evolution," IEEE Trans. Software Eng., vol. 27, no. 12, pp. 1085-1110, Dec. 2001.
[40] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[41] P. McMinn, "Search-Based Failure Discovery Using Testability Transformations to Generate Pseudo Oracles," Proc. Genetic and Evolutionary Computation Conf., pp. 1689-1696, 2009.
[42] P. McMinn, D. Binkley, and M. Harman, "Empirical Evaluation of a Nesting Testability Transformation for Evolutionary Testing," ACM Trans. Software Eng. Methodology, vol. 18, no. 3, 2009.
[43] P. McMinn, M. Harman, D. Binkley, and P. Tonella, "The Species Per Path Approach to Search-Based Test Data Generation," Proc. Int'l Symp. Software Testing and Analysis, pp. 13-24, 2006.
[44] P. McMinn and M. Holcombe, "Evolutionary Testing Using an Extended Chaining Approach," Evolutionary Computation, vol. 14, pp. 41-64, 2006.
[45] W. Miller and D. Spooner, "Automatic Generation of Floating-Point Test Data," IEEE Trans. Software Eng., vol. 2, no. 3, pp. 223-226, Sept. 1976.
[46] B.S. Mitchell and S. Mancoridis, "On the Automatic Modularization of Software Systems Using the Bunch Tool," IEEE Trans. Software Eng., vol. 32, no. 3, pp. 193-208, Mar. 2006.
[47] M. Mitchell, S. Forrest, and J.H. Holland, "The Royal Road for Genetic Algorithms: Fitness Landscapes and GA Performance," Proc. First European Conf. Artificial Life, pp. 245-254, 1992.
[48] H. Mühlenbein and D. Schlierkamp-Voosen, "Predictive Models for the Breeder Genetic Algorithm: I. Continuous Parameter Optimization," Evolutionary Computation, vol. 1, no. 1, pp. 25-49, 1993.
[49] National Institute of Standards and Technology, "The Economic Impacts of Inadequate Infrastructure for Software Testing," Planning Report 02-3 May 2002.
[50] M. O'Keeffe and M. O'Cinneide, "Search-Based Software Maintenance," Proc. Conf. Software Maintenance and ReEng., pp. 249-260, 2006.
[51] R. Pargas, M. Harrold, and R. Peck, "Test-Data Generation Using Genetic Algorithms," Software Testing, Verification and Reliability, vol. 9, no. 4, pp. 263-282, 1999.
[52] Radio Technical Commission for Aeronautics, "RTCA DO178-B Software Considerations in Airborne Systems and Equipment Certification," 1992.
[53] C.R. Reeves and J.E. Rowe, Genetic Algorithms—Principles and Perspectives: A Guide to GA Theory. Springer, 2002.
[54] O. Seng, J. Stammel, and D. Burkhart, "Search-Based Determination of Refactorings for Improving the Class Structure of Object-Oriented Systems," Proc. Genetic and Evolutionary Computation Conf., pp. 1909-1916, 2006.
[55] N. Tracey, J. Clark, K. Mander, and J. McDermid, "Automated Test Data Generation for Exception Conditions," Software— Practice and Experience, vol. 30, no. 1, pp. 61-79, 2000.
[56] H.-C. Wang and B. Jeng, "Structural Testing Using Memetic Algorithm," Proc. Second Taiwan Conf. Software Eng., 2006.
[57] J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing," Information and Software Technology, vol. 43, no. 14, pp. 841-854, 2001.
[58] J. Wegener, H. Sthamer, B.F. Jones, and D.E. Eyres, "Testing Real-Time Systems Using Genetic Algorithms," Software Quality J., vol. 6, no. 2, pp. 127-135, 1997.
[59] D. Whitley, "The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials Is Best," Proc. Int'l Conf. Genetic Algorithms, J.D. Schaffer, ed., pp. 116-121, 1989.
[60] S. Xanthakis, C. Ellis, C. Skourlas, A.L. Gall, S. Katsikas, and K. Karapoulios, "Application of Genetic Algorithms to Software Testing (Application Des Algorithms Génétiques Au Test Des Logiciels)," Proc. Fifth Int'l Conf. Software Eng. and Its Applications, pp. 625-636, 1992.
[61] M. Xiao, M. El-Attar, M. Reformat, and J. Miller, "Empirical Evaluation of Optimization Algorithms When Used in Goal-Oriented Automated Test Data Generation Techniques," Empirical Software Eng., vol. 12, no. 2, pp. 183-239, 2007.
15 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool