The Community for Technology Leaders
RSS Icon
Issue No.02 - March-April (2012 vol.38)
pp: 453-477
Phil McMinn , University of Sheffield, Sheffield
Mark Harman , University College London, London
Kiran Lakhotia , University College London, London
Youssef Hassoun , King's College London, London
Joachim Wegener , Berner & Mattner Systemtechnik GmbH, Berlin
Search-Based Test Data Generation reformulates testing goals as fitness functions so that test input generation can be automated by some chosen search-based optimization algorithm. The optimization algorithm searches the space of potential inputs, seeking those that are “fit for purpose,” guided by the fitness function. The search space of potential inputs can be very large, even for very small systems under test. Its size is, of course, a key determining factor affecting the performance of any search-based approach. However, despite the large volume of work on Search-Based Software Testing, the literature contains little that concerns the performance impact of search space reduction. This paper proposes a static dependence analysis derived from program slicing that can be used to support search space reduction. The paper presents both a theoretical and empirical analysis of the application of this approach to open source and industrial production code. The results provide evidence to support the claim that input domain reduction has a significant effect on the performance of local, global, and hybrid search, while a purely random search is unaffected.
Search-based software testing, evolutionary testing, automated test data generation, input domain reduction.
Phil McMinn, Mark Harman, Kiran Lakhotia, Youssef Hassoun, Joachim Wegener, "Input Domain Reduction through Irrelevant Variable Removal and Its Effect on Local, Global, and Hybrid Search-Based Structural Test Data Generation", IEEE Transactions on Software Engineering, vol.38, no. 2, pp. 453-477, March-April 2012, doi:10.1109/TSE.2011.18
[1] The Software-Artifact Infrastructure Repository, http://sir.unl. edu/portalindex.html, 2011.
[2] W. Afzal, R. Torkar, and R. Feldt, "A Systematic Review of Search-Based Testing for Non-Functional System Properties," Information and Software Technology, vol. 51, no. 6, pp. 957-976, 2009.
[3] S. Ali, L.C. Briand, H. Hemmati, and R.K. Panesar-Walawege, "A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation," IEEE Trans. Software Eng., vol. 36, no. 6, pp. 742-762, Nov./Dec. 2010.
[4] L.O. Andersen, "Program Analysis and Specialization for the C Programming Language," PhD thesis, DIKU, Univ. of Copenhagen, May 1994.
[5] A. Arcuri, "Theoretical Analysis of Local Search in Software Testing," Proc. Symp. Stochastic Algorithms, Foundations and Applications, pp. 156-168, 2009.
[6] A. Arcuri, "It Does Matter How You Normalise the Branch Distance in Search Based Software Testing," Proc. Int'l Conf. Software Testing, Verification and Validation, pp. 205-214, 2010.
[7] A. Arcuri, M.Z. Iqbal, and L. Briand, "Formal Analysis of the Effectiveness and Predictability of Random Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 219-230, 2010.
[8] J.E. Baker, "Reducing Bias and Inefficiency in the Selection Algorithm," Proc. Second Int'l Conf. Genetic Algorithms and Their Application, 1987.
[9] D.W. Binkley, N. Gold, and M. Harman, "An Empirical Study of Static Program Slice Size," ACM Trans. Software Eng. and Methodology, vol. 16, no. 2, pp. 1-32, 2007.
[10] L.C. Briand, J. Feng, and Y. Labiche, "Using Genetic Algorithms and Coupling Measures to Devise Optimal Integration Test Orders," Proc. 14th Int'l Conf. Software Eng. and Knowledge Eng., pp. 43-50, 2002.
[11] O. Buehler and J. Wegener, "Evolutionary Functional Testing of an Automated Parking System," Proc. Int'l Conf. Computer, Comm. and Control Technologies and Ninth Int'l Conf. Information Systems Analysis and Synthesis, 2003.
[12] S. Chen and S. Smith, "Improving Genetic Algorithms by Search Space Reductions," Proc. Genetic and Evolutionary Computation Conf., pp. 135-140, 1999.
[13] P. Cousot and R. Cousot, "Abstract Interpretation Frameworks," J. Logic and Computation, vol. 2, no. 4, pp. 511-547, 1992.
[14] H. Do, S. Elbaum, and G. Rothermel, "Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact," Empirical Software Eng., vol. 10, no. 4, pp. 405-435, 2005.
[15] J.W. Duran and S.C. Ntafos, "An Evaluation of Random Testing," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.
[16] Grammatech Inc. "The Codesurfer Slicing System," 2002.
[17] M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. Future of Software Eng., L. Briand and A. Wolf, eds., pp. 342-357, 2007.
[18] M. Harman, C. Fox, R.M. Hierons, L. Hu, S. Danicic, and J. Wegener, "Vada: A Transformation-Based System for Variable Dependence Analysis," Proc. IEEE Int'l Workshop Source Code Analysis and Manipulation, pp. 55-64, Oct. 2002.
[19] M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, and J. Wegener, "The Impact of Input Domain Reduction on Search-Based Test Data Generation," Proc. ACM SIGSOFT Symp. Foundations of Software Eng., pp. 155-164, 2007.
[20] M. Harman, F. Islam, T. Xie, and S. Wappler, "Automated Test Data Generation for Aspect-Oriented Programs," Proc. Eighth Int'l Conf. Aspect-Oriented Software Development, pp. 185-196, 2009.
[21] M. Harman, S.G. Kim, K. Lakhotia, P. McMinn, and S. Yoo, "Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem," Proc. Third Int'l Workshop Search-Based Testing, 2010.
[22] M. Harman, A. Mansouri, and Y. Zhang, "Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications," Technical Report TR-09-03, Dept. of Computer Science, King's College London, Apr. 2009.
[23] M. Harman and P. McMinn, "A Theoretical and Empirical Study of Search-Based Testing: Local, Global and Hybrid Search," IEEE Trans. Software Eng., vol. 36, no. 2, pp. 226-247, Mar./Apr. 2010.
[24] S. Horwitz, T. Reps, and D.W. Binkley, "Interprocedural Slicing Using Dependence Graphs," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 35-46, June 1988.
[25] J. King, "Symbolic Execution and Program Testing," Comm. ACM, vol. 19, no. 7, pp. 385-394, 1976.
[26] B. Korel, "Automated Software Test Data Generation," IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.
[27] J. Krinke and G. Snelting, "Validation of Measurement Software as an Application of Slicing and Constraint Solving," Information and Software Technology, special issue on program slicing, vol. 40, nos. 11/12, pp. 661-675, 1998.
[28] K. Lakhotia, P. McMinn, and M. Harman, "An Empirical Investigation into Branch Coverage for C Programs Using CUTE and AUSTIN," J. Systems and Software, vol. 83, pp. 2379-2391, 2010.
[29] F. Lammermann, A. Baresel, and J. Wegener, "Evaluating Evolutionary Testability with Software-Measurements," Proc. Genetic and Evolutionary Computation Conf., pp. 1350-1362, 2004.
[30] P. Larrañaga and J.A. Lozano, Estimation of Distribution Algorithms. Kluwer, 2002.
[31] N. Mansour and M. Salame, "Data Generation for Path Testing," Software Quality J., vol. 12, no. 2, pp. 121-134, 2004.
[32] G. McGraw, C. Michael, and M. Schatz, "Generating Software Test Data by Evolution," IEEE Trans. Software Eng., vol. 27, no. 12, pp. 1085-1110, Dec. 2001.
[33] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[34] P. McMinn, M. Harman, D. Binkley, and P. Tonella, "The Species per Path Approach to Search-Based Test Data Generation," Proc. Int'l Symp. Software Testing and Analysis, pp. 13-24, 2006.
[35] P. McMinn, M. Stevenson, and M. Harman, "Reducing Qualitative Human Oracle Costs Associated with Automatically Generated Test Data," Proc. First Int'l Workshop Software Test Output Validation, pp. 1-4, 2010.
[36] W. Miller and D. Spooner, "Automatic Generation of Floating-Point Test Data," IEEE Trans. Software Eng., vol. 2, no. 3, pp. 223-226, Sept. 1976.
[37] H. Mühlenbein and D. Schlierkamp-Voosen, "Predictive Models for the Breeder Genetic Algorithm: I. Continuous Parameter Optimization," Evolutionary Computation, vol. 1, no. 1, pp. 25-49, 1993.
[38] R. Pargas, M. Harrold, and R. Peck, "Test-Data Generation Using Genetic Algorithms," Software Testing, Verification and Reliability, vol. 9, no. 4, pp. 263-282, 1999.
[39] R Development Core Team, R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2009.
[40] O. Räihä, "A Survey on Search Based Software Design," Technical Report D-2009-1, Dept. of Computer Sciences, Univ. of Tampere, 2009.
[41] J. Ribeiro, M. Zenha-Rela, and F. de Vega, "Test Case Evaluation and Input Domain Reduction Strategies for the Evolutionary Testing of Object-Oriented Software," Information and Software Technology, vol. 51, pp. 1534-1548, 2009.
[42] R. Sagarna and J. Lozano, "Dynamic Search Space Transformations for Software Test Data Generation," Computational Intelligence, vol. 24, no. 1, pp. 23-61, 2008.
[43] F. Schmiedle, R. Drechsler, and B. Becker, "Exact Routing with Search Space Reduction," IEEE Trans. Computers, vol. 52, no. 6, pp. 815-825, June 2003.
[44] B. Steensgaard, "Points-to Analysis in Almost Linear Time," Proc. Conf. Record of the 23rd ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages, pp. 32-41, Jan. 1996.
[45] N. Tracey, J. Clark, K. Mander, and J. McDermid, "An Automated Framework for Structural Test-Data Generation," Proc. Int'l Conf. Automated Software Eng., pp. 285-288, 1998.
[46] J. Voas and K. Miller, "Software Testability: The New Verification," IEEE Software, vol. 12, no. 3, pp. 17-28, May 1995.
[47] H.-C. Wang and B. Jeng, "Structural Testing Using Memetic Algorithm," Proc. Second Taiwan Conf. Software Eng., 2006.
[48] J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing," Information and Software Technology, vol. 43, no. 14, pp. 841-854, 2001.
[49] J. Wegener and M. Grochtmann, "Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing," Real-Time Systems, vol. 15, no. 3, pp. 275-298, 1998.
[50] M. Weiser, "Program Slicing," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 352-357, July 1984.
[51] D. Whitley, "The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials Is Best," Proc. Int'l Conf. Genetic Algorithms, J.D. Schaffer, ed., pp. 116-121, 1989.
[52] S. Xanthakis, C. Ellis, C. Skourlas, A. Le Gall, S. Katsikas, and K. Karapoulios, "Application of Genetic Algorithms to Software Testing (Application des Algorithmes Génétiques au Test des Logiciels)," Proc. Fifth Int'l Conf. Software Eng. and Its Applications, pp. 625-636, 1992.
24 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool