The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.06 - November/December (2010 vol.36)
pp: 742-762
Shaukat Ali , Simula Research Laboratory, Lysaker and University of Oslo, Norway
Lionel C. Briand , Simula Research Laboratory, Lysaker and University of Oslo, Norway
Hadi Hemmati , Simula Research Laboratory, Lysaker and University of Oslo, Norway
Rajwinder K. Panesar-Walawege , Simula Research Laboratory, Lysaker and University of Oslo, Norway
ABSTRACT
Metaheuristic search techniques have been extensively used to automate the process of generating test cases, and thus providing solutions for a more cost-effective testing process. This approach to test automation, often coined “Search-based Software Testing” (SBST), has been used for a wide variety of test case generation purposes. Since SBST techniques are heuristic by nature, they must be empirically investigated in terms of how costly and effective they are at reaching their test objectives and whether they scale up to realistic development artifacts. However, approaches to empirically study SBST techniques have shown wide variation in the literature. This paper presents the results of a systematic, comprehensive review that aims at characterizing how empirical studies have been designed to investigate SBST cost-effectiveness and what empirical evidence is available in the literature regarding SBST cost-effectiveness and scalability. We also provide a framework that drives the data collection process of this systematic review and can be the starting point of guidelines on how SBST techniques can be empirically assessed. The intent is to aid future researchers doing empirical studies in SBST by providing an unbiased view of the body of empirical evidence and by guiding them in performing well-designed and executed empirical studies.
INDEX TERMS
Evolutionary computing and genetic algorithms, frameworks, heuristics design, review and evaluation, test generation, testing strategies, validation.
CITATION
Shaukat Ali, Lionel C. Briand, Hadi Hemmati, Rajwinder K. Panesar-Walawege, "A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation", IEEE Transactions on Software Engineering, vol.36, no. 6, pp. 742-762, November/December 2010, doi:10.1109/TSE.2009.52
REFERENCES
[1] "Computer Science Conference Ranking," http://www.cs- conference-ranking.org/ conferencerankingstopicsii.html., 2008.
[2] "Genetic Algorithms Framework," Rubicite Interactive, http://sourceforge.net/projectsga-fwork, 2004.
[3] "UML Profile for Modeling and Analysis of Real-Time and Embedded Systems (MARTE)," Object Management Group (OMG), http://www.omg.org/cgi-bin/doc?ptc2008-06-08 , 2008.
[4] W. Afzal, R. Torkar, and R. Feldt, "A Systematic Review of Search-Based Testing for Non-Functional System Properties," Information and Software Technology, vol. 51, pp. 957-976, 2009.
[5] S. Ali, L.C. Briand, H. Hemmati, and R.K. Panesar-Walawege, "A Systematic Review of the Application and Empirical Investigation of Evolutionary Testing," Technical Report Simula.SE.293, Simula Research Laboratory, 2008.
[6] B. Beizer, Software Testing Techniques. Van Nostrand Reinhold Co., 1990.
[7] A. Bertolino, "Software Testing Research: Achievements, Challenges, Dreams," Proc. 2007 Int'l Conf. Future of Software Eng., 2007.
[8] L.C. Briand, Y. Labiche, and M. Shousha, "Stress Testing Real-Time Systems with Genetic Algorithms," Proc. Genetic and Evolutionary Computation Conf., 2005.
[9] L.C. Briand, Y. Labiche, and M. Shousha, "Using Genetic Algorithms for Early Schedulability Analysis and Stress Testing in Real-Time Systems," Genetic Programming and Evolvable Machines, vol. 7, pp. 145-170, 2006.
[10] P.M.S. Bueno and M. Jino, "Identification of Potentially Infeasible Program Paths by Monitoring the Search for Test Data," Proc. 15th IEEE Int'l Conf. Automated Software Eng., pp. 209-218, 2000.
[11] P.M.S. Bueno, W.E. Wong, and M. Jino, "Improving Random Test Sets Using the Diversity Oriented Test Data Generation," Proc. Second Int'l Workshop Random Testing: Co-Located with the 22nd IEEE/ACM Int'l Conf. Automated Software Eng., 2007.
[12] E.K. Burke and G. Kendall, Search Methodologies: Introductory Tutorials in Optimization and Decision Support Techniques. Springer 2006.
[13] K.Y. Cai and D. Card, "An Analysis of Research Topics in Software Engineering—2006," J. Systems and Software, vol. 81, pp. 1051-1058, 2008.
[14] J.A. Capon, Elementary Statistics for the Social Sciences. Wadsworth Publishing Co., Inc., 1988.
[15] J. Clarke, J.J. Dolado, M. Harman, R. Hierons, B. Jones, M. Lumkin, B. Mitchell, S. Mancoridis, K. Rees, M. Roper, and M. Shepperd, "Reformulating Software Engineering as a Search Problem," IEE Software, vol. 150, pp. 161-175, 2003.
[16] D.A. Coley, An Introduction to Genetic Algorithms for Scientists and Engineers. World Scientific Publishing Company, 1997.
[17] R. Drechsler and N. Drechsler, Evolutionary Algorithms for Embedded System Design. Kluwer Academic Publishers, 2002.
[18] T. Dyba, V.B. Kampenes, and D.I.K. Sjoberg, "A Systematic Review of Statistical Power in Software Engineering Experiments," Information and Software Technology, vol. 48, pp. 745-755, 2006.
[19] T. Dyba, B.A. Kitchenham, and M. Jorgensen, "Evidence-Based Software Engineering for Practitioners," IEEE Software, vol. 22, no. 1, pp. 58-65, Jan./Feb. 2005.
[20] S.A. Ghazi and M.A. Ahmed, "Pair-Wise Test Coverage Using Genetic Algorithms," Proc. 2003 Congress on Evolutionary Computation, pp. 1420-1424, 2003.
[21] M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. 2007 Int'l Conf. Future of Software Eng., 2007.
[22] M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, and J. Wegener, "The Impact of Input Domain Reduction on Search-Based Test Data Generation," Proc. Sixth Joint Meeting of the European Software Eng. Conf. and the ACM SIGSOFT Symp. Foundations of Software Eng., 2007.
[23] M. Harman and B.F. Jones, "Search-Based Software Engineering," Information and Software Technology, vol. 43, pp. 833-839, 2001.
[24] M. Harman and P. McMinn, "A Theoretical Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation," Proc. 2007 Int'l Symp. Software Testing and Analysis, 2007.
[25] C. Hart, Doing a Literature Review: Releasing the Social Science Research Imagination. Sage Publications, Ltd., 1999.
[26] R.L. Haupt and S.E. Haupt, Practical Genetic Algorithms. Wiley-Interscience, 1997.
[27] D. Johnson, "A Theoretician's Guide to the Experimental Analysis of Algorithms," Proc. Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, pp. 215-250, 2002.
[28] B.F. Jones, H.H. Sthamer, and D.E. Eyres, "Automatic Structural Testing Using Genetic Algorithms," Software Eng. J., vol. 11, pp. 299-306, 1996.
[29] K.S. Khan, R. Kunz, J. Kleijnen, and G. Antes, Systematic Review to Support Evidence-Based Medicine: How to Review and Apply Findings of Healthcare Research. Royal Soc. of Medicine Press, Ltd., 2003.
[30] B.A. Kitchenham, "Guidelines for Performing Systematic Literature Reviews in Software Engineering," Technical Report EBSE-2007-01, 2007.
[31] B.A. Kitchenham, T. Dyba, and M. Jorgensen, "Evidence-Based Software Engineering," Proc. 26th Int'l Conf. Software Eng., 2004.
[32] B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg, "Preliminary Guidelines for Empirical Research in Software Engineering," IEEE Trans. Software Eng., vol. 28, no. 8, pp. 721-734, Aug. 2002.
[33] K. Lakhotia, M. Harman, and P. McMinn, "A Multi-Objective Approach to Search-Based Test Data Generation," Proc. Genetic and Evolutionary Computation Conf., 2007.
[34] S. Luke, L. Panait, G. Balan, S. Paus, Z. Skolicki, E. Popovici, K. Sullivan, J. Harrison, J. Bassett, R. Hubley, and A. Chircop, "A Java-Based Evolutionary Computation Research System," George Mason University's ECLab Evolutionary Computation Laboratory, http://www.cs.gmu.edu/~eclab/projectsecj /, 2007.
[35] T. Mantere and J.T. Alander, "Evolutionary Software Engineering, a Review," Applied Soft Computing, vol. 5, pp. 315-331, 2005.
[36] A.P. Mathur, Foundations of Software Testing. Pearson Education, 2008.
[37] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, pp. 105-156, 2004.
[38] C.C. Michael, G. McGraw, and M.A. Schatz, "Generating Software Test Data by Evolution," IEEE Trans. Software Eng., vol. 27, no. 12, pp. 1085-1110, Dec. 2001.
[39] J. Miller, M. Reformat, and H. Zhang, "Automatic Test Data Generation Using Genetic Algorithm and Program Dependence Graphs," Information and Software Technology, vol. 48, pp. 586-605, 2006.
[40] D.S. Moore and G.P. McCabe, Introduction to the Practice of Statistics, fourth ed. W.H. Freeman, 2002.
[41] H. Osman and J.P. Kelly, Metaheuristics: Theory and Applications. Kluwer Academic Publishers, 1996.
[42] H. Pohlheim, "GEATbx—The Genetic and Evolutionary Algorithm Toolbox for Matlab," 2007.
[43] P. Puschner and R. Nossal, "Testing the Results of Static Worst-Case Execution-Time Analysis," Proc. 19th IEEE Real-Time Systems Symp., pp. 134-143, 1998.
[44] D.J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures, third ed. Chapman & Hall/CRC, 2003.
[45] T. Shiba, T. Tsuchiya, and T. Kikuno, "Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing," Proc. 28th Ann. Int'l Computer Software and Applications Conf., pp. 72-77, 2004.
[46] M. Srinivas and L.M. Patnaik, "Genetic Algorithms: A Survey," Computer, vol. 27, no. 6, pp. 17-26, June 1994.
[47] N. Tracey, J. Clark, and K. Mander, "Automated Program Flaw Finding Using Simulated Annealing," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, 1998.
[48] R.H. Untch, A.J. Offutt, and M.J. Harrold, "Mutation Analysis Using Mutant Schemata," Proc. 1993 ACM SIGSOFT Int'l Symp. Software Testing and Analysis, 1993.
[49] S. Wappler and I. Schieferdecker, "Improving Evolutionary Class Testing in the Presence of Non-Public Methods," Proc. 22nd IEEE/ACM Int'l Conf. Automated Software Eng., 2007.
[50] A. Watkins and E.M. Hufnagel, "Evolutionary Test Data Generation: A Comparison of Fitness Functions," Software: Practice and Experience, vol. 36, pp. 95-116, 2006.
[51] J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing," Information and Software Technology, vol. 43, pp. 841-854, 2001.
[52] C. Wohlin, P. Runeson, M. Host, M.C. Ohlsson, B. Regnell, and A. Wesslen, Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, 2000.
[53] M. Xiao, M. El-Attar, M. Reformat, and J. Miller, "Empirical Evaluation of Optimization Algorithms When Used in Goal-Oriented Automated Test Data Generation Techniques," Empirical Software Eng., vol. 12, pp. 183-239, 2007.
[54] Y. Zhan and J.A. Clark, "Search-Based Mutation Testing for Simulink Models," Proc. Genetic and Evolutionary Computation Conf., 2005.
[55] Y. Zhan and J.A. Clark, "The State Problem for Test Generation in Simulink," Proc. Genetic and Evolutionary Computation Conf., 2006.
17 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool