The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - Feb. (2013 vol.39)
pp: 276-291
Gordon Fraser , Saarland University, Saarbrücken
Andrea Arcuri , Simula Research Laboratory, Lysaker
ABSTRACT
Not all bugs lead to program crashes, and not always is there a formal specification to check the correctness of a software test's outcome. A common scenario in software testing is therefore that test data are generated, and a tester manually adds test oracles. As this is a difficult task, it is important to produce small yet representative test sets, and this representativeness is typically measured using code coverage. There is, however, a fundamental problem with the common approach of targeting one coverage goal at a time: Coverage goals are not independent, not equally difficult, and sometimes infeasible—the result of test generation is therefore dependent on the order of coverage goals and how many of them are feasible. To overcome this problem, we propose a novel paradigm in which whole test suites are evolved with the aim of covering all coverage goals at the same time while keeping the total size as small as possible. This approach has several advantages, as for example, its effectiveness is not affected by the number of infeasible targets in the code. We have implemented this novel approach in the EvoSuite tool, and compared it to the common approach of addressing one goal at a time. Evaluated on open source libraries and an industrial case study for a total of 1,741 classes, we show that EvoSuite achieved up to 188 times the branch coverage of a traditional approach targeting single branches, with up to 62 percent smaller test suites.
INDEX TERMS
Software, Genetic algorithms, Search problems, Arrays, Genetic programming, Software testing, collateral coverage, Search-based software engineering, length, branch coverage, genetic algorithm, infeasible goal
CITATION
Gordon Fraser, Andrea Arcuri, "Whole Test Suite Generation", IEEE Transactions on Software Engineering, vol.39, no. 2, pp. 276-291, Feb. 2013, doi:10.1109/TSE.2012.14
REFERENCES
[1] S. Ali, L. Briand, H. Hemmati, and R. Panesar-Walawege, "A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation," IEEE Trans. Software Eng., vol. 36, no. 6, pp. 742-762, Nov./Dec. 2010.
[2] M. Alshraideh and L. Bottaci, "Search-Based Software Test Data Generation for String Data Using Program-Specific Search Operators: Research Articles," Software Testing, Verification, and Reliability, vol. 16, no. 3, pp. 175-203, 2006.
[3] L. Araujo and J. Merelo, "Diversity through Multiculturality: Assessing Migrant Choice Policies in an Island Model," IEEE Trans. Evolutionary Computation, vol. 15, no. 4, pp. 1-14, Aug. 2011.
[4] A. Arcuri, "It Really Does Matter How You Normalize the Branch Distance in Search-Based Software Testing," Software Testing, Verification and Reliability, http://dx.doi.org/10.1002stvr.457, 2011.
[5] A. Arcuri, "A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage," IEEE Trans. Software Eng., vol. 38, no. 3, pp. 497-519, May/June 2011.
[6] A. Arcuri and L. Briand, "Adaptive Random Testing: An Illusion of Effectiveness?" Proc. ACM Int'l Symp. Software Testing and Analysis, 2011.
[7] A. Arcuri and L. Briand, "A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering," Proc. 33rd Int'l Conf. Software Eng., pp. 1-10. 2011,
[8] A. Arcuri and G. Fraser, "On Parameter Tuning in Search Based Software Engineering," Proc. Third Int'l Conf. Search Based Software Eng., pp. 33-47, 2011.
[9] A. Arcuri, M.Z. Iqbal, and L. Briand, "Black-Box System Testing of Real-Time Embedded Systems Using Random and Search-Based Testing," Proc. 22nd IFIP Int'l Conf. Testing Software and Systems, pp. 95-110, 2010,
[10] A. Arcuri, M.Z. Iqbal, and L. Briand, "Random Testing: Theoretical Results and Practical Implications," IEEE Trans. Software Eng., vol. 38, no. 2,http://doi.ieeecomputersoc..org/10.1109TSE.2011.121 , pp. 258-277, Mar./Apr. 2011.
[11] A. Arcuri and X. Yao, "Search Based Software Testing of Object-Oriented Containers," Information Sciences, vol. 178, no. 15, pp. 3075-3095, 2008.
[12] A. Baars, M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, P. Tonella, and T. Vos, "Symbolic Search-Based Testing," Proc. IEEE/ACM 26th Int'l Conf. Automated Software Eng., 2011.
[13] L. Baresi, P.L. Lanzi, and M. Miraz, "Testful: An Evolutionary Test Approach for Java," Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 185-194, 2010.
[14] B. Baudry, F. Fleurey, J.-M. Jézéquel, and Y. Le Traon, "Automatic Test Cases Optimization: A Bacteriologic Algorithm," IEEE Software, vol. 22, no. 2, pp. 76-82, Mar./Apr. 2005.
[15] C. Csallner and Y. Smaragdakis, "JCrasher: An Automatic Robustness Tester for Java," Software Practice and Experience, vol. 34, pp. 1025-1050, 2004.
[16] W. Feller, An Introduction to Probability Theory and Its Applications, vol. 1, third ed. Wiley, 1968.
[17] G. Fraser and A. Arcuri, "Evolutionary Generation of Whole Test Suites," Proc. 11th Int'l Conf. Quality Software, pp. 31-40, 2011.
[18] G. Fraser and A. Arcuri, "Evosuite: Automatic Test Suite Generation for Object-Oriented Software," Proc. 19th ACM SIGSOFT Symp. and the 13th European Conf. Foundations of Software Eng., 2011.
[19] G. Fraser and A. Arcuri, "It Is Not the Length That Matters, It Is How You Control it," Proc. Fourth IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 150-159, 2011.
[20] G. Fraser and A. Zeller, "Exploiting Common Object Usage in Test Case Generation," Proc. Fourth IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 80-89, 2011.
[21] G. Fraser and A. Zeller, "Mutation-Driven Generation of Unit Tests and Oracles," IEEE Trans. Software Eng., vol. 38, no. 2, pp. 278-292, Mar./Apr. 2011.
[22] P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 213-223, 2005.
[23] A. Goldberg, T.C. Wang, and D. Zimmerman, "Applications of Feasible Path Analysis to Program Testing," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 80-94, 1994.
[24] M. Harman, L. Hu, R. Hierons, J. Wegener, H. Sthamer, A. Baresel, and M. Roper, "Testability Transformation," IEEE Trans. Software Eng., vol. 30, no. 1, pp. 3-16, Jan. 2004.
[25] M. Harman, S.G. Kim, K. Lakhotia, P. McMinn, and S. Yoo, "Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem," Proc. Third Int'l Conf. Software Testing, Verification, and Validation Workshops, 2010.
[26] M. Harman and P. McMinn, "A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search," IEEE Trans. Software Eng., vol. 36, no. 2, pp. 226-247, Mar./Apr. 2010.
[27] K. Inkumsah and T. Xie, "Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution," Proc. 23rd IEEE/ACM Int'l Conf. Automated Software Eng., pp. 297-306, 2008.
[28] M. Islam and C. Csallner, "Dsc+Mock: A Test Case + Mock Class Generator in Support of Coding against Interfaces," Proc. Eighth Int'l Workshop Dynamic Analysis, pp. 26-31, 2010.
[29] Y. Jia and M. Harman, "An Analysis and Survey of the Development of Mutation Testing," Technical Report TR-09-06, CREST Centre, King's College London, Sept. 2009.
[30] B. Korel, "Automated Software Test Data Generation," IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.
[31] F. Lammermann, A. Baresel, and J. Wegener, "Evaluating Evolutionary Testability for Structure-Oriented Testing with Software Measurements," Applied Soft Computing, vol. 8, no. 2, pp. 1018-1028, 2008.
[32] F. Lobo, C. Lima, and Z. Michalewicz, Parameter Setting in Evolutionary Algorithms. Springer, 2007.
[33] J. Malburg and G. Fraser, "Combining Search-Based and Constraint-Based Testing," Proc. IEEE/ACM 26th Int'l Conf. Automated Software Eng., 2011.
[34] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[35] W. Miller and D.L. Spooner, "Automatic Generation of Floating-Point Test Data," IEEE Trans. Software Eng., vol. 2, no. 3, pp. 223-226, May 1976.
[36] C. Pacheco and M.D. Ernst, "Randoop: Feedback-Directed Random Testing for Java," Proc. Companion to the 22nd ACM SIGPLAN Conf. Object-Oriented Programming Systems and Application, pp. 815-816, 2007.
[37] J.C.B. Ribeiro, "Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming," Proc. GECCO Conf. Companion on Genetic and Evolutionary Computation, pp. 1819-1822, 2008.
[38] G. Rudolph, "Convergence Analysis of Canonical Genetic Algorithms," IEEE Trans. Neural Networks, vol. 5, no. 1, pp. 96-101, Jan. 1994.
[39] K. Sen, D. Marinov, and G. Agha, "CUTE: A Concolic Unit Testing Engine for C," Proc. 10th European Software Eng. Conf. Held Jointly with 13th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 263-272, 2005.
[40] R. Sharma, M. Gligoric, A. Arcuri, G. Fraser, and D. Marinov, "Testing Container Classes: Random or Systematic?" Proc. 14th Int'l Conf. Fundamental Approaches to Software Eng.: Part of the Joint European Conf. Theory and Practice of Software Fundamental Approaches to Software Eng., 2011.
[41] S. Silva and E. Costa, "Dynamic Limits for Bloat Control in Genetic Programming and a Review of Past and Current Bloat Theories," Genetic Programming and Evolvable Machines, vol. 10, no. 2, pp. 141-179, 2009.
[42] N. Tillmann and J.N. de Halleux, "Pex—White Box Test Generation for .NET," Proc. Int'l Conf. Tests and Proofs, pp. 134-253, 2008.
[43] P. Tonella, "Evolutionary Testing of Classes," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 119-128, 2004.
[44] A. Vargha and H.D. Delaney, "A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong," J. Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101-132, 2000.
[45] W. Visser, C.S. Pasareanu, and S. Khurshid, "Test Input Generation with Java PathFinder," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 97-107, 2004.
[46] S. Wappler and I. Schieferdecker, "Improving Evolutionary Class Testing in the Presence of Non-Public Methods," Proc. IEEE/ACM 22nd Int'l Conf. Automated Software Eng., pp. 381-384, 2007.
[47] S. Wappler and F. Lammermann, "Using Evolutionary Algorithms for the Unit Testing of Object-Oriented Software," Proc. Conf. Genetic and Evolutionary Computation, pp. 1053-1060, 2005.
[48] J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing," Information and Software Technology, vol. 43, no. 14, pp. 841-854, 2001.
[49] D. Whitley, "The Genitor Algorithm and Selective Pressure: Why Rank-Based Allocation of Reproductive Trials Is Best," Proc. Third Int'l Conf. Genetic Algorithms, pp. 116-121, 1989.
[50] N. Williams, B. Marre, P. Mouy, and M. Roger, "PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis," Proc. Fifth European Dependable Computing Conf., pp. 281-292, 2005.
[51] T. Xie, D. Marinov, W. Schulte, and D. Notkin, "Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution," Proc. 11th Int'l Conf. Tools and Algorithms for the Construction and Analysis of Systems, pp. 365-381, 2005.
[52] S. Zhang, D. Saff, Y. Bu, and M. Ernst, "Combined Static and Dynamic Automated Test Generation," Proc. ACM Int'l Symp. Software Testing and Analysis, 2011.
26 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool