The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - March-April (2012 vol.38)
pp: 278-292
Gordon Fraser , Saarland University, Saarbrücken
Andreas Zeller , Saarland University, Saarbrücken
ABSTRACT
To assess the quality of test suites, mutation analysis seeds artificial defects (mutations) into programs; a nondetected mutation indicates a weakness in the test suite. We present an automated approach to generate unit tests that detect these mutations for object-oriented classes. This has two advantages: First, the resulting test suite is optimized toward finding defects modeled by mutation operators rather than covering code. Second, the state change caused by mutations induces oracles that precisely detect the mutants. Evaluated on 10 open source libraries, our μtest prototype generates test suites that find significantly more seeded defects than the original manually written test suites.
INDEX TERMS
Mutation analysis, test case generation, unit testing, test oracles, assertions, search-based testing.
CITATION
Gordon Fraser, Andreas Zeller, "Mutation-Driven Generation of Unit Tests and Oracles", IEEE Transactions on Software Engineering, vol.38, no. 2, pp. 278-292, March-April 2012, doi:10.1109/TSE.2011.93
REFERENCES
[1] S. Ali, L.C. Briand, H. Hemmati, and R.K. Panesar-Walawege, "A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation," IEEE Trans. Software Eng., vol. 36, no. 6, pp. 742-762, Nov./Dec. 2010.
[2] J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. 27th Int'l Conf. Software Eng., pp. 402-411, 2005.
[3] J.H. Andrews, A. Groce, M. Weston, and R.G. Xu, "Random Test Run Length and Effectiveness," Proc. IEEE/ACM 23rd Int'l Conf. Automated Software Eng., pp. 19-28, 2008.
[4] J.H. Andrews, S. Haldar, Y. Lei, and F.C.H. Li, "Tool Support for Randomized Unit Testing," Proc. First Int'l Workshop Random Testing, pp. 36-45, 2006.
[5] A. Arcuri, "It Does Matter How You Normalise the Branch Distance in Search Based Software Testing," Proc. Third Int'l Conf. Software Testing, Verification and Validation, pp. 205-214, 2010.
[6] A. Arcuri, "Longer Is Better: On the Role of Test Sequence Length in Software Testing," Proc. Third Int'l Conf. Software Testing, Verification and Validation, pp. 469-478, 2010.
[7] A. Arcuri and L. Briand, "A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering," Proc. IEEE Int'l Conf. Software Eng., pp. 1-10, 2011.
[8] A. Arcuri and X. Yao, "Search Based Software Testing of Object-Oriented Containers," Information Sciences, vol. 178, no. 15, pp. 3075-3095, 2008.
[9] K. Ayari, S. Bouktif, and G. Antoniol, "Automatic Mutation Test Input Data Generation via Ant Colony," Proc. Ninth Ann. Conf. Genetic and Evolutionary Computation, pp. 1074-1081, 2007.
[10] M. Boshernitsan, R. Doong, and A. Savoia, "From Daikon to Agitator: Lessons and Challenges in Building a Commercial Tool for Developer Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 169-180, 2006.
[11] L. Bottaci, "A Genetic Algorithm Fitness Function for Mutation Testing," Proc. Int'l Workshop Software Eng. Using Metaheuristic Inovative Algorithms, a Workshop at 23rd Int'l Conf. Software Eng., pp. 3-7, 2001.
[12] V. Chvatal, "A Greedy Heuristic for the Set-Covering Problem," Math. Operations Research, vol. 4, no. 3, pp. 233-235, 1979.
[13] I. Ciupa, A. Leitner, M. Oriol, and B. Meyer, "Artoo: Adaptive Random Testing for Object-Oriented Software," Proc. 30th Int'l Conf. Software Eng., pp. 71-80, 2008.
[14] C. Csallner and Y. Smaragdakis, "JCrasher: An Automatic Robustness Tester for Java," Software Practice and Experience, vol. 34, no. 11, pp. 1025-1050, 2004.
[15] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
[16] R.A. DeMillo and A.J. Offutt, "Constraint-Based Automatic Test Data Generation," IEEE Trans. Software Eng., vol. 17, no. 9, pp. 900-910, Sept. 1991.
[17] R.B. Evans and A. Savoia, "Differential Testing: A New Approach to Change Detection," Proc. Sixth Joint Meeting of the European Software Eng. Conf. and the ACM SIGSOFT Symp. Foundations of Software Eng., pp. 549-552, 2007.
[18] G. Fraser and A. Arcuri, "It Is Not the Length that Matters, It Is How You Control It," Proc. IEEE Fourth Int'l Conf. Software Testing, Verification and Validation, 2011.
[19] G. Fraser and A. Zeller, "Mutation-Driven Generation of Unit Tests and Oracles," Proc. 19th Int'l Symp. Software Testing and Analysis, pp. 147-158, 2010.
[20] P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 213-223, 2005.
[21] M. Harman, L. Hu, R. Hierons, J. Wegener, H. Sthamer, A. Baresel, and M. Roper, "Testability Transformation," IEEE Trans. Software Eng., vol. 30, no. 1, pp. 3-16, Jan. 2004.
[22] M. Harman and P. McMinn, "A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search," IEEE Trans. Software Eng., vol. 36, pp. 226-247, Mar./Apr. 2010.
[23] M.J. Harrold, R. Gupta, and M.L. Soffa, "A Methodology for Controlling the Size of a Test Suite," ACM Trans. Software Eng. Methodology, vol. 2, pp. 270-285, July 1993.
[24] K. Inkumsah and T. Xie, "Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution," Proc. IEEE/ACM 23rd Int'l Conf. Automated Software Eng., pp. 297-306, 2008.
[25] Y. Jia and M. Harman, "An Analysis and Survey of the Development of Mutation Testing," IEEE Trans. Software Eng., vol. 37, no. 5, pp. 649-678, Sept./Oct. 2011.
[26] B.F. Jones, D.E. Eyres, and H.-H. Sthamer, "A Strategy for Using Genetic Algorithms to Automate Branch and Fault-Based Testing," The Computer J., vol. 41, no. 2, pp. 98-107, 1998.
[27] Y. Lei and J.H. Andrews, "Minimization of Randomized Unit Test Cases," Proc. IEEE 16th Int'l Symp. Software Reliability Eng., pp. 267-276, 2005.
[28] A. Leitner, M. Oriol, A. Zeller, I. Ciupa, and B. Meyer, "Efficient Unit Test Case Minimization," Proc. IEEE/ACM 22nd Int'l Conf. Automated Software Eng., pp. 417-420, 2007.
[29] P. McMinn, "Search-Based Software Test Data Generation: A Survey: Research Articles," Software Testing Verification Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[30] A.J. Offutt, "Investigations of the Software Testing Coupling Effect," ACM Trans. Software Eng. Methodology, vol. 1, pp. 5-20, Jan. 1992.
[31] A.J. Offutt, Z. Jin, and J. Pan, "The Dynamic Domain Reduction Procedure for Test Data Generation," Software Practice and Experience, vol. 29, pp. 167-193, Feb. 1999.
[32] A.J. Offutt, A. Lee, G. Rothermel, R.H. Untch, and C. Zapf, "An Experimental Determination of Sufficient Mutant Operators," ACM Trans. Software Eng. and Methodology, vol. 5, pp. 99-118, Apr. 1996.
[33] A. Orso and T. Xie, "BERT: BEhavioral Regression Testing," Proc. Int'l Workshop Dynamic Analysis, pp. 36-42, July 2008.
[34] C. Pacheco and M.D. Ernst, "Eclat: Automatic Generation and Classification of Test Inputs," Proc. 19th European Conf. Object-Oriented Programming, pp. 504-527, July 2005.
[35] C. Pacheco and M.D. Ernst, "Randoop: Feedback-Directed Random Testing for Java," Companion to the 22nd ACM SIGPLAN Conf. Object-Oriented Programming Systems and Applications Companion, pp. 815-816, 2007.
[36] G. Rothermel, M.J. Harrold, J. Ostrin, and C. Hong, "An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites," Proc. Int'l Conf. Software Maintenance, pp. 34-43, 1998.
[37] A. Salcianu and M.C. Rinard, "Purity and Side Effect Analysis for Java Programs," Proc. Sixth Int'l Conf. Verification, Model Checking, and Abstract Interpretation, pp. 199-215, 2005.
[38] D. Schuler, V. Dallmeier, and A. Zeller, "Efficient Mutation Testing by Checking Invariant Violations," Proc. 18th Int'l Symp. Software Testing and Analysis, pp. 69-80, 2009.
[39] D. Schuler and A. Zeller, "Javalanche: Efficient Mutation Testing for Java," Proc. Seventh Joint Meeting of the European Software Eng. Conf. and the ACM SIGSOFT Symp. Foundations of Software Eng., pp. 297-298, 2009.
[40] D. Schuler and A. Zeller, "(Un-)Covering Equivalent Mutants," Proc. Third Int'l Conf. Software Testing, Verification, and Validation, pp. 45-54, 2010.
[41] K. Sen, D. Marinov, and G. Agha, "CUTE: A Concolic Unit Testing Engine for C," Proc. 10th European Software Eng. Conf. Held Jointly with 13th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 263-272, 2005.
[42] R. Sharma, M. Gligoric, A. Arcuri, G. Fraser, and D. Marinov, "Testing Container Classes: Random or Systematic?" Proc. 14th Int'l Conf. Fundamental Approaches to Software Eng., D. Giannakopoulou and F. Orejas, eds., pp. 262-277, 2011.
[43] A. Siami Namin, J.H. Andrews, and D.J. Murdoch, "Sufficient Mutation Operators for Measuring Test Effectiveness," Proc. 30th Int'l Conf. Software Eng., pp. 351-360, 2008.
[44] K. Taneja and T. Xie, "DiffGen: Automated Regression Unit-Test Generation," Proc. IEEE/ACM 23rd Int'l Conf. Automated Software Eng., pp. 407-410, 2008.
[45] N. Tillmann and J.N. de, "Halleux, Pex—White Box Test Generation for .NET," Proc. Int'l Conf. Tests and Proofs, pp. 134-253, 2008.
[46] P. Tonella, "Evolutionary Testing of Classes," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 119-128, 2004.
[47] A. Vargha and H.D. Delaney, "A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong," J. Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101-132, 2000.
[48] S. Wappler and F. Lammermann, "Using Evolutionary Algorithms for the Unit Testing of Object-Oriented Software," Proc. Conf. Genetic and Evolutionary Computation, pp. 1053-1060, 2005.
[49] N. Williams, B. Marre, P. Mouy, and M. Roger, "PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis," Proc. Fifth European Dependable Computing Conf., pp. 281-292, 2005.
[50] T. Xie, "Augmenting Automatically Generated Unit-Test Suites with Regression Oracle Checking," Proc. 20th European Conf. Object-Oriented Programming, pp. 380-403, July 2006.
[51] Z. Xu, M.B. Cohen, and G. Rothermel, "Factors Affecting the Use of Genetic Algorithms in Test Suite Augmentation," Proc. 12th Ann. Conf. Genetic and Evolutionary Computation, pp. 1365-1372, 2010.
[52] L. Zhang, T. Xie, L. Zhang, N. Tillmann, J. de Halleux, and H. Mei, "Test Generation via Dynamic Symbolic Execution for Mutation Testing," Proc. IEEE 26th Int'l Conf. Software Maintenance, Sept. 2010.
16 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool