The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.04 - April (2013 vol.39)
pp: 570-587
Pedro Reales Mateo , University of Castilla-La Mancha, Ciudad Real
Macario Polo Usaola , University of Castilla-La Mancha, Ciudad Real
José Luis Fernández Alemán , University of Murcia, Murcia
ABSTRACT
Mutation has been recognized to be an effective software testing technique. It is based on the insertion of artificial faults in the system under test (SUT) by means of a set of mutation operators. Different operators can mutate each program statement in several ways, which may produce a huge number of mutants. This leads to very high costs for test case execution and result analysis. Several works have approached techniques for cost reduction in mutation testing, like $(n)$-order mutation where each mutant contains $(n)$ artificial faults instead of one. There are two approaches to $(n)$-order mutation: increasing the effectiveness of mutation by searching for good $(n)$-order mutants, and decreasing the costs of mutation testing by reducing the mutants set through the combination of the first-order mutants into $(n)$-order mutants. This paper is focused on the second approach. However, this second use entails a risk: the possibility of leaving undiscovered faults in the SUT, which may distort the perception of the test suite quality. This paper describes an empirical study of different combination strategies to compose second-order mutants at system level as well as a cost-risk analysis of $(n)$-order mutation at system level.
INDEX TERMS
Algorithm design and analysis, Concrete, Educational institutions, Benchmark testing, Optimization, Software testing, mutation testing, Empirical evaluation, high-order mutation
CITATION
Pedro Reales Mateo, Macario Polo Usaola, José Luis Fernández Alemán, "Validating Second-Order Mutation at System Level", IEEE Transactions on Software Engineering, vol.39, no. 4, pp. 570-587, April 2013, doi:10.1109/TSE.2012.39
REFERENCES
[1] Y. Jia and M. Harman, "Higher Order Mutation Testing," Information and Software Technology, vol. 51, no. 10, pp. 1379-1393, 2009.
[2] P. Ammann and J. Offutt, Introduction to Software Testing. Cambridge Univ. Press, 2008.
[3] M. Polo, M. Piattini, and I. García-Rodríguez, "Decreasing the Cost of Mutation Testing with Second-Order Mutants," Software Testing, Verification, and Reliability, vol. 19, no. 2, pp. 111-131, 2008.
[4] A.J. Offutt and J. Pan, "Automatically Detecting Equivalent Mutants and Infeasible Paths," Software Testing, Verification, and Reliability, vol. 7, no. 3, pp. 165-192, 1997.
[5] D. Schuler and A. Zeller, "(Un-)Covering Equivalent Mutants," Proc. Third Int'l Conf. Software Testing Verification and Validation, Apr. 2010.
[6] B.J.M. Grün, D. Schuler, and A. Zeller, "The Impact of Equivalent Mutants," Proc. IEEE Int'l Conf. Software Testing, Verification, and Validation Workshops, pp. 192-199, Apr. 2009.
[7] M. Polo and P. Reales, "Mutation Testing Cost Reduction Techniques: A Survey," IEEE Software, vol. 27, no. 3, pp. 80-86, May/June 2010.
[8] Y. Jia and M. Harman, "Constructing Subtle Faults Using Higher Order Mutation Testing," Proc. Eighth Int'l Working Conf. Source Code Analysis and Manipulation, pp. 28-29, Sept. 2008.
[9] M. Papadakis, N. Malevris, and M. Kintis, "Mutation Testing Strategies: A Collateral Approach," Proc. Fifth Int'l Conf. Software and Data Technologies, pp. 325-328, June 2011.
[10] M. Harman, Y. Jia, and W. Langdon, "A Manifesto for Higher Order Mutation Testing," Proc. Third Int'l Conf. Software Testing, Verification, and Validation Workshops, pp. 80-89, 2010.
[11] M. Papadakis and N. Malevris, "An Empirical Evaluation of the First and Second Order Mutation Testing Strategies," Proc. Software Testing, Verification, and Validation Workshops, pp. 90-99, Apr. 2010.
[12] P. Reales, M. Polo, and J. Offutt, "Mutation at System and Functional Levels," Proc. Third Int'l Conf. Software Testing, Verification, and Validation Workshops, pp. 110-119, Apr. 2010.
[13] T.A. Budd and D. Angluin, "Two Notions of Correctness and Their Relation to Testing," Acta Informatica, vol. 18, no. 1, pp. 31-45, Mar. 1982.
[14] D. Baldwin and F.G. Sayward, "Heuristics for Determining Equivalence of Program Mutations," report, 1979.
[15] A. Offutt and J. Pan, "Detecting Equivalent Mutants and the Feasible Path Problem," Wiley's Software Testing, Verification, and Reliability, vol. 7, no. 3, pp. 165-192, Sept. 1997.
[16] R. Hierons, M. Harman, and S. Danicic, "Using Program Slicing to Assist in the Detection of Equivalent Mutants," Software Testing, Verification, and Reliability, vol. 9, no. 4, pp. 233-262, Dec. 1999.
[17] E. Mresa and L. Bottaci, "Efficiency of Mutation Operators and Selective Mutation Strategies: An Empirical Study," Software Testing, Verification, and Reliability, vol. 9, no. 4, pp. 205-232, Dec. 1999.
[18] A.J. Offutt, G. Rothermel, R.H. Untch, and C. Zapf, "An Experimental Determination of Sufficient Mutant Operators," ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 99-118, 1996.
[19] E.F. Barbosa, J.C. Maldonado, and A.M.R. Vincenzi, "Toward the Determination of Sufficient Mutant Operators for C," Software Testing, Verification, and Reliability, vol. 11, no. 2, pp. 113-136, 2001.
[20] R.A. DeMillo and E.H. Spafford, "The Mothra Software Testing Environment," Proc. 11th NASA Software Eng. Laboratory Workshop, 1986.
[21] A.T. Acree, "On Mutation," thesis, School of Information and Computer Science, Georgia Inst. of Tech nology, 1980.
[22] K.N. King and A.J. Offutt, "A Fortran Language System for Mutation Based Software Testing," Software: Practice and Experience, vol. 21, no. 7, pp. 685-718, 1991.
[23] R. Untch, A. Offutt, and M. Harrold, "Mutation Analysis Using Program Schemata," Proc. Int'l Symp. Software Testing and Analysis, pp. 139-148, June 1993.
[24] Y.-S. Ma, J. Offutt, and Y.R. Kwon, "MuJava: An Automated Class Mutation System," Software Testing, Verification, and Reliability, vol. 15, no. 2, pp. 97-133, 2005.
[25] M. Grindal, A.J. Offutt, and S.F. Andler, "Combination Testing Strategies: A Survey," Software Testing, Verification, and Reliability, vol. 15, pp. 167-199, 2005.
[26] K.-C. Tai and Y. Lei, "A Test Generation Strategy for Pairwise Testing," IEEE Trans. Software Eng., vol. 28, no. 1, pp. 109-111, Jan. 2002.
[27] A.J. Offutt, "Investigations of the Software Testing Coupling Effect," ACM Trans. Software Eng. and Methodology, vol. 1, no. 1, pp. 15-20, 1992.
[28] Y. Jia and M. Harman, "MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language," Proc. Third Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 94-98, Aug. 2008.
[29] M. Polo, M. Piattini, and S. Tendero, "Integrating Techniques and Tools for Testing Automation," Software Testing, Verification, and Reliability, vol. 17, no. 1, pp. 3-39, 2007.
[30] H. Do, S. Elbaum, and G. Rothermel, "Infrastructure Support for Controlled Experimentation with Software Testing and Regression Testing Techniques," Proc. Int'l Symp. Empirical Software Eng., pp. 60-70, Aug. 2004.
[31] M.E. Delamaro, J.C. Maldonado, and A. Vincenzi, "Proteum/IM 2.0: An Integrated Mutation Testing Environment," Proc. First Workshop Mutation Analysis, pp. 91-101, Oct. 2000.
[32] M. Kintis, M. Papadakis, and N. Malevris, "Evaluating Mutation Testing Alternatives: A Collateral Experiment," Proc. Asia Pacific Eng. Conf., Nov./Dec. 2010.
[33] H. Agrawal, "Dominators, Super Blocks, and Program Coverage," Proc. 21st ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages, pp. 25-34, Jan. 1994.
[34] W. Langdon, M. Harman, and Y. Jia, "Efficient Multi-Objective Higher Order Mutation Testing with Genetic Programming," J. Systems and Software, vol. 83, no. 12, pp. 2416-2430, 2010.
[35] W. Langdon, M. Harman, and Y. Jia, "Multi Objective Higher Order Mutation Testing with GP," Proc. 11th Ann. Genetic and Evolutionary Computation Conf., pp. 1945-1946, 2009.
[36] W. Langdon, M. Harman, and Y. Jia, "Multi Objective Higher Order Mutation Testing with Genetic Programming," Proc. Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 21-29, 2009.
[37] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, "Experiments of the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria," Proc. 16th Int'l Conf. Software Eng., 1994.
[38] R. DeMillo, R.J. Lipton, and F.G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, April 1978.
[39] G. Kaminski, U. Praphamontripong, P. Ammann, and J. Offutt, "A Logic Mutation Approach to Selective Mutation for Programs and Queries," Information and Software Technology, vol. 53, pp. 1137-1152, 2011.
[40] G. Kaminski, P. Amman, and A.J. Offut, "Better Predicate Testing," Proc. Sixth Int'l Workshop Automation of Software Test, pp. 57-63, 2011.
[41] G. Kaminski and P. Amman, "Using Logic Criterion Feasibility to Reduce Test Set Size While Guaranteeing Fault Detection," Proc. Int'l Conf. Software Testing, Apr. 2009.
[42] R. Just, G.M. Kapfhammer, and F. Schweiggert, "Using Conditional Mutation to Increase the Efficiency of Mutation Analysis," Proc. Sixth Int'l Workshop Automation of Software Test, pp. 50-56, 2010, 2011.
[43] R. Purushothaman and D.E. Perry, "Toward Understanding the Rhetoric of Small Source Code Changes," IEEE Trans. Software Eng., vol. 31, no. 6 pp. 511-526, June 2005.
[44] V.R. Basili and H.D. Rombach, "The TAME Project: Towards Improvement-Oriented Softwareenvironments," IEEE Trans. Software Eng., vol. 14, no. 6, pp. 758-773, June 1988.
[45] V.R. Basili, F. Shull, and F. Lanubile, "Building Knowledge through Families of Experiments," IEEE Trans. Software Eng., vol. 25, no. 4, pp. 456-473, July/Aug. 1999.
[46] M. Harrold, R. Gupta, and M. Soffa, "A Methodology for Controlling the Size of a Test Suite," ACM Trans. Software Eng. and Methodology, vol. 2, no. 3, pp. 270-285, 1993.
[47] D. Jeffrey and N. Gupta, "Test Suite Reduction with Selective Redundancy," Proc. Int'l Conf. Software Maintenance, pp. 549-558, 2005.
[48] S. Tallam and N. Gupta, "A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization," Proc. Sixth ACM SIGPLAN-SIGSOFT Workshop Program Analysis for Software Tools and Eng., pp. 35-42, 2005.
[49] "Eclipse Framework," http:/www.eclipse.org/, Feb. 2011.
[50] L. Walton, "Eclipse Metrics Plugin," http:/eclipse-metrics. sourceforge.net/, Feb. 2011.
[51] "SPSS Statistics," http:/www.spss.com/, Feb. 2011.
[52] S. Ghosh and A.P. Mathur, "Interface Mutation," Software Testing, Verification, and Reliability, no. 11, pp. 227-247, 2001.
[53] C. Wohlin, P. Runeson, M. Höst, M. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering: An Introduction. Kluwer Academic, 2000.
46 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool