This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
An Analysis and Survey of the Development of Mutation Testing
September/October 2011 (vol. 37 no. 5)
pp. 649-678
Yue Jia, University College London, London
Mark Harman, University College London, London
Mutation Testing is a fault-based software testing technique that has been widely studied for over three decades. The literature on Mutation Testing has contributed a set of approaches, tools, developments, and empirical results. This paper provides a comprehensive analysis and survey of Mutation Testing. The paper also presents the results of several development trend analyses. These analyses provide evidence that Mutation Testing techniques and tools are reaching a state of maturity and applicability, while the topic of Mutation Testing itself is the subject of increasing interest.

[1] R. Abraham and M. Erwig, "Mutation Operators for Spreadsheets," IEEE Trans. Software Eng., vol. 35, no. 1, pp. 94-108, Jan./Feb. 2009.
[2] A.T. Acree, "On Mutation," PhD thesis, Georgia Inst. of Tech nology, 1980.
[3] A.T. Acree, T.A. Budd, R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Mutation Analysis," Technical Report GIT-ICS-79/08, Georgia Inst. of Tech nology, 1979.
[4] K. Adamopoulos, "Search Based Test Selection and Tailored Mutation," master's thesis, King's College London, 2009.
[5] K. Adamopoulos, M. Harman, and R.M. Hierons, "How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-Evolution," Proc. Genetic and Evolutionary Computation Conf., pp. 1338-1349, June 2004.
[6] H. Agrawal, R.A. DeMillo, B. Hathaway, W. Hsu, W. Hsu, E.W. Krauser, R.J. Martin, A.P. Mathur, and E. Spafford, "Design of Mutant Operators for the C Programming Language," Technical Report SERC-TR-41-P, Purdue Univ., Mar. 1989.
[7] B.K. Aichernig, "Mutation Testing in the Refinement Calculus," Formal Aspects of Computing, vol. 15, nos. 2-3, pp. 280-295, Nov. 2003.
[8] B.K. Aichernig and C.C. Delgado, "From Faults via Test Purposes to Test Cases: On the Fault-Based Testing of Concurrent Systems," Proc. Ninth Int'l Conf. Fundamental Approaches to Software Eng., pp. 324-338, Mar. 2006.
[9] R.T. Alexander, J.M. Bieman, S. Ghosh, and B. Ji, "Mutation of Java Objects," Proc. 13th Int'l Symp. Software Reliability Eng., pp. 341-351, Nov. 2002.
[10] S. Ali, L.C. Briand, H. Hemmati, and R.K. Panesar-Walawege, "A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation," IEEE Trans. Software Eng., vol. 36, no. 6, pp. 742-762, Nov./Dec. 2010.
[11] P. Ammann and J. Offutt, Introduction to Software Testing. Cambridge Univ. Press, 2008.
[12] P. Anbalagan and T. Xie, "Efficient Mutant Generation for Mutation Testing of Pointcuts in Aspect-Oriented Programs," Proc. Second Workshop Mutation Analysis, pp. 51-56, Nov. 2006.
[13] P. Anbalagan and T. Xie, "Automated Generation of Pointcut Mutants for Testing Pointcuts in AspectJ Programs," Proc. 19th Int'l Symp. Software Reliability Eng., pp. 239-248, Nov. 2008.
[14] J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. 27th Int'l Conf. Software Eng., pp. 402-411, May 2005.
[15] J.H. Andrews, L.C. Briand, Y. Labiche, and A.S. Namin, "Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria," IEEE Trans. Software Eng., vol. 32, no. 8, pp. 608-624, Aug. 2006.
[16] K. Ayari, S. Bouktif, and G. Antoniol, "Automatic Mutation Test Input Data Generation via Ant Colony," Proc. Genetic and Evolutionary Computation Conf., pp. 1074-1081, July 2007.
[17] J.S. Baekken and R.T. Alexander, "A Candidate Fault Model for AspectJ Pointcuts," Proc. 17th Int'l Symp. Software Reliability Eng., pp. 169-178, Nov. 2006.
[18] D. Baldwin and F.G. Sayward, "Heuristics for Determining Equivalence of Program Mutations," Research Report 276, Yale Univ., 1979.
[19] E.F. Barbosa, J.C. Maldonado, and A.M.R. Vincenzi, "Toward the Determination of Sufficient Mutant Operators for C," Software Testing, Verification, and Reliability, vol. 11, no. 2, pp. 113-136, May 2001.
[20] S.S. Batth, E.R. Vieira, A.R. Cavalli, and M.U. Uyar, "Specification of Timed EFSM Fault Models in SDL," Proc. 27th IFIP WG 6.1 Int'l Conf. Formal Techniques for Networked and Distributed Systems, pp. 50-65, June 2007.
[21] B. Baudry, F. Fleurey, J.-M. Jezequel, and Y. Le Traon, "Genes and Bacteria for Automatic Test Cases Optimization in the .NET Environment," Proc. 13th Int'l Symp. Software Reliability Eng., pp. 195-206, Nov. 2002.
[22] B. Baudry, V. Le Hanh, J.-M. Jézéquel, and Y. Le Traon, "Trustable Components: Yet Another Mutation-Based Approach," Proc. First Workshop Mutation Analysis, pp. 47-54, Oct. 2000.
[23] F. Belli, C.J. Budnik, and W.E. Wong, "Basic Operations for Generating Behavioral Mutants," Proc. Second Workshop Mutation Analysis, p. 9, 2006.
[24] J. Bieman, S. Ghosh, and R.T. Alexander, "A Technique for Mutation of Java Objects," Proc. 16th IEEE Int'l Conf. Automated Software Eng., p. 337, Nov. 2001.
[25] P.E. Black, V. Okun, and Y. Yesha, "Mutation of Model Checker Specifications for Test Generation and Evaluation," Proc. First Workshop Mutation Analysis, pp.14-20, Oct. 2000.
[26] B. Bogacki and B. Walter, "Evaluation of Test Code Quality with Aspect-Oriented Mutations," Proc. Seventh Int'l Conf. eXtreme Programming and Agile Processes in Software Eng., pp. 202-204, June 2006.
[27] B. Bogacki and B. Walter, "Aspect-Oriented Response Injection: An Alternative to Classical Mutation Testing," Software Eng. Techniques: Design for Quality, pp. 273-282, Springer, 2007.
[28] N. Bombieri, F. Fummi, and G. Pravadelli, "A Mutation Model for the SystemC TLM2.0 Communication Interfaces," Proc. Conf. Design, Automation and Test in Europe, pp. 396-401, Mar. 2008.
[29] J.H. Bowser, "Reference Manual for Ada Mutant Operators," Technical Report GIT-SERC-88/02, Georgia Inst. of Tech nology, 1988.
[30] J.S. Bradbury, J.R. Cordy, and J. Dingel, "ExMAn: A Generic and Customizable Framework for Experimental Mutation Analysis," Proc. Second Workshop Mutation Analysis, pp. 57-62, Nov. 2006.
[31] J.S. Bradbury, J.R. Cordy, and J. Dingel, "Mutation Operators for Concurrent Java (J2SE 5.0)," Proc. Second Workshop Mutation Analysis, pp. 83-92, Nov. 2006.
[32] J.S. Bradbury, J.R. Cordy, and J. Dingel, "Comparative Assessment of Testing and Model Checking Using Program Mutation," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 210-222, 2007.
[33] P. Brady, "MutateMe," http://github.com/padraic/mutateme/treemaster , 2007.
[34] T.A. Budd, "Mutation Analysis of Program Test Data," PhD thesis, Yale Univ., 1980.
[35] T.A. Budd and D. Angluin, "Two Notions of Correctness and Their Relation to Testing," Acta Informatica, vol. 18, no. 1, pp. 31-45, Mar. 1982.
[36] T.A. Budd, R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "The Design of a Prototype Mutation System for Program Testing," Proc. Am. Fed. of Information Processing Soc. Nat'l Computer Conf., vol. 74, pp. 623-627, June 1978.
[37] T.A. Budd, R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Theoretical and Empirical Studies on Using Program Mutation to Test the Functional Correctness of Programs," Proc. Seventh ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages, pp. 220-233, Jan. 1980.
[38] T.A. Budd and A.S. Gopal, "Program Testing by Specification Mutation," Computer Languages, vol. 10, no. 1, pp. 63-73, 1985.
[39] T.A. Budd, R. Hess, and F.G. Sayward, "EXPER Implementor's Guide," technical report, Yale Univ., 1980.
[40] T.A. Budd and F.G. Sayward, "Users Guide to the Pilot Mutation System," Technical Report 114, Yale Univ., 1977.
[41] R.H. Carver, "Mutation-Based Testing of Concurrent Programs," Proc. IEEE Int'l Test Conf. Designing, Testing, and Diagnostics, pp. 845-853, Oct. 1993.
[42] Cetress, "Certitude," http://www.certess.comproduct/, 2006.
[43] W.K. Chan, S.C. Cheung, and T.H. Tse, "Fault-Based Testing of Database Application Programs with Conceptual Data Model," Proc. Fifth Int'l Conf. Quality Software, pp. 187-196, Sept. 2005.
[44] P. Chevalley, "Applying Mutation Analysis for Object-Oriented Programs Using a Reflective Approach," Proc. Eighth Asia-Pacific Software Eng. Conf., p. 267, Dec. 2001.
[45] P. Chevalley and P. Thévenod-Fosse, "A Mutation Analysis Tool for Java Programs," Int'l J. Software Tools for Technology Transfer, vol. 5, no. 1, pp. 90-103, Nov. 2002.
[46] B. Choi, "Software Testing Using High Performance Computers," PhD thesis, Purdue Univ., July 1991.
[47] B. Choi and A.P. Mathur, "High-Performance Mutation Testing," J. Systems and Software, vol. 20, no. 2, pp. 135-152, Feb. 1993.
[48] W.M. Craft, "Detecting Equivalent Mutants Using Compiler Optimization Techniques," master's thesis, Clemson Univ., Sept. 1989.
[49] Y. Crouzet, H. Waeselynck, B. Lussier, and D. Powell, "The SESAME Experience: From Assembly Languages to Declarative Models," Proc. Second Workshop Mutation Analysis, p. 7, Nov. 2006.
[50] M. Daran and P. Thévenod-Fosse, "Software Error Analysis: A Real Case Study Involving Real Faults and Mutations," ACM SIGSOFT Software Eng. Notes, vol. 21, no. 3, pp. 158-177, May 1996.
[51] R. Delamare, B. Baudry, and Y. Le Traon, "AjMutator: A Tool for the Mutation Analysis of AspectJ Pointcut Descriptors," Proc. Fourth Int'l Workshop Mutation Analysis, published with Proc. Second Int'l Conf. Software Testing, Verification, and Validation Workshops, pp. 200-204, Apr. 2009.
[52] M.E. Delamaro, "Proteum—A Mutation Analysis Based Testing Environmen," master's thesis, Univ. of São Paulo, 1993.
[53] M.E. Delamaro and J.C. Maldonado, "Proteum—A Tool for the Assessment of Test Adequacy for C Programs," Proc. Conf. Performability in Computing Systems, pp. 79-95, July 1996.
[54] M.E. Delamaro and J.C. Maldonado, "Interface Mutation: Assessing Testing Quality at Interprocedural Level," Proc. 19th Int'l Conf. Chilean Computer Science Soc., pp. 78-86, Nov. 1999.
[55] M.E. Delamaro, J.C. Maldonado, and A.P. Mathur, "Integration Testing Using Interface Mutation," Proc. Seventh Int'l Symp. Software Reliability Eng., pp. 112-121, Oct.-Nov. 1996.
[56] M.E. Delamaro, J.C. Maldonado, and A.P. Mathur, "Interface Mutation: An Approach for Integration Testing," IEEE Trans. Software Eng., vol. 27, no. 3, pp. 228-247, Mar. 2001.
[57] M.E. Delamaro, J.C. Maldonado, A. Pasquini, and A.P. Mathur, "Interface Mutation Test Adequacy Criterion: An Empirical Evaluation," technical report, State Univ. of Maringá, 2000.
[58] M.E. Delamaro, J.C. Maldonado, A. Pasquini, and A.P. Mathur, "Interface Mutation Test Adequacy Criterion: An Empirical Evaluation," Empirical Software Eng., vol. 6, no. 2, pp. 111-142, June 2001.
[59] M.E. Delamaro, J.C. Maldonado, and A. Vincenzi, "Proteum/IM 2.0: An Integrated Mutation Testing Environment," Proc. First Workshop Mutation Analysis, Oct. 2000.
[60] R.A. DeMillo, "Program Mutation: An Approach to Software Testing," technical report, Georgia Inst. of Tech nology, 1983.
[61] R.A. DeMillo and A.P. Mathur, "On the Use of Software Artifacts to Evaluate the Effectiveness of Mutation Analysis in Detecting Errors in Production Software," Technical Report SERC-TR-92-P, Purdue Univ., 1992.
[62] R.A. DeMillo, "Test Adequacy and Program Mutation," Proc. 11th Int'l Conf. Software Eng., pp. 355-356, May 1989.
[63] R.A. DeMillo, D.S. Guindi, K.N. King, and W.M. McCracken, "An Overview of the Mothra Software Testing Environment," Technical Report SERC-TR-3-P, Purdue Univ., 1987.
[64] R.A. DeMillo, D.S. Guindi, K.N. King, W.M. McCracken, and A.J. Offutt, "An Extended Overview of the Mothra Software Testing Environment," Proc. Second Workshop Software Testing, Verification, and Analysis, pp. 142-151, July 1988.
[65] R.A. DeMillo, E.W. Krauser, and A.P. Mathur, "Compiler-Integrated Program Mutation," Proc. Fifth Ann. Computer Software and Applications Conf., pp. 351-356, Sept. 1991.
[66] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
[67] R.A. DeMillo and A.J. Offutt, "Constraint-Based Automatic Test Data Generation," IEEE Trans. Software Eng., vol. 17, no. 9, pp. 900-910, Sept. 1991.
[68] R.A. DeMillo and A.J. Offutt, "Experimental Results from an Automatic Test Case Generator," ACM Trans. Software Eng. and Methodology, vol. 2, no. 2, pp. 109-127, Apr. 1993.
[69] A. Derezińska, "Object-Oriented Mutation to Assess the Quality of Tests," Proc. 29th Euromicro Conf., pp. 417-420, Sept. 2003.
[70] A. Derezińska, "Advanced Mutation Operators Applicable in C# Programs," technical report, Warsaw Univ. of Tech nology, 2005.
[71] A. Derezińska, "Quality Assessment of Mutation Operators Dedicated for C# Programs," Proc. Sixth Int'l Conf. Quality Software, Oct. 2006.
[72] A. Derezińska and A. Szustek, "CREAM—A System for Object-Oriented Mutation of C# Programs," technical report, Warsaw Univ. of Tech nology, 2007.
[73] A. Derezińska and A. Szustek, "Tool-Supported Advanced Mutation Approach for Verification of C# Programs," Proc. Third Int'l Conf. Dependability of Computer Systems, pp. 261-268, June 2008.
[74] W. Ding, "Using Mutation to Generate Tests from Specifications," master's thesis, George Mason Univ., 2000.
[75] H. Do and G. Rothermel, "A Controlled Experiment Assessing Test Case Prioritization Techniques via Mutation Faults," Proc. 21st IEEE Int'l Conf. Software Maintenance, pp. 411-420, Sept. 2005.
[76] H. Do and G. Rothermel, "On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques," IEEE Trans. Software Eng., vol. 32, no. 9, pp. 733-752, Sept. 2006.
[77] J.J. Domínguez-Jiménez, A. Estero-Botaro, and I. Medina-Bulo, "A Framework for Mutant Genetic Generation for WS-BPEL," Proc. 35th Conf. Current Trends in Theory and Practice of Computer Science, pp. 229-240, Jan. 2009.
[78] W. Du and A.P. Mathur, "Vulnerability Testing of Software System Using Fault Injection," Technical Report COAST TR 98-02, Purdue Univ., 1998.
[79] W. Du and A.P. Mathur, "Testing for Software Vulnerability Using Environment Perturbation," Proc. Int'l Conf. Dependable Systems and Networks, pp. 603-612, June 2000.
[80] L.du Bousquet and M. Delaunay, "Mutation Analysis for Lustre Programs: Fault Model Description and Validation," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 176-184, Sept. 2007.
[81] L. du Bousquet and M. Delaunay, "Using Mutation Analysis to Evaluate Test Generation Strategies in a Synchronous Context," Proc. Second Int'l Conf. Software Eng. Advances, p. 40, Aug. 2007.
[82] Ellims, "Csaw," http://www.skicambridge.com/papersCsaw_v1_files.html , 2007.
[83] M. Ellims, D.C. Ince, and M. Petre, "The Csaw C Mutation Tool: Initial Results," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 185-192, Sept. 2007.
[84] A. Estero-Botaro, F. Palomo-Lozano, and I. Medina-Bulo, "Mutation Operators for WS-BPEL 2.0," Proc. 21st Int'l Conf. Software and Systems Eng. and Their Applications, Dec. 2008.
[85] S.C.P.F. Fabbri, J.C. Maldonado, P.C. Masiero, and M.E. Delamaro, "Proteum/FSM: A Tool to Support Finite State Machine Validation Based on Mutation Testing," Proc. 19th Int'l Conf. Chilean Computer Science Soc., pp. 96-104, Nov. 1999.
[86] S.C.P.F. Fabbri, J.C. Maldonado, P.C. Masiero, M.E. Delamaro, and W.E. Wong, "Mutation Testing Applied to Validate Specifications Based on Petri Nets," Proc. IFIP TC6 Eighth Int'l Conf. Formal Description Techniques VIII, vol. 43, pp. 329-337, 1995.
[87] S.C.P.F. Fabbri, J.C. Maldonado, T. Sugeta, and P.C. Masiero, "Mutation Testing Applied to Validate Specifications Based on Statecharts," Proc. 10th Int'l Symp. Software Reliability Eng., pp. 210-219, Nov. 1999.
[88] S.P.F. Fabbri, M.E. Delamaro, J.C. Maldonado, and P. Masiero, "Mutation Analysis Testing for Finite State Machines," Proc. Fifth Int'l Symp. Software Reliability Eng., pp. 220-229, Nov. 1994.
[89] X. Feng, S. Marr, and T. O'Callaghan, "ESTP: An Experimental Software Testing Platform," Proc. Third Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 59-63, Aug. 2008.
[90] F.C. Ferrari, J.C. Maldonado, and A. Rashid, "Mutation Testing for Aspect-Oriented Programs," Proc. First Int'l Conf. Software Testing, Verification, and Validation, pp. 52-61, Apr. 2008.
[91] S. Fichter, "Parallelizing Mutation on a Hypercube," master's thesis, Clemson Univ., 1991.
[92] V.N. Fleyshgakker and S.N. Weiss, "Efficient Mutation Analysis: A New Approach," Proc. Int'l Symp. Software Testing and Analysis, pp. 185-195, Aug. 1994.
[93] P.G. Frankl, S.N. Weiss, and C. Hu, "All-Uses versus Mutation Testing: An Experimental Comparison of Effectiveness," technical report, Polytechnic Univ., 1994.
[94] P.G. Frankl, S.N. Weiss, and C. Hu, "All-Uses vs Mutation Testing: An Experimental Comparison of Effectiveness," J. Systems and Software, vol. 38, no. 3, pp. 235-253, Sept. 1997.
[95] G. Fraser and F. Wotawa, "Mutant Minimization for Model-Checker Based Test-Case Generation," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 161-168, Sept. 2007.
[96] R. Geist, A.J. Offutt, and F.C. Harris, "Estimation and Enhancement of Real-Time Software Reliability through Mutation Analysis," IEEE Trans. Computers, vol. 41, no. 5, pp. 550-558, May 1992.
[97] A.K. Ghosh, T. O'Connor, and G. McGraw, "An Automated Approach for Identifying Potential Vulnerabilities in Software," Proc. IEEE Symp. Security and Privacy, pp. 104-114, May 1998.
[98] S. Ghosh, "Testing Component-Based Distributed Applications," PhD thesis, Purdue Univ., 2000.
[99] S. Ghosh, "Towards Measurement of Testability of Concurrent Object-Oriented Programs Using Fault Insertion: A Preliminary Investigation," Proc. Second IEEE Int'l Workshop Source Code Analysis and Manipulation, pp. 17-25, 2002.
[100] S. Ghosh, P. Govindarajan, and A.P. Mathur, "TDS: A Tool for Testing Distributed Component-Based Applications," Proc. First Workshop Mutation Analysis, pp. 103-112, Oct. 2000.
[101] S. Ghosh and A.P. Mathur, "Interface Mutation to Assess the Adequacy of Tests for Components and Systems," Proc. 34th Int'l Conf. Technology of Object-Oriented Languages and Systems, pp. 37-46, July-Aug. 2000.
[102] S. Ghosh and A.P. Mathur, "Interface Mutation," Software Testing, Verification and Reliability, vol. 11, no. 3, pp. 227-247, Mar. 2001.
[103] M.R. Girgis and M.R. Woodward, "An Integrated System for Program Testing Using Weak Mutation and Data Flow Analysis," Proc. Eighth Int'l Conf. Software Eng., pp. 313-319, Aug. 1985.
[104] P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 213-223, June 2005.
[105] A.S. Gopal and T.A. Budd, "Program Testing by Specification Mutation," Technical Report TR 83-17, Univ. of Arizona, 1983.
[106] B.J.M. Grün, D. Schuler, and A. Zeller, "The Impact of Equivalent Mutants," Proc. Fourth Int'l Workshop Mutation Analysis, published with Proc. Second Int'l Conf. Software Testing, Verification, and Validation Workshops, pp. 192-199, Apr. 2009.
[107] R.G. Hamlet, "Testing Programs with the Aid of a Compiler," IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279-290, July 1977.
[108] J.M. Hanks, "Testing Cobol Programs by Mutation," PhD thesis, Georgia Inst. of Tech nology, 1980.
[109] M. Harman, R. Hierons, and S. Danicic, "The Relationship between Program Dependence and Mutation Analysis," Proc. First Workshop Mutation Analysis, pp. 5-13, Oct. 2000.
[110] R.M. Hierons, M. Harman, and S. Danicic, "Using Program Slicing to Assist in the Detection of Equivalent Mutants," Software Testing, Verification, and Reliability, vol. 9, no. 4, pp. 233-262, Dec. 1999.
[111] R.M. Hierons and M.G. Merayo, "Mutation Testing from Probabilistic Finite State Machines," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 141-150, Sept. 2007.
[112] R.M. Hierons and M.G. Merayo, "Mutation Testing from Probabilistic and Stochastic Finite State Machines," J. Systems and Software, vol. 82, no. 11, pp. 1804-1818, Nov. 2009.
[113] J.R. Horgan and A.P. Mathur, "Weak Mutation is Probably Strong Mutation," Technical Report SERC-TR-83-P, Purdue Univ., 1990.
[114] S.-S. Hou, L. Zhang, T. Xie, H. Mei, and J.-S. Sun, "Applying Interface-Contract Mutation in Regression Testing of Component-Based Software," Proc. 23rd Int'l Conf. Software Maintenance, pp. 174-183, Oct. 2007.
[115] W.E. Howden, "Weak Mutation Testing and Completeness of Test Sets," IEEE Trans. Software Eng., vol. 8, no. 4, pp. 371-379, July 1982.
[116] S. Hussain, "Mutation Clustering," master's thesis, King's College London, 2008.
[117] J. Hwang, T. Xie, F. Chen, and A.X. Liu, "Systematic Structural Testing of Firewall Policies," Proc. IEEE Symp. Reliable Distributed Systems, pp. 105-114, Oct. 2008.
[118] Itregister, "Plextest," http://www.itregister.com.au/productsplextest.htm , 2007.
[119] D. Jackson and M.R. Woodward, "Parallel Firm Mutation of Java Programs," Proc. First Workshop Mutation Analysis, pp. 55-61, Oct. 2000.
[120] C. Ji, Z. Chen, B. Xu, and Z. Zhao, "A Novel Method of Mutation Clustering Based on Domain Analysis," Proc. 21st Int'l Conf. Software Eng. and Knowledge Eng., July 2009.
[121] Y. Jia, "Mutation Testing Repository," http://www.dcs.kcl. ac.uk/pg/jiayuerepository /, 2009.
[122] Y. Jia and M. Harman, "Constructing Subtle Faults Using Higher Order Mutation Testing," Proc. Eighth Int'l Working Conf. Source Code Analysis and Manipulation, pp. 249-258, Sept. 2008.
[123] Y. Jia and M. Harman, "MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language," Proc. Third Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 94-98, Aug. 2008.
[124] C. Jing, Z. Wang, X. Shi, X. Yin, and J. Wu, "Mutation Testing of Protocol Messages Based on Extended TTCN-3," Proc. 22nd Int'l Conf. Advanced Information Networking and Applications, pp. 667-674, Mar. 2008.
[125] K. Kapoor, "Formal Analysis of Coupling Hypothesis for Logical Faults," Innovations in Systems and Software Eng., vol. 2, no. 2, pp. 80-87, July 2006.
[126] S.-W. Kim, M.J. Harrold, and Y.-R. Kwon, "MUGAMMA: Mutation Analysis of Deployed Software to Increase Confidence Assist Evolution," Proc. Second Workshop Mutation Analysis, p. 10, Nov. 2006.
[127] S. Kim, J.A. Clark, and J.A. McDermid, "Assessing Test Set Adequacy for Object Oriented Programs Using Class Mutation," Proc. Third Symp. Software Technology, Sept. 1999.
[128] S. Kim, J.A. Clark, and J.A. McDermid, "The Rigorous Generation of Java Mutation Operators Using HAZOP," Proc. 12th Int'l Conf. Software and Systems Eng. and Their Applications, Nov.-Dec. 1999.
[129] S. Kim, J.A. Clark, and J.A. McDermid, "Class Mutation: Mutation Testing for Object-Oriented Programs," Proc. Net.ObjectDays Conf. Object-Oriented Software Systems, 2000.
[130] S. Kim, J.A. Clark, and J.A. McDermid, "Investigating the Effectiveness of Object-Oriented Testing Strategies Using the Mutation Method," Proc. First Workshop Mutation Analysis, pp. 207-225, Oct. 2000.
[131] K.N. King and A.J. Offutt, "A Fortran Language System for Mutation-Based Software Testing," Software: Practice and Experience, vol. 21, no. 7, pp. 685-718, Oct. 1991.
[132] E.W. Krauser, "Compiler-Integrated Software Testing," PhD thesis, Purdue Univ., 1991.
[133] E.W. Krauser, A.P. Mathur, and V.J. Rego, "High Performance Software Testing on SIMD Machines," IEEE Trans. Software Eng., vol. 17, no. 5, pp. 403-423, May 1991.
[134] E.W. Krauser, A.P. Mathur, and V.J. Rego, "High Performance Software Testing on SIMD Machines," Proc. Second Workshop Software Testing, Verification, and Analysis, pp. 171-177, July 1988.
[135] K. Lakhotia, P. McMinn, and M. Harman, "Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet?" Proc. Testing: Academia & Industry Conf.—Practice And Research Techniques, pp. 95-104, Sept. 2009.
[136] W.B. Langdon, M. Harman, and Y. Jia, "Multi Objective Higher Order Mutation Testing with Genetic Programming," Proc. Fourth Testing: Academic and Industrial Conf.—Practice and Research, Sept. 2009.
[137] W.B. Langdon, M. Harman, and Y. Jia, "Multi Objective Mutation Testing with Genetic Programming," Proc. Genetic and Evolutionary Computation Conf., July 2009.
[138] Y. Le Traon, B. Baudry, and J.-M. Jézéquel, "Design by Contract to Improve Software Vigilance," IEEE Trans. Software Eng., vol. 32, no. 8, pp. 571-586, Aug. 2006.
[139] Y.L. Traon, T. Mouelhi, and B. Baudry, "Testing Security Policies: Going Beyond Functional Testing," Proc. 18th IEEE Int'l Symp. Software Reliability, pp. 93-102, Nov. 2007.
[140] S. Lee, X. Bai, and Y. Chen, "Automatic Mutation Testing and Simulation on OWL-S Specified Web Services," Proc. 41st Ann. Simulation Symp., pp. 149-156, Apr. 2008.
[141] S.D. Lee, "Weak versus Strong: An Empirical Comparison of Mutation Variants," master's thesis, Clemson Univ., 1991.
[142] S.C. Lee and A.J. Offutt, "Generating Test Cases for XML-Based Web Component Interactions Using Mutation Analysis," Proc. 12th Int'l Symp. Software Reliability Eng., pp. 200-209, Nov. 2001.
[143] J.B. Li and J. Miller, "Testing the Semantics of W3C XML Schema," Proc. 29th Ann. Int'l Computer Software and Applications Conf., pp. 443-448, July 2005.
[144] R. Lipton, "Fault Diagnosis of Computer Programs," student report, Carnegie Mellon Univ., 1971.
[145] R.J. Lipton and F.G. Sayward, "The Status of Research on Program Mutation," Proc. Workshop Software Testing and Test Documentation, pp. 355-373, Dec. 1978.
[146] M.-H. Liu, Y.-F. Gao, J.-H. Shan, J.-H. Liu, L. Zhang, and J.-S. Sun, "An Approach to Test Data Generation for Killing Multiple Mutants," Proc. 22nd IEEE Int'l Conf. Software Maintenance, pp. 113-122, Sept. 2006.
[147] B. Long, R. Duke, D. Goldson, P. Strooper, and L. Wildman, "Mutation-Based Exploration of a Method for Verifying Concurrent Java Components," Proc. 18th Int'l Parallel and Distributed Processing Symp., p. 265, Apr. 2004.
[148] Y.-S. Ma, "Object-Oriented Mutation Testing for Java," PhD thesis, KAIST Univ. in Korea, 2005.
[149] Y.-S. Ma, M.J. Harrold, and Y.-R. Kwon, "Evaluation of Mutation Testing for Object-Oriented Programs," Proc. 28th Int'l Conf. Software Eng., pp. 869-872, May 2006.
[150] Y.-S. Ma, Y.-R. Kwon, and A.J. Offutt, "Inter-Class Mutation Operators for Java," Proc. 13th Int'l Symp. Software Reliability Eng., pp. 352-363, Nov. 2002.
[151] Y.-S. Ma, A.J. Offutt, and Y.-R. Kwon, "MuJava: An Automated Class Mutation System," Software Testing, Verification, and Reliability, vol. 15, no. 2, pp. 97-133, June 2005.
[152] Y.-S. Ma, A.J. Offutt, and Y.-R. Kwon, "MuJava: A Mutation System for Java," Proc. 28th Int'l Conf. Software Eng., pp. 827-830, May 2006.
[153] B. Marick, "The Weak Mutation Hypothesis," Proc. Fourth Symp. Software Testing, Analysis, and Verification pp. 190-199, Oct. 1991.
[154] E.E. Martin and T. Xie, "A Fault Model and Mutation Testing of Access Control Policies," Proc. 16th Int'l Conf. World Wide Web, pp. 667-676, May 2007.
[155] A.P. Mathur, Foundations of Software Testing. Pearson Education, 2008.
[156] A.P. Mathur, "Performance, Effectiveness, and Reliability Issues in Software Testing," Proc. Fifth Int'l Computer Software and Applications Conf., pp. 604-605, Sept. 1991.
[157] A.P. Mathur, "CS 406 Software Engineering I," Course Project Handout, Aug. 1992.
[158] A.P. Mathur and E.W. Krauser, "Mutant Unification for Improved Vectorization," Technical Report SERC-TR-14-P, Purdue Univ., 1988.
[159] A.P. Mathur and W.E. Wong, "An Empirical Comparison of Mutation and Data Flow Based Test Adequacy Criteria," technical report, Purdue Univ., 1993.
[160] A.P. Mathur and W.E. Wong, "An Empirical Comparison of Data Flow and Mutation-Based Test Adequacy Criteria," Software Testing, Verification, and Reliability, vol. 4, no. 1, pp. 9-31, 1994.
[161] P.S. May, "Test Data Generation: Two Evolutionary Approaches to Mutation Testing," PhD thesis, Univ. of Kent, 2007.
[162] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification, and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[163] I. Moore, "Jester and Pester," http:/jester.sourceforge.net/, 2001.
[164] L.J. Morell, "A Theory of Error-Based Testing," PhD thesis, Univ. of Maryland at College Park, 1984.
[165] T. Mouelhi, F. Fleurey, and B. Baudry, "A Generic Metamodel For Security Policies Mutation," Proc. IEEE Int'l Conf. Software Testing, Verification, and Validation Workshop, pp. 278-286, Apr. 2008.
[166] T. Mouelhi, Y. Le Traon, and B. Baudry, "Mutation Analysis for Security Tests Qualification," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 233-242, Sept. 2007.
[167] E.S. Mresa and L. Bottaci, "Efficiency of Mutation Operators and Selective Mutation Strategies: An Empirical Study," Software Testing, Verification, and Reliability, vol. 9, no. 4, pp. 205-232, Dec. 1999.
[168] A.S. Namin and J.H. Andrews, "Finding Sufficient Mutation Operators via Variable Reduction," Proc. Second Workshop Mutation Analysis, p. 5, Nov. 2006.
[169] A.S. Namin and J.H. Andrews, "On Sufficiency of Mutants," Proc. 29th Int'l Conf. Software Eng., pp. 73-74, May 2007.
[170] A.S. Namin, J.H. Andrews, and D.J. Murdoch, "Sufficient Mutation Operators for Measuring Test Effectiveness," Proc. 30th Int'l Conf. Software Eng., pp. 351-360, May 2008.
[171] R. Nilsson, A.J. Offutt, and S.F. Andler, "Mutation-Based Testing Criteria for Timeliness," Proc. 28th Ann. Int'l Computer Software and Applications Conf., pp. 306-311, Sept. 2004.
[172] R. Nilsson, A.J. Offutt, and J. Mellin, "Test Case Generation for Mutation-Based Testing of Timeliness," Proc. Second Workshop Model Based Testing, pp. 97-114, Mar. 2006.
[173] A.J. Offutt, J. Pan, and J.M. Voas, "Procedures for Reducing the Size of Coverage-Based Test Sets," Proc. 12th Int'l Conf. Testing Computer Software, pp. 111-123, June 1995.
[174] A.J. Offutt, "The Coupling Effect: Fact or Fiction," ACM SIGSOFT Software Eng. Notes, vol. 14, no. 8, pp. 131-140, Dec. 1989.
[175] A.J. Offutt, "Investigations of the Software Testing Coupling Effect," ACM Trans. Software Eng. and Methodology, vol. 1, no. 1, pp. 5-20, Jan. 1992.
[176] A.J. Offutt, private communication, July 2008.
[177] A.J. Offutt, P. Ammann, and L.L. Liu, "Mutation Testing Implements Grammar-Based Testing ," Proc. Second Workshop Mutation Analysis, p. 12, Nov. 2006.
[178] A.J. Offutt and W.M. Craft, "Using Compiler Optimization Techniques to Detect Equivalent Mutants," Software Testing, Verification, and Reliability, vol. 4, no. 3, pp. 131-154, Sept. 1994.
[179] A.J. Offutt, Z. Jin, and J. Pan, "The Dynamic Domain Reduction Approach for Test Data Generation: Design and Algorithms," Technical Report ISSE-TR-94-110, George Mason Univ., 1994.
[180] A.J. Offutt, Z. Jin, and J. Pan, "The Dynamic Domain Reduction Procedure for Test Data Generation," Software: Practice and Experience, vol. 29, no. 2, pp. 167-193, Feb. 1999.
[181] A.J. Offutt and K.N. King, "A Fortran 77 Interpreter for Mutation Analysis," ACM SIGPLAN Notices, vol. 22, no. 7, pp. 177-188, July 1987.
[182] A.J. Offutt, A. Lee, G. Rothermel, R.H. Untch, and C. Zapf, "An Experimental Determination of Sufficient Mutant Operators," ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 99-118, Apr. 1996.
[183] A.J. Offutt and S. Lee, "An Empirical Evaluation of Weak Mutation," IEEE Trans. Software Eng., vol. 20, no. 5, pp. 337-344, May 1994.
[184] A.J. Offutt and S.D. Lee, "How Strong is Weak Mutation?" Proc. Fourth Symp. Software Testing, Analysis, and Verification, pp. 200-213, Oct. 1991.
[185] A.J. Offutt, Y.-S. Ma, and Y.-R. Kwon, "An Experimental Mutation System for Java," ACM SIGSOFT Software Eng. Notes, vol. 29, no. 5, pp. 1-4, Sept. 2004.
[186] A.J. Offutt and J. Pan, "Detecting Equivalent Mutants and the Feasible Path Problem," Proc. Ann. Conf. Computer Assurance, pp. 224-236, June 1996.
[187] A.J. Offutt and J. Pan, "Automatically Detecting Equivalent Mutants and Infeasible Paths," Software Testing, Verification, and Reliability, vol. 7, no. 3, pp. 165-192, Sept. 1997.
[188] A.J. Offutt, J. Pan, K. Tewary, and T. Zhang, "An Experimental Evaluation of Data Flow and Mutation Testing," Software: Practice and Experience, vol. 26, no. 2, pp. 165-176, Feb. 1996.
[189] A.J. Offutt, R.P. Pargas, S.V. Fichter, and P.K. Khambekar, "Mutation Testing of Software Using a MIMD Computer," Proc. Int'l Conf. Parallel Processing, pp. 255-266, Aug. 1992.
[190] A.J. Offutt, G. Rothermel, and C. Zapf, "An Experimental Evaluation of Selective Mutation," Proc. 15th Int'l Conf. Software Eng., pp. 100-107, May 1993.
[191] A.J. Offutt and R.H. Untch, "Mutation 2000: Uniting the Orthogonal," Proc. First Workshop Mutation Analysis, pp. 34-44, Oct. 2000.
[192] A.J. Offutt, J. Voas, and J. Payn, "Mutation Operators for Ada," Technical Report ISSE-TR-96-09, George Mason Univ., 1996.
[193] A.J. Offutt and W. Xu, "Generating Test Cases for Web Services Using Data Perturbation," Proc. Workshop Testing, Analysis and Verification of Web Services, pp. 1-10, July 2004.
[194] A.J. Offutt, "Automatic Test Data Generation," PhD thesis, Georgia Inst. of Tech nology, 1988.
[195] V. Okun, "Specification Mutation for Test Generation and Analysis," PhD thesis, Univ. of Maryland, 2004.
[196] T. Olsson and P. Runeson, "System Level Mutation Analysis Applied to a State-Based Language," Proc. Eighth Ann. IEEE Int'l Conf. and Workshop Eng. of Computer Based Systems, pp. 222-228, Apr. 2001.
[197] J. Pan, "Using Constraints to Detect Equivalent Mutants," master's thesis, George Mason Univ., 1994.
[198] Parasoft, "Parasoft Insure++," http://www.parasoft.com/jsp/productshome.jsp?product=Insure , 2006.
[199] M. Polo, M. Piattini, and I. Garcia-Rodriguez, "Decreasing the Cost of Mutation Testing with Second-Order Mutants," Software Testing, Verification, and Reliability, vol. 19, no. 2, pp. 111-131, June 2008.
[200] M. Polo, S. Tendero, and M. Piattini, "Integrating Techniques and Tools for Testing Automation: Research Articles," Software Testing, Verification, and Reliability, vol. 17, no. 1, pp. 3-39, Mar. 2007.
[201] A. Pretschner, T. Mouelhi, and Y.L. Traon, "Model-Based Tests for Access Control Policies," Proc. First Int'l Conf. Software Testing, Verification, and Validation, pp. 338-347, Apr. 2008.
[202] R. Probert and F. Guo, "Mutation Testing of Protocols: Principles and Preliminary Experimental Results," Proc. Workshop Protocol Test Systems, pp. 57-76, Oct. 1991.
[203] S.T. Redwine and W.E. Riddle, "Software Technology Maturation," Proc. Eighth Int'l Conf. Software Eng., pp. 189-200, 1985.
[204] C.K. Roy and J.R. Cordy, "Towards a Mutation-Based Automatic Framework for Evaluating Code Clone Detection Tools ," Proc. Canadian Conf. Computer Science and Software Eng., pp. 137-140, May 2008.
[205] C.K. Roy and J.R. Cordy, "A Mutation/Injection-Based Automatic Framework for Evaluating Code Clone Detection Tools," Proc. Fourth Int'l Workshop Mutation Analysis, published with Proc. Second Int'l Conf. Software Testing, Verification, and Validation Workshops, pp. 157-166, Apr. 2009.
[206] Rubyforge, "Heckle," http://seattlerb.rubyforge.orgheckle/, 2007.
[207] M. Sahinoglu and E.H. Spafford, "A Bayes Sequential Statistical Procedure for Approving Software Products," Proc. IFIP Conf. Approving Software Products, pp. 43-56, Sept. 1990.
[208] D. Schuler, V. Dallmeier, and A. Zeller, "Efficient Mutation Testing by Checking Invariant Violations," Proc. Int'l Symp. Software Testing and Analysis, July 2009.
[209] K. Sen, D. Marinov, and G. Agha, "CUTE: A Concolic Unit Testing Engine for C," Proc. 13th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 263-272, 2005.
[210] Y. Serrestou, V. Beroulle, and C. Robach, "Functional Verification of RTL Designs Driven by Mutation Testing Metrics," Proc. 10th Euromicro Conf. Digital System Design Architectures, Methods and Tools, pp. 222-227, Aug. 2007.
[211] Y. Serrestou, V. Beroulle, and C. Robach, "Impact of Hardware Emulation on the Verification Quality Improvement," Proc. IFIP WG 10.5 Int'l Conf. Very Large Scale Integration of System-on-Chip, pp. 218-223, Oct. 2007.
[212] H. Shahriar and M. Zulkernine, "MUSIC: Mutation-Based SQL Injection Vulnerability Checking," Proc. Eighth Int'l Conf. Quality Software, pp. 77-86, Aug. 2008.
[213] H. Shahriar and M. Zulkernine, "Mutation-Based Testing of Buffer Overflow Vulnerabilities," Proc. Second Ann. IEEE Int'l Workshop Security in Software Eng., pp. 979-984, July-Aug. 2008.
[214] H. Shahriar and M. Zulkernine, "Mutation-Based Testing of Format String Bugs," Proc. 11th IEEE High Assurance Systems Eng. Symp., pp. 229-238, Dec. 2008.
[215] H. Shahriar and M. Zulkernine, "MUTEC: Mutation-Based Testing of Cross Site Scripting," Proc. Fifth Int'l Workshop Software Eng. for Secure Systems, pp. 47-53, May 2009.
[216] D.P. Sidhu and T.K. Leung, "Fault Coverage of Protocol Test Methods," Proc. IEEE INFOCOM, pp. 80-85, Mar. 1988.
[217] A. Simao, J.C. Maldonado, and R. da Silva Bigonha, "A Transformational Language for Mutant Description," Computer Languages, Systems, and Structures, vol. 35, no. 3, pp. 322-339, Oct. 2009.
[218] B.H. Smith and L. Williams, "An Empirical Evaluation of the MuJava Mutation Operators," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 193-202, Sept. 2007.
[219] B.H. Smith and L. Williams, "On Guiding the Augmentation of an Automated Test Suite via Mutation Analysis," Empirical Software Eng., vol. 14, no. 3, pp. 341-369, 2009.
[220] SourceForge, "Nester," http:/nester.sourceforge.net/, 2002.
[221] SourceForge, "Jumble," http:/jumble.sourceforge.net/, 2007.
[222] S.D.R.S.D. Souza, J.C. Maldonado, S.C.P.F. Fabbri, and W.L.D. Souza, "Mutation Testing Applied to Estelle Specifications," Software Quality Control, vol. 8, no. 4, pp. 285-301, Dec. 1999.
[223] S.D.R.S.D. Souza, J.C. Maldonado, S.C.P.F. Fabbri, and W.L.D. Souza, "Mutation Testing Applied to Estelle Specifications," Proc. 33rd Hawaii Int'l Conf. System Sciences, vol. 8, p. 8011, Jan. 2000.
[224] E.H. Spafford, "Extending Mutation Testing to Find Environmental Bugs," Software: Practice and Experience, vol. 20, no. 2, pp. 181-189, Feb. 1990.
[225] T. Srivatanakul, J.A. Clark, S. Stepney, and F. Polack, "Challenging Formal Specifications by Mutation: A CSP Security Example," Proc. 10th Asia-Pacific Software Eng. Conf., pp. 340-350, Dec. 2003.
[226] T. Sugeta, J.C. Maldonado, and W.E. Wong, "Mutation Testing Applied to Validate SDL Specifications," Proc. 16th IFIP Int'l Conf. Testing of Comm. Systems, p. 2741, Mar. 2004.
[227] A. Sung, J. Jang, and B. Choi, "Fault-Based Interface Testing between Real-Time Operating System and Application," Proc. Second Workshop Mutation Analysis, p. 8, Nov. 2006.
[228] A. Tanaka, "Equivalence Testing for Fortran Mutation System Using Data Flow Analysis," master's thesis, Georgia Inst. of Tech nology, 1981.
[229] P. Thévenod-Fosse, H. Waeselynck, and Y. Crouzet, "An Experimental Study on Software Structural Testing: Deterministic versus Random Input Generation," Proc. 25th Int'l Symp. Fault-Tolerant Computing, pp. 410-417, June 1991.
[230] N. Tillmann and J. de Halleux, "Pex-White Box Test Generation for .NET," Proc. Second Int'l Conf. Tests and Proofs, pp. 134-153, Apr. 2008.
[231] M. Trakhtenbrot, "New Mutations for Evaluation of Specification and Implementation Levels of Adequacy in Testing of Statecharts Models," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 151-160, Sept. 2007.
[232] U. Trier, "DBLP," http://www.informatik.uni-trier.de/leydb /, 2009.
[233] J. Tuya, M.J.S. Cabal, and C. de la Riva, "SQLMutation: A Tool to Generate Mutants of SQL Database Queries," Proc. Second Workshop Mutation Analysis, p. 1, Nov. 2006.
[234] J. Tuya, M.J.S. Cabal, and C. de la Riva, "Mutating Database Queries," Information and Software Technology, vol. 49, no. 4, pp. 398-417, Apr. 2007.
[235] R.H. Untch, "Mutation-Based Software Testing Using Program Schemata," Proc. 30th Ann. Southeast Regional Conf., pp. 285-291, 1992.
[236] R.H. Untch, "Schema-Based Mutation Analysis: A New Test Data Adequacy Assessment Method," PhD thesis, Clemson Univ., Dec. 1995.
[237] R.H. Untch, A.J. Offutt, and M.J. Harrold, "Mutation Analysis Using Mutant Schemata," Proc. Int'l Symp. Software Testing and Analysis, pp. 139-148, 1993.
[238] G. Vigna, W. Robertson, and D. Balzarotti, "Testing Network-Based Intrusion Detection Signatures Using Mutant Exploits," Proc. 11th ACM Conf. Computer and Comm. Security, pp. 21-30, 2004.
[239] P. Vilela, M. Machado, and W.E. Wong, "Testing for Security Vulnerabilities in Software," Proc. Conf. Software Eng. and Applications, 2002.
[240] A.M.R. Vincenzi, J.C. Maldonado, E.F. Barbosa, and M.E. Delamaro, "Unit and Integration Testing Strategies for C Programs Using Mutation," Software Testing, Verification, and Reliability, vol. 11, no. 4, pp. 249-268, Nov. 2001.
[241] J. Voas and G. McGraw, Software Fault Injection: Inoculating Programs against Errors. John Wiley & Sons, 1997.
[242] K.S.H.T. Wah, "Fault Coupling in Finite Bijective Functions," Software Testing, Verification, and Reliability, vol. 5, no. 1, pp. 3-47, 1995.
[243] K.S.H.T. Wah, "A Theoretical Study of Fault Coupling," Software Testing, Verification, and Reliability, vol. 10, no. 1, pp. 3-46, Apr. 2000.
[244] K.S.H.T. Wah, "An Analysis of the Coupling Effect I: Single Test Data," Science of Computer Programming, vol. 48, nos. 2-3, pp. 119-161, Aug.-Sept. 2003.
[245] R. Wang and N. Huang, "Requirement Model-Based Mutation Testing for Web Service," Proc. Fourth Int'l Conf. Next Generation Web Services Practices, pp. 71-76, Oct. 2008.
[246] S.N. Weiss and V.N. Fleyshgakker, "Improved Serial Algorithms for Mutation Analysis," ACM SIGSOFT Software Eng. Notes, vol. 18, no. 3, pp. 149-158, July 1993.
[247] E.J. Weyuker, "On Testing Non-Testable Programs," The Computer J., vol. 25, pp. 456-470, 1982.
[248] W.E. Wong, "On Mutation and Data Flow," PhD thesis, Purdue Univ., 1993.
[249] W.E. Wong, J.R. Horgan, S. London, and A.P. Mathur, "Effect of Test Set Minimization on Fault Detection Effectiveness," Software: Practice and Experience, vol. 28, pp. 347-369, 1998.
[250] W.E. Wong, J.R. Horgan, A.P. Mathur, and A. Pasquini, "Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application," J. Systems and Software, vol. 48, no. 2, pp. 79-89, Oct. 1999.
[251] W.E. Wong and A.P. Mathur, "Fault Detection Effectiveness of Mutation and Data Flow Testing," Software Quality J., vol. 4, no. 1, pp. 69-83, Mar. 1995.
[252] W.E. Wong and A.P. Mathur, "Reducing the Cost of Mutation Testing: An Empirical Study," J. Systems and Software, vol. 31, no. 3, pp. 185-196, Dec. 1995.
[253] M.R. Woodward, "Mutation Testing—An Evolving Technique," Proc. IEE Colloquium on Software Testing for Critical Systems, pp. 3/1-3/6, June 1990.
[254] M.R. Woodward, "OBJTEST: An Experimental Testing Tool for Algebraic Specifications," Proc. IEE Colloquium on Automating Formal Methods for Computer Assisted Prototying, p. 2, Jan. 1990.
[255] M.R. Woodward, "Errors in Algebraic Specifications and an Experimental Mutation Testing Tool," Software Eng. J., vol. 8, no. 4, pp. 221-224, July 1993.
[256] M.R. Woodward, "Mutation Testing—Its Origin and Evolution," J. Information and Software Technology, vol. 35, no. 3, pp. 163-169, Mar. 1993.
[257] M.R. Woodward and K. Halewood, "From Weak to Strong Dead or Alive? An Analysis of Some Mutationtesting Issues," Proc. Second Workshop Software Testing, Verification, and Analysis, pp. 152-158, July 1988.
[258] T. Xie, N. Tillmann, J. de Halleux, and W. Schulte, "Mutation Analysis of Parameterized Unit Tests," Proc. Fourth Int'l Workshop Mutation Analysis, published with Proc. Second Int'l Conf. Software Testing, Verification, and Validation Workshops, pp. 177-181, Apr. 2009.
[259] W. Xu, A.J. Offutt, and J. Luo, "Testing Web Services by XML Perturbation," Proc. 16th IEEE Int'l Symp. Software Reliability Eng., pp. 257-266, July 2005.
[260] H. Yoon, B. Choi, and J.O. Jeon, "Mutation-Based Inter-Class Testing," Proc. Fifth Asia Pacific Software Eng. Conf., pp. 174-181, Dec. 1998.
[261] C.N. Zapf, "A Distributed Interpreter for the Mothra Mutation Testing System," master's thesis, Clemson Univ., 1993.
[262] Y. Zhan and J.A. Clark, "Search-Based Mutation Testing for Simulink Models," Proc. Conf. Genetic and Evolutionary Computation, pp. 1061-1068, June 2005.
[263] S. Zhang, T.R. Dean, and G.S. Knight, "Lightweight State Based Mutation Testing for Security," Proc. Third Workshop Mutation Analysis, published with Proc. Second Testing: Academic and Industrial Conf. Practice and Research Techniques, pp. 223-232, Sept. 2007.
[264] C. Zhou and P. Frankl, "Mutation Testing for Java Database Applications," Proc. Second Int'l Conf. Software Testing Verification and Validation, pp. 396-405, Apr. 2009.

Index Terms:
Mutation testing, survey.
Citation:
Yue Jia, Mark Harman, "An Analysis and Survey of the Development of Mutation Testing," IEEE Transactions on Software Engineering, vol. 37, no. 5, pp. 649-678, Sept.-Oct. 2011, doi:10.1109/TSE.2010.62
Usage of this product signifies your acceptance of the Terms of Use.