The Community for Technology Leaders
RSS Icon
Issue No.06 - June (2013 vol.39)
pp: 787-805
Richard Baker , Aero Engine Controls, Birmingham
Ibrahim Habli , University of York, York
Testing provides a primary means for assuring software in safety-critical systems. To demonstrate, particularly to a certification authority, that sufficient testing has been performed, it is necessary to achieve the test coverage levels recommended or mandated by safety standards and industry guidelines. Mutation testing provides an alternative or complementary method of measuring test sufficiency, but has not been widely adopted in the safety-critical industry. In this study, we provide an empirical evaluation of the application of mutation testing to airborne software systems which have already satisfied the coverage requirements for certification. Specifically, we apply mutation testing to safety-critical software developed using high-integrity subsets of C and Ada, identify the most effective mutant types, and analyze the root causes of failures in test cases. Our findings show how mutation testing could be effective where traditional structural coverage analysis and manual peer review have failed. They also show that several testing issues have origins beyond the test activity, and this suggests improvements to the requirements definition and coding process. Our study also examines the relationship between program characteristics and mutation survival and considers how program size can provide a means for targeting test areas most likely to have dormant faults. Industry feedback is also provided, particularly on how mutation testing can be integrated into a typical verification life cycle of airborne software.
Testing, Certification, Software systems, Safety, Industries, Guidelines, certification, Mutation, safety-critical software, verification, testing
Richard Baker, Ibrahim Habli, "An Empirical Evaluation of Mutation Testing for Improving the Test Quality of Safety-Critical Software", IEEE Transactions on Software Engineering, vol.39, no. 6, pp. 787-805, June 2013, doi:10.1109/TSE.2012.56
[1] H. Agrawal, R.A. DeMillo, B. Hathaway, W. Hsu, W. Hsu, E.W. Krauser, R.J. Martin, A.P. Mathur, and E. Spafford, "Design of Mutant Operators for the C Programming Language," Technical Report SERC-TR-41P, Purdue Univ., West Lafayette, Ind., Mar. 1989.
[2] J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. IEEE Int'l Conf. Software Eng., pp. 402-411, 2005.
[3] J.H. Andrews, L.C. Briand, and Y. Labiche, A.S. Namin, "Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria," IEEE Trans. Software Eng., vol. 32, no. 8, pp. 608-624, Aug. 2006.
[4] D. Baldwin and F.G. Sayward, "Heuristics for Determining Equivalence of Program Mutations," Research Report 276, Yale Univ., New Haven, Conn., 1979.
[5] E. Barbosa, J.C. Maldonado, and A. Vincenzi, "Toward the Determination of Sufficient Mutant Operators for C," Software Testing, Verification, and Reliability, vol. 11, pp. 113-136, 2001.
[6] J. Barnes, High Integrity Software: The SPARK Approach to Safety and Security. Addison-Wesley, 2003.
[7] R. Butler and G. Finelli, "The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software," IEEE Trans. Software Eng., vol. 19, no. 1, pp. 3-12, Jan. 1993.
[8] J.J. Chilenski and S.P. Miller, "Applicability of Modified Condition/Decision Coverage to Software Testing," Software Eng. J., vol. 9, no. 5, pp. 193-200, 1994.
[9] J.J. Chilenski, "An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion," Report DOT/FAA/AR-01/18, Office of Aviation Research, Washington, D.C., Apr. 2001.
[10] H. Do and G.E Rothermel, "On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques," IEEE Trans. Software Eng., vol. 32, no. 9, pp. 733-752, Aug. 2006.
[11] D. Daniels, R. Myers, and A. Hilton, "White Box Software Development," Proc. 11th Safety-Critical Systems Symp., Feb. 2003.
[12] M. Daran and P. Thévenod-Fosse, "Software Error Analysis: A Real Case Study Involving Real Faults and Mutations," ACM SIGSOFT Software Eng. Notes, vol. 21, no. 3, pp. 158-177, May 1996.
[13] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Hints on Test Data Selection: Help for the Practical Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
[14] K. Hayhurst, D.S. Veerhusen, J.J. Chilenski, and L.K. Rierson, "A Practical Tutorial Decision Coverage," NASA Report, NASA/TM-2001-210876, 2001.
[15] C.R.M. Hierons, M. Harman, and S. Danicic, "Using Program Slicing to Assist in the Detection of Equivalent Mutants," Software Testing, Verification, and Reliability, vol. 9, no. 4, pp. 233-262, Dec. 1999.
[16] Functional Safety of Electrical/Electronic/Programmable Electronic Safety-Related Systems, IEC 61508, Int'l Electrotechnical Commission, Mar. 2010.
[17] ISO 26262: Road Vehicles: Functional Safety, Int'l Organization for Standardization, June 2011.
[18] G. Jay, J.E. Hale, R.K. Smith, D.P. Hale, N.A. Kraft, and C. Ward, "Cyclomatic Complexity and Lines of Code: Empirical Evidence of a Stable Linear Relationship," J. Software Eng. and Applications, vol. 3, no. 2, pp. 137-143, 2009.
[19] Y. Jia and M. Harman, "Higher Order Mutation Testing," Information and Software Technology, vol. 51, no. 10, pp. 1379-1393, 2009.
[20] Y. Jia and M. Harman, "MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language," Proc. Third Testing Academia and Industry Conf.— Practice and Research Techniques, Aug. 2008.
[21] Y. Jia and M. Harman, "An Analysis and Survey of the Development of Mutation Testing, Software Engineering," IEEE Trans. Software Eng., vol. 37, no. 5, pp. 649-678, Sept./Oct. 2011.
[22] J.A. Jones and M.J. Harrold, "Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage," IEEE Trans. Software Eng., vol. 29, no. 3, pp. 195-209, Mar. 2003.
[23] B. Littlewood and L. Strigini, "Validation of Ultrahigh Dependability for Software-Based Systems," Comm. ACM, vol. 36, no. 11, pp. 69-80, 1993.
[24] T.J. McCabe, "A Complexity Measure," IEEE Trans. Software Eng., vol. 2, no. 4, pp. 308-320, Dec. 1976.
[25] T.J. McCabe and A.H. Watson, "Combining Comprehension and Testing in Object-Oriented Development," Object Magazine, vol. 4, pp. 63-64, Mar./Apr. 1994.
[26] J.A. McDermid, Software Engineer's Reference Book. Butterworth-Heinemann Newton, 1991.
[27] V.D. Meulen and M.A. Revilla, "Correlations between Internal Software Metrics and Software Dependability in a Large Population of Small C/C++ Programs," Proc. 18th IEEE Int'l Symp. Reliability, Nov. 2007.
[28] MISRA, "Guidelines for the Use of the C Language in Critical Systems," Oct. 2004.
[29] P.R. Muessig, "Cost vs Credibility: How Much V&V Is Enough?" Naval Air Warfare Center, Weapons Division, China Lake, Calif., 2001.
[30] G.J. Myers, The Art of Software Testing. John Wiley & Sons, 2004.
[31] A.J. Offutt, J. Voas, and J. Payne, "Mutation Operators for Ada," Technical Report ISSE-TR-96-09, Information and Software Systems Eng., George Mason Univ., 1996.
[32] A.J. Offutt, A. Lee, G. Rothermel, R.H. Untch, and C. Xapf, "An Experimental Determination of Sufficient Mutant Operators," ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 99-118, Apr. 1996.
[33] A.J. Offutt and J. Pan, "Automatically Detecting Equivalent Mutants and Infeasible Paths," Software Testing, Verification, and Reliability, vol. 7, no. 3, pp. 165-192, Sept. 1997.
[34] A.J. Offutt and R.H. Untch, "Mutation 2000: Uniting the Orthogonal," Proc. Mutation 2000: Mutation Testing in the Twentieth and the Twenty-First Centuries, 2000.
[35] "DO-178B Software Considerations in Airborne Systems and Equipment Certification," RTCA, Washington, D.C., 1992.
[36] M. Shepperd, "A Critique of Cyclomatic Complexity as a Software Metric," Software Eng. J., vol. 3, no. 2, pp. 30-36, Mar. 1988.
[37] "Aerospace Recommended Practice 4754: Certification Considerations for Highly-Integrated or Complex Aircraft Systems," Soc. of Automotive Eng. (SAE), Nov. 1996.
[38] "ARP4761—Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment," Soc. of Automotive Eng. (SAE), 1996.
[39] M. Umar, "An Evaluation of Mutation Operators for Equivalent Mutants," master's thesis, King's College, London, 2006.
[40] A.H. Watson and T.J. McCabe, "Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric," Computer Systems Laboratory, Nat'l Inst. of Standards and Technology, Gaithersburg, Md., Sept. 1996.
[41] W.E. Wong, J.C. Maldonado, M.E. Delamaro, and S.R.S Souza, "A Comparison of Selective Mutation in C and Fortran," Proc. Workshop Validation and Testing of Operational Systems Project, Jan. 1997.
31 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool