This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Mutation Operators for Spreadsheets
January/February 2009 (vol. 35 no. 1)
pp. 94-108
Robin Abraham, Oregon State University, Corvallis
Martin Erwig, Oregon State University, Corvallis
Based on (1) research into mutation testing for general purpose programming languages, and (2) spreadsheet errors that have been reported in the literature, we have developed a suite of mutation operators for spreadsheets. We present an evaluation of the mutation adequacy of du-adequate test suites generated by a constraint-based automatic test-case generation system we have developed in previous work. The results of the evaluation suggest additional constraints that can be incorporated into the system to target mutation adequacy. In addition to being useful in mutation testing of spreadsheets, the operators can be used in the evaluation of error-detection tools and also for seeding spreadsheets with errors for empirical studies. We describe two case studies where the suite of mutation operators helped us carry out such empirical evaluations. The main contribution of this paper is a suite of mutation operators for spreadsheets that can be used for carrying out empirical evaluations of spreadsheet tools to indicate ways in which the tools can be improved.

[1] C. Scaffidi, M. Shaw, and B. Myers, “Estimating the Numbers of End Users and End User Programmers,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 207-214, 2005.
[2] S. Ditlea, “Spreadsheets Can Be Hazardous to Your Health,” Personal Computing, vol. 11, no. 1, pp. 60-69, 1987.
[3] K. Rajalingham, D. Chadwick, B. Knight, and D. Edwards, “Quality Control in Spreadsheets: A Software Engineering-Based Approach to Spreadsheet Development,” Proc. 33rd Hawaii Int'l Conf. System Sciences, pp. 1-9, 2000.
[4] G.J. Croll, “The Importance and Criticality of Spreadsheets in the City of London,” Proc. Symp. European Spreadsheet Risks Interest Group, 2005.
[5] European Spreadsheet Risks Interest Group, EuSpRIG, http:/www.eusprig.org/, 2008.
[6] R.R. Panko and R.P. Halverson Jr., “Spreadsheets on Trial: A Survey of Research on Spreadsheet Risks,” Proc. 29th Hawaii Int'l Conf. System Sciences, 1996.
[7] R.R. Panko, “Spreadsheet Errors: What We Know. What We Think We Can Do,” Proc. Symp. European Spreadsheet Risks Interest Group, 2000.
[8] B. Ronen, M.A. Palley, and H.C. Lucas Jr., “Spreadsheet Analysis and Design,” Comm. ACM, vol. 32, no. 1, pp. 84-93, 1989.
[9] A.G. Yoder and D.L. Cohn, “Real Spreadsheets for Real Programmers,” Proc. Int'l Conf. Computer Languages, pp. 20-30, 1994.
[10] T. Isakowitz, S. Schocken, and H.C. Lucas Jr., “Toward a Logical/Physical Theory of Spreadsheet Modelling,” ACM Trans. Information Systems, vol. 13, no. 1, pp. 1-37, 1995.
[11] S.G. Powell and K.R. Baker, The Art of Modeling with Spreadsheets: Management Science, Spreadsheet Engineering, and Modeling Craft. Wiley, 2004.
[12] R.R. Panko, “Applying Code Inspection to Spreadsheet Testing,” J.Management Information Systems, vol. 16, no. 2, pp. 159-176, 1999.
[13] J. Sajaniemi, “Modeling Spreadsheet Audit: A Rigorous Approach to Automatic Visualization,” J. Visual Languages and Computing, vol. 11, no. 1, pp. 49-82, http:/www.idealibrary.com, 2000.
[14] R. Mittermeir and M. Clermont, “Finding High-Level Structures in Spreadsheet Programs,” Proc. Ninth Working Conf. Reverse Eng., pp. 221-232, 2002.
[15] M. Erwig and M.M. Burnett, “Adding Apples and Oranges,” Proc. Fourth Int'l Symp. Practical Aspects of Declarative Languages, pp. 173-191, 2002.
[16] Y. Ahmad, T. Antoniu, S. Goldwater, and S. Krishnamurthi, “A Type System for Statically Detecting Spreadsheet Errors,” Proc. 18th IEEE Int'l Conf. Automated Software Eng., pp. 174-183, 2003.
[17] T. Antoniu, P.A. Steckler, S. Krishnamurthi, E. Neuwirth, and M. Felleisen, “Validating the Unit Correctness of Spreadsheet Programs,” Proc. 26th IEEE Int'l Conf. Software Eng., pp. 439-448, 2004.
[18] M.M. Burnett, C. Cook, J. Summet, G. Rothermel, and C. Wallace, “End-User Software Engineering with Assertions,” Proc. 25th IEEE Int'l Conf. Software Eng., pp. 93-103, 2003.
[19] M.J. Coblenz, A.J. Ko, and B.A. Myers, “Using Objects of Measurement to Detect Spreadsheet Errors,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 314-316, 2005.
[20] R. Abraham and M. Erwig, “Header and Unit Inference for Spreadsheets through Spatial Analyses,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 165-172, 2004.
[21] M. Erwig, R. Abraham, I. Cooperstein, and S. Kollmansberger, “Automatic Generation and Maintenance of Correct Spreadsheets,” Proc. 27th IEEE Int'l Conf. Software Eng., pp. 136-145, 2005.
[22] G. Engels and M. Erwig, “ClassSheets: Automatic Generation of Spreadsheet Applications from Object-Oriented Specifications,” Proc. 20th IEEE/ACM Int'l Conf. Automated Software Eng., pp. 124-133, 2005.
[23] G. Rothermel, M.M. Burnett, L. Li, C. DuPuis, and A. Sheretov, “A Methodology for Testing Spreadsheets,” ACM Trans. Software Eng. and Methodology, vol. 10, no. 2, pp. 110-147, 2001.
[24] M. Fisher II, M. Cao, G. Rothermel, C. Cook, and M.M. Burnett, “Automated Test Case Generation for Spreadsheets,” Proc. 24th IEEE Int'l Conf. Software Eng., pp. 141-151, 2002.
[25] M.M. Burnett, A. Sheretov, B. Ren, and G. Rothermel, “Testing Homogeneous Spreadsheet Grids with the “What You See Is What You Test” Methodology,” IEEE Trans. Software Eng., vol. 28, no. 6, pp. 576-594, June 2002.
[26] End Users Shaping Effective Software, EUSES, http:/EUSESconsortium.org, 2008.
[27] R.G. Hamlet, “Testing Programs with the Aid of a Compiler,” IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279-290, July 1977.
[28] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, vol. 11, no. 4, pp. 34-41, 1978.
[29] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, “Program Mutation: A New Approach to Program Testing,” Infotech State of the Art Report, Software Testing, vol. 2, pp. 107-126, 1979.
[30] T.A. Budd, F.G. Sayward, R.A. DeMillo, and R.J. Lipton, “The Design of a Prototype Mutation System for Program Testing,” Proc. Nat'l Computer Conf., pp. 623-627, 1978.
[31] A.T. Acree, “On Mutation,” PhD dissertation, Georgia Inst. of Tech nology, 1980.
[32] T.A. Budd, “Mutation Analysis of Program Test Data,” PhD dissertation, Yale Univ., 1980.
[33] T.A. Budd, R.A. DeMillo, R.J. Lipton, and F.G. Sayward, “Theoretical and Empirical Studies on Using Program Mutation to Test the Functional Correctness of Programs,” Proc. Seventh ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages, pp. 220-233, 1980.
[34] J.H. Andrews, L.C. Briand, and Y. Labiche, “Is Mutation an Appropriate Tool for Testing Experiments?” Proc. 27th IEEE Int'l Conf. Software Eng., pp. 402-411, 2005.
[35] E.J. Weyuker, S.N. Weiss, and D. Hamlet, “Comparison of Program Testing Strategies,” Proc. Fourth Symp. Software Testing, Analysis and Verification, pp. 154-164, 1991.
[36] P.J. Frankl and E.J. Weyuker, “A Formal Analysis of the Fault-Detecting Ability of Testing Methods,” IEEE Trans. Software Eng., vol. 19, no. 3, pp. 202-213, Mar. 1993.
[37] H. Zhu, P.A.V. Hall, and J.H.R. May, “Software Unit Test Coverage and Adequacy,” ACM Computing Surveys, vol. 29, no. 4, pp. 367-427, Dec. 1997.
[38] M.M. Burnett, J. Atwood, R. Djang, H. Gottfried, J. Reichwein, and S. Yang, “Forms/3: A First-Order Visual Language to Explore the Boundaries of the Spreadsheet Paradigm,” J. Functional Programming, vol. 11, no. 2, pp. 155-206, Mar. 2001.
[39] A.J. Offutt, J. Pan, K. Tewary, and T. Zhang, “An Experimental Evaluation of Data Flow and Mutation Testing,” Software: Practice and Experience, vol. 26, no. 2, pp. 165-176, 1996.
[40] J. Ruthruff, E. Creswick, M.M. Burnett, C. Cook, S. Prabhakararao, M. Fisher II, and M. Main, “End-User Software Visualizations for Fault Localization,” Proc. ACM Symp. Software Visualization, pp.123-132, 2003.
[41] S. Prabhakarao, C. Cook, J. Ruthruff, E. Creswick, M. Main, M. Durham, and M. Burnett, “Strategies and Behaviors of End-User Programmers with Interactive Fault Localization,” Proc. IEEE Int'l Symp. Human-Centric Computing Languages and Environments, pp.203-210, 2003.
[42] C. Allwood, “Error Detection Processes in Statistical Problem Solving,” Cognitive Science, vol. 8, no. 4, pp. 413-437, 1984.
[43] S.G. Powell, K.R. Baker, and B. Lawson, “A Critical Review of the Literature on Spreadsheet Errors,” working paper, http://mba.tuck.dartmouth.edu/spreadsheet product_pubs.html, 2006.
[44] R. Abraham and M. Erwig, “AutoTest: A Tool for Automatic Test Case Generation in Spreadsheets,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 43-50, 2006.
[45] R. Abraham and M. Erwig, “Goal-Directed Debugging of Spreadsheets,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 37-44, 2005.
[46] R. Abraham and M. Erwig, “GoalDebug: A Spreadsheet Debugger for End Users,” Proc. 29th IEEE Int'l Conf. Software Eng., pp. 251-260, 2007.
[47] J. Lawrence, R. Abraham, M.M. Burnett, and M. Erwig, “Sharing Reasoning about Faults in Spreadsheets: An Empirical Study,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 35-42, 2006.
[48] R. Abraham and M. Erwig, “UCheck: A Spreadsheet Unit Checker for End Users,” J. Visual Languages and Computing, vol. 18, no. 1, pp. 71-95, 2007.
[49] K.N. King and A.J. Offutt, “Fortran Language System for Mutation-Based Software Testing,” Software: Practice and Experience, vol. 21, no. 7, pp. 685-718, July 1991.
[50] M. Delamaro and J. Maldonado, “Proteum—A Tool for the Assessment of Test Adequacy for C Programs,” Proc. Conf. Performability in Computing Systems, pp. 79-95, 1996.
[51] Y. Ma, A.J. Offutt, and Y. Kwon, “MuJava: An Automated Class Mutation System,” Software Testing, Verification and Reliability, vol. 15, no. 2, pp. 97-133, 2005.
[52] P. Anbalagan and T. Xie, “Efficient Mutant Generation for Mutation Testing of Pointcuts in Aspect-Oriented Programs,” Proc. Second Workshop Mutation Analysis, 2006.
[53] N. Mansour and M. Houri, “Testing Web Applications,” Information and Software Technology, vol. 48, no. 1, pp. 31-42, 2006.
[54] W. Xu, A.J. Offutt, and J. Luo, “Testing Web Services by XML Perturbation,” Proc. IEEE Int'l Symp. Software Reliability Eng., pp.257-266, 2005.
[55] J. Tuya, M.J. Suarez-Cabal, and C. Riva, “Mutating Database Queries,” Information and Software Technology, vol. 49, no. 4, pp.398-417, 2007.
[56] A.J. Offutt, P. Ammann, and L. Liu, “Mutation Testing Implements Grammar-Based Testing,” Proc. Second Workshop Mutation Analysis, 2006.
[57] A.J. Offutt and R.H. Untch, “Mutation 2000: Uniting the Orthogonal,” Mutation 2000: Mutation Testing in the Twentieth and the Twenty-First Centuries, pp. 45-55, 2000.
[58] M.R. Woodward, “Mutation Testing—Its Origin and Evolution,” Information and Software Technology, vol. 35, no. 3, pp. 163-169, Mar. 1993.
[59] W.E. Wong, J.C. Maldonado, M.E. Delamaro, and S. Souza, “Use of Proteum to Accelerate Mutation Testing in C Programs,” Proc. Third ISSAT Int'l Conf. Reliability and Quality in Design, pp. 254-258, 1997.
[60] R.H. Untch, A.J. Offutt, and M.J. Harrold, “Mutation Analysis Using Program Schemata,” Proc. Int'l Symp. Software Testing and Analysis, pp. 139-148, 1993.
[61] W.E. Howden, “Weak Mutation Testing and Completeness of Test Sets,” IEEE Trans. Software Eng., vol. 8, no. 4, pp. 371-379, Apr. 1992.
[62] A.P. Mathur, “Performance, Effectiveness, and Reliability Issues in Software Testing,” Proc. 15th Ann. Int'l Computer Software and Applications Conf., pp. 604-605, 1991.
[63] A.J. Offutt, A. Lee, G. Rothermel, R.H. Untch, and C. Zapf, “An Experimental Determination of Sufficient Mutant Operators,” ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 99-118, 1996.
[64] R.A. DeMillo and A.J. Offutt, “Constraint-Based Automatic Test Data Generation,” IEEE Trans. Software Eng., vol. 17, no. 9, pp. 900-910, Sept. 1991.
[65] A.J. Offutt, “An Integrated Automatic Test Data Generation System,” J. Systems Integration, vol. 1, no. 3, pp. 391-409, 1991.
[66] A.J. Offutt and J. Pan, “Automatically Detecting Equivalent Mutants and Infeasible Paths,” Software Testing, Verification and Reliability, vol. 7, no. 3, pp. 165-192, 1997.
[67] R. Hierons, M. Harman, and S. Danicic, “Using Program Slicing to Assist in the Detection of Equivalent Mutants,” Software Testing, Verification and Reliability, vol. 9, no. 4, pp. 233-262, 1999.
[68] J.H. Andrews, L.C. Briand, Y. Labiche, and A.S. Namin, “Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria,” IEEE Trans. Software Eng., vol. 32, no. 8, pp. 608-624, Aug. 2006.
[69] M. Erwig, R. Abraham, S. Kollmansberger, and I. Cooperstein, “Gencel—A Program Generator for Correct Spreadsheets,” J.Functional Programming, vol. 16, no. 3, pp. 293-325, 2006.
[70] R. Abraham, M. Erwig, S. Kollmansberger, and E. Seifert, “Visual Specifications of Correct Spreadsheets,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 189-196, 2005.
[71] P. Saariluoma and J. Sajaniemi, “Transforming Verbal Descriptions into Mathematical Formulas in Spreadsheet Calculation,” Int'l J. Human-Computer Studies, vol. 41, no. 6, pp. 915-948, 1994.
[72] T. Teo and M. Tan, “Quantitative and Qualitative Errors in Spreadsheet Development,” Proc. 30th Hawaii Int'l Conf. System Sciences, pp. 25-38, 1997.
[73] T. Teo and M. Tan, “Spreadsheet Development and “What-if” Analysis: Quantitative versus Qualitative Errors,” Accounting Management and Information Technologies, vol. 9, pp. 141-160, 2000.
[74] D. Mason and D. Keane, “Spreadsheet Modeling in Practice: Solution or Problem,” Interface, pp. 82-84, 1989.
[75] T. Williams, “Spreadsheet Standards,” technical report, 1987.
[76] M.G. Simkin, “How to Validate Spreadsheets,” J. Accountancy, vol. 180, no. 11, pp. 130-138, 1987.
[77] A.J. Offutt, “Investigations of the Software Testing Coupling Effect,” ACM Trans. Software Eng. and Methodology, vol. 1, no. 1, pp.5-20, 1992.
[78] K.S. How Tai Wah, “A Theoretical Study of Fault Coupling,” Software Testing, Verification and Reliability, vol. 10, no. 1, pp. 3-45, 2000.
[79] K.S. How Tai Wah, “An Analysis of the Coupling Effect: I. Single Test Data,” Science of Computer Programming, vol. 48, nos. 2/3, pp.119-161, 2003.
[80] M. Purser and D. Chadwick, “Does an Awareness of Differing Types of Spreadsheet Errors Aid End-Users in Identifying Spreadsheet Errors,” Proc. Symp. European Spreadsheet Risks Interest Group, pp. 185-204, 2006.
[81] R. Abraham and M. Erwig, “Type Inference for Spreadsheets,” Proc. Eighth ACM-SIGPLAN Int'l Symp. Principles and Practice of Declarative Programming, pp. 73-84, 2006.
[82] M. Clermont, C. Hanin, and R. Mittermeir, “A Spreadsheet Auditing Tool Evaluated in an Industrial Context,” Spreadsheet Risks, Audit and Development Methods, vol. 3, pp. 35-46, 2002.
[83] M. Fisher II, G. Rothermel, D. Brown, M. Cao, C. Cook, and B. Burnett, “Integrating Automated Test Generation into the WYSIWYT Spreadsheet Testing Methodology,” ACM Trans. Software Eng. and Methodology, vol. 15, no. 2, pp. 150-194, 2006.
[84] R. Abraham and M. Erwig, “Inferring Templates from Spreadsheets,” Proc. 28th IEEE Int'l Conf. Software Eng., pp. 182-191, 2006.
[85] J.R. Ruthruff, S. Prabhakararao, J. Reichwein, C. Cook, E. Creswick, and M.M. Burnett, “Interactive, Visual Fault Localization Support for End-User Programmers,” J. Visual Languages and Computing, vol. 16, nos. 1/2, pp. 3-40, 2005.
[86] A. Phalgune, C. Kissinger, M. Burnett, C. Cook, L. Beckwith, and J. Ruthruff, “Garbage In, Garbage Out? An Empirical Look at Oracle Mistakes by End-User Programmers,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 45-52, 2005.

Index Terms:
Test coverage of code, Test design, Spreadsheets
Citation:
Robin Abraham, Martin Erwig, "Mutation Operators for Spreadsheets," IEEE Transactions on Software Engineering, vol. 35, no. 1, pp. 94-108, Jan.-Feb. 2009, doi:10.1109/TSE.2008.73
Usage of this product signifies your acceptance of the Terms of Use.