This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Interactive Fault Localization Techniques in a Spreadsheet Environment
April 2006 (vol. 32 no. 4)
pp. 213-239
End-user programmers develop more software than any other group of programmers, using software authoring devices such as multimedia simulation builders, e-mail filtering editors, by-demonstration macro builders, and spreadsheet environments. Despite this, there has been only a little research on finding ways to help these programmers with the dependability of the software they create. We have been working to address this problem in several ways, one of which includes supporting end-user debugging activities through interactive fault localization techniques. This paper investigates fault localization techniques in the spreadsheet domain, the most common type of end-user programming environment. We investigate a technique previously described in the research literature and two new techniques. We present the results of an empirical study to examine the impact of two individual factors on the effectiveness of fault localization techniques. Our results reveal several insights into the contributions such techniques can make to the end-user debugging process and highlight key issues of interest to researchers and practitioners who may design and evaluate future fault localization techniques.

[1] R. Abraham and M. Erwig, “Header and Unit Inference for Spreadsheets through Spatial Analyses,” Proc. IEEE Int'l Symp. Visual Languages and Human-Centric Computing, pp. 165-172, Sept. 2004.
[2] H. Agrawal and J.R. Horgan, “Dynamic Program Slicing,” Proc. ACM SIGPLAN 1990 Conf. Programming Language Design and Implementation, pp. 246-256, June 1990.
[3] H. Agrawal, J.R. Horgan, S. London, and W.E. Wong, “Fault Localization Using Execution Slices and Dataflow Tests,” Proc. Sixth IEEE Int'l Symp. Software Reliability Eng., pp. 143-151, Oct. 1995.
[4] Y. Ahmad, T. Antoniu, S. Goldwater, and S. Krishnamurthi, “A Type System for Statically Detecting Spreadsheet Errors,” Proc. 18th IEEE Int'l Conf. Automated Software Eng., pp. 174-183, Oct. 2003.
[5] C.M. Allwood, “Error Detection Processes in Statistical Problem Solving,” Cognitive Science, vol. 8, no. 4, pp. 413-437, Oct.-Dec. 1984.
[6] T. Antoniu, P.A. Steckler, S. Krishnamurthi, E. Neuwirth, and M. Felleisen, “Validating the Unit Correctness of Spreadsheet Programs,” Proc. 26th Int'l Conf. Software Eng., pp. 439-448, May 2004.
[7] Y. Ayalew and R. Mittermeir, “Spreadsheet Debugging,” Proc. European Spreadsheet Risks Interest Group, July 2003.
[8] M. Betts and A.S. Horowitz, “Oops! Audits Find Errors in 49 Out of 54 Spreadsheets,” Computerworld, p. 47, May 2004.
[9] B. Boehm and V.R. Basili, “Software Defect Reduction Top 10 List,” Computer, vol. 34, no. 1, pp. 135-137, Jan. 2001.
[10] D. Brown, M. Burnett, G. Rothermel, H. Fujita, and F. Negoro, “Generalizing WYSIWYT Visual Testing to Screen Transition Languages,” Proc. IEEE Symp. Human-Centric Computing, Languages, and Environments, pp. 203-210, Oct. 2003.
[11] P. Bunus and P. Fritzson, “Semi-Automatic Fault Localization and Behavior Verification for Physical System Simulation Models,” Proc. 18th IEEE Int'l Conf. Automated Software Eng., pp. 253-258, Oct. 2003.
[12] M. Burnett, J. Atwood, R. Djang, H. Gottfried, J. Reichwein, and S. Yang, “Forms/3: A First-Order Visual Language to Explore the Boundaries of the Spreadsheet Paradigm,” J. Functional Programming, vol. 11, no. 2, pp. 155-206, Mar. 2001.
[13] M. Burnett, C. Cook, O. Pendse, G. Rothermel, J. Summet, and C. Wallace, “End-User Software Engineering with Assertions in the Spreadsheet Paradigm,” Proc. 25th Int'l Conf. Software Eng., pp. 93-103, May 2003.
[14] M. Burnett, C. Cook, and G. Rothermel, “End-User Software Engineering,” Comm. ACM, vol. 47, no. 9, pp. 53-58, Sept. 2004.
[15] M. Burnett, A. Sheretov, and G. Rothermel, “Scaling Up a `What You See Is What You Test' Methodology to Spreadsheet Grids,” Proc. IEEE Symp. Visual Languages, pp. 30-37, Sept. 1999.
[16] T.Y. Chen and Y.Y. Cheung, “On Program Dicing,” Software Maintenance: Research and Practice, vol. 9, no. 1, pp. 33-46, Jan.-Feb. 1997.
[17] M. Clermont, “Analyzing Large Spreadsheet Programs,” Proc. 10th Working Conf. Reverse Eng., pp. 306-315, Nov. 2003.
[18] M. Clermont and R. Mittermeir, “Auditing Large Spreadsheet Programs,” Proc. Int'l Conf. Information Systems Implementation and Modelling, pp. 87-97, Apr. 2003.
[19] H. Cleve and A. Zeller, “Locating Causes of Program Failures,” Proc. 27th Int'l Conf. Software Eng., pp. 342-351, May 2005.
[20] C. Cook, M. Burnett, and D. Boom, “A Bug's Eye View of Immediate Visual Feedback in Direct-Manipulation Programming Systems,” Proc. Empirical Studies of Programmers: Seventh Workshop, pp. 20-41, Oct. 1997.
[21] P.T. Cox, F.R. Giles, and T. Pietrzykowski, “Prograph: A Step towards Liberating Programming from Textual Conditioning,” Proc. IEEE Workshop Visual Languages, pp. 150-156, Oct. 1989.
[22] R.A. DeMillo, H. Pan, and E.H. Spafford, “Critical Slicing for Software Fault Localization,” Proc. Int'l Symp. Software Testing and Analysis, pp. 121-134, Jan. 1996.
[23] E. Duesterwald, R. Gupta, and M.L. Soffa, “Rigorous Data Flow Testing through Output Influences,” Proc. Second Irvine Software Symp., pp. 131-145, Mar. 1992.
[24] M. Erwig and M. Burnett, “Adding Apples and Oranges,” Proc. Fourth Int'l Symp. Practical Aspects of Declarative Languages, pp. 173-191, Jan. 2002.
[25] M. Fisher II, D. Jin, G. Rothermel, and M. Burnett, “Test Reuse in the Spreadsheet Paradigm,” Proc. IEEE Int'l Symp. Software Reliability Eng., pp. 257-268, Nov. 2002.
[26] M. Fisher II and G. Rothermel, “The EUSES Spreadsheet Corpus: A Shared Resource for Supporting Experimentation with Spreadsheet Dependability Mechanisms,” Proc. First Workshop End-User Software Eng., pp. 47-51, May 2005.
[27] M. Fisher II, G. Rothermel, T. Creelan, and M. Burnett, “Scaling a Dataflow Testing Methodology to the Multiparadigm World of Commercial Spreadsheets,” Technical Report TR-UNL-CSE-2005-0003, Univ. of Nebraska-Lincoln, Sept. 2005.
[28] M. Francel and S. Rugaber, “Fault Localization Using Execution Traces,” Proc. ACM 30th Ann. Southeast Regional Conf., pp. 69-76, Apr. 1992.
[29] D.S. Hilzenrath, “Finding Errors a Plus, Fannie Says; Mortgage Giant Tries to Soften Effect of $1 Billion in Mistakes,” The Washington Post, 31 Oct. 2003.
[30] T. Igarashi, J.D. Mackinlay, B.W. Chang, and P.T. Zellweger, “Fluid Visualization of Spreadsheet Structures,” Proc. IEEE Symp. Visual Languages, pp. 118-125, Sept. 1998.
[31] J.A. Jones, M.J. Harrold, and J. Stasko, “Visualization of Test Information to Assist Fault Localization,” Proc. 24th Int'l Conf. Software Eng., pp. 467-477, May 2002.
[32] M. Karam and T. Smedley, “A Testing Methodology for a Dataflow Based Visual Programming Language,” Proc. IEEE Symp. Human-Centric Computing, Languages, and Environments, pp. 280-287, Sept. 2001.
[33] A.J. Ko and B.A. Myers, “Development and Evaluation of a Model of Programming Errors,” Proc. IEEE Symp. Human-Centric Computing, Languages, and Environments, pp. 7-14, Oct. 2003.
[34] A.J. Ko and B.A. Myers, “Designing the Whyline: A Debugging Interface for Asking Questions about Program Failures,” Proc. ACM Conf. Human Factors in Computing Systems, pp. 151-158. Apr. 2004.
[35] B. Korel and J. Laski, “Dynamic Slicing of Computer Programs,” J. Systems and Software, vol. 13, no. 3, pp. 187-195, Nov. 1990.
[36] V. Krishna, C. Cook, D. Keller, J. Cantrell, C. Wallace, M. Burnett, and G. Rothermel, “Incorporating Incremental Validation and Impact Analysis into Spreadsheet Maintenance: An Empirical Study,” Proc. IEEE Int'l Conf. Software Maintenance, pp. 72-81, Nov. 2001.
[37] J. Laski and B. Korel, “A Data Flow Oriented Program Testing Strategy,” IEEE Trans. Software Eng., vol. 9, no. 3, pp. 347-354, May 1993.
[38] J.R. Lyle and M. Weiser, “Automatic Program Bug Location by Program Slicing,” Proc. Second Int'l Conf. Computers and Applications, pp. 877-883, June 1987.
[39] R.C. Miller and B.A. Myers, “Outlier Finding: Focusing User Attention on Possible Errors,” Proc. ACM Symp. User Interface Software and Technology, pp. 81-90, Nov. 2001.
[40] S.C. Ntafos, “On Required Element Testing,” IEEE Trans. Software Eng., vol. 10, no. 6, pp. 795-803, Nov. 1984.
[41] H. Pan and E. Spafford, “Toward Automatic Localization of Software Faults,” Proc. 10th Pacific Northwest Software Quality Conf., Oct. 1992.
[42] R. Panko, “Finding Spreadsheet Errors: Most Spreadsheet Errors Have Design Flaws that May Lead to Long-Term Miscalculation,” Information Week, p. 100, May 1995.
[43] R. Panko, “What We Know about Spreadsheet Errors,” J. End User Computing, pp. 15-21, Spring 1998.
[44] A. Phalgune, C. Kissinger, M. Burnett, C. Cook, L. Beckwith, and J.R. Ruthruff, “Garbage In, Garbage Out? An Empirical Look at Oracle Mistakes by End-User Programmers,” Proc. 2005 IEEE Symp. Visual Languages and Human-Centric Computing, pp. 45-52, Sept. 2005.
[45] B.C. Pierce and D.N. Turner, “Local Type Inference,” ACM Trans. Programming Languages and Systems, vol. 22, no. 1, pp. 1-44, Jan. 2000.
[46] B. Pytlik, M. Renieris, S. Krishnamurthi, and S.P. Reiss, “Automated Fault Localization Using Potential Invariants,” Proc. Fifth Int'l Workshop Automated and Algorithmic Debugging, pp. 273-276, Sept. 2003.
[47] S. Rapps and E.J. Weyuker, “Selected Software Test Data Using Data Flow Information,” IEEE Trans. Software Eng., vol. 11, no. 4, pp. 367-375, Apr. 1985.
[48] O. Raz, P. Koopman, and M. Shaw, “Semantic Anomaly Detection on Online Data Sources,” Proc. 24th Int'l Conf. Software Eng., pp. 302-312, May 2002.
[49] M. Renieris and S.P. Reiss, “Fault Localization with Nearest Neighbor Queries,” Proc. 18th IEEE Int'l Conf. Automated Software Eng., pp. 30-39, Oct. 2003.
[50] G. Robertson, “Officials Red-Faced by $24m Gaffe: Error in Contract Bid Hits Bottom Line of TransAlta Corp.,” Ottawa Citizen, 5 June 2003.
[51] G. Rothermel, M. Burnett, L. Li, C. Dupuis, and A. Sheretov, “A Methodology for Testing Spreadsheets,” ACM Trans. Software Eng. and Methodology, vol. 10, no. 1, pp. 110-147, Jan. 2001.
[52] K.J. Rothermel, C.R. Cook, M.M. Burnett, J. Schonfeld, T.R.G. Green, and G. Rothermel, “WYSIWYT Testing in the Spreadsheet Paradigm: An Empirical Evaluation,” Proc. 22nd Int'l Conf. Software Eng., pp. 230-239, June 2000.
[53] J. Ruthruff, E. Creswick, M. Burnett, C. Cook, S. Prabhakararao, M. Fisher II, and M. Main, “End-User Software Visualizations for Fault Localization,” Proc. ACM Symp. Software Visualization, pp. 123-132, June 2003.
[54] J.R. Ruthruff and M. Burnett, “Six Challenges in Supporting End-User Debugging,” Proc. First Workshop End-User Software Eng., pp. 57-61, May 2005.
[55] J.R. Ruthruff, M. Burnett, and G. Rothermel, “An Empirical Study of Fault Localization for End-User Programmers,” Proc. 27th Int'l Conf. Software Eng., pp. 352-361, May 2005.
[56] J.R. Ruthruff, A. Phalgune, L. Beckwith, M. Burnett, and C. Cook, “Rewarding `Good' Behavior: End-User Debugging and Rewards,” Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 115-122, Sept. 2004.
[57] J.R. Ruthruff, S. Prabhakararao, J. Reichwein, C. Cook, E. Creswick, and M. Burnett, “Interactive, Visual Fault Localization Support for End-User Programmers,” J. Visual Languages and Computing, vol. 16, nos. 1-2, pp. 3-40, Feb.-Apr. 2005.
[58] J. Sajaniemi, “Modeling Spreadsheet Audit: A Rigorous Approach to Automatic Visualization,” J. Visual Languages and Computing, vol. 11, no. 1, pp. 49-82, Feb. 2000.
[59] C. Scaffidi, M. Shaw, and B. Myers, “Estimating the Numbers of End Users and End User Programmers,” Proc. 2005 IEEE Symp. Visual Languages and Human-Centric Computing, pp. 207-214, Sept. 2005.
[60] S. Siegel and N.J. Castellan Jr., Non-Parametric Statistics for the Behavioral Sciences. Boston: McGraw-Hill, 1998.
[61] D.H. Stamatis, Failure Mode Effect Analysis: FMEA from Theory to Execution, second ed. ASQ Quality Press, June 2003.
[62] F. Tip, “A Survey of Program Slicing Techniques,” J. Programming Languages, vol. 3, no. 3, pp. 121-189, 1995.
[63] J.M. Voas, “Software Testability Measurement for Assertion Placement and Fault Localization,” Proc. Int'l Workshop Automated and Algorithmic Debugging, pp. 133-144, 1995.
[64] E. Wagner and H. Lieberman, “An End-User Tool for e-Commerce Debugging,” Proc. Int'l Conf. Intelligent User Interfaces, pp. 331, Jan. 2003.
[65] E.J. Wagner and H. Lieberman, “Supporting User Hypotheses in Problem Diagnosis on the Web and Elsewhere,” Proc. Int'l Conf. Intelligent User Interfaces, pp. 30-37, Jan. 2004.
[66] M. Weiser, “Program Slicing,” IEEE Trans. Software Eng., vol. 10, no. 4, pp. 352-357, July 1984.
[67] A. Wilson, M. Burnett, L. Beckwith, O. Granatir, L. Casburn, C. Cook, M. Durham, and G. Rothermel, “Harnessing Curiosity to Increase Correctness in End-User Programming,” Proc. ACM Conf. Human Factors in Computing Systems, pp. 305-312, Apr. 2003.
[68] C. Wohlin, P. Runeson, M. Host, B. Regnell, and A. Wesslen, Experimentation in Software Engineering. Boston: Kluwer Academic, 2000.

Index Terms:
Fault localization, debugging, end-user software engineering, spreadsheets, end-user programming.
Citation:
Joseph R. Ruthruff, Margaret Burnett, Gregg Rothermel, "Interactive Fault Localization Techniques in a Spreadsheet Environment," IEEE Transactions on Software Engineering, vol. 32, no. 4, pp. 213-239, April 2006, doi:10.1109/TSE.2006.37
Usage of this product signifies your acceptance of the Terms of Use.