The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.05 - September/October (2010 vol.36)
pp: 618-643
Rahul Premraj , Vrije Universiteit Amsterdam, Amsterdam
Nicolas Bettenburg , Queen's University, Kingston
Sascha Just , Saarland University, Saarbruecken
Adrian Schröter , University of Victoria, Victoria
Thomas Zimmermann , Microsoft Research, Redmond
ABSTRACT
In software development, bug reports provide crucial information to developers. However, these reports widely differ in their quality. We conducted a survey among developers and users of APACHE, ECLIPSE, and MOZILLA to find out what makes a good bug report. The analysis of the 466 responses revealed an information mismatch between what developers need and what users supply. Most developers consider steps to reproduce, stack traces, and test cases as helpful, which are, at the same time, most difficult to provide for users. Such insight is helpful for designing new bug tracking tools that guide users at collecting and providing more helpful information. Our CUEZILLA prototype is such a tool and measures the quality of new bug reports; it also recommends which elements should be added to improve the quality. We trained CUEZILLA on a sample of 289 bug reports, rated by developers as part of the survey. The participants of our survey also provided 175 comments on hurdles in reporting and resolving bugs. Based on these comments, we discuss several recommendations for better bug tracking systems, which should focus on engaging bug reporters, better tool support, and improved handling of bug duplicates.
INDEX TERMS
Testing and debugging, distribution, maintenance, and enhancement, human factors, management, measurement.
CITATION
Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schröter, Thomas Zimmermann, "What Makes a Good Bug Report?", IEEE Transactions on Software Engineering, vol.36, no. 5, pp. 618-643, September/October 2010, doi:10.1109/TSE.2010.63
REFERENCES
[1] P. Anbalagan and M. Vouk, "On Predicting the Time Taken to Correct Bugs in Open Source Projects (Short Paper)," Proc. 25th IEEE Int'l Conf. Software Maintenance, Sept. 2009.
[2] G. Antoniol, K. Ayari, M. Di Penta, F. Khomh, and Y.-G. Guéhéneuc, "Is It a Bug or An Enhancement?: A Text-Based Approach to Classify Change Requests," Proc. 2008 Conf. Center for Advanced Studies on Collaborative Research, pp. 304-318, 2008.
[3] G. Antoniol, H. Gall, M.D. Penta, and M. Pinzger, "Mozilla: Closing the Circle," Technical Report TUV-1841-2004-05, Technical Univ. of Vienna, 2004.
[4] J. Anvik, L. Hiew, and G.C. Murphy, "Who Should Fix This Bug?" Proc. 28th Int'l Conf. Software Eng., pp. 361-370, 2006.
[5] J. Anvik and G. Murphy, "Reducing the Effort of Bug Report Triage: Recommenders for Development-Oriented Decisions," ACM Trans. Software Eng. and Methodology, to appear.
[6] J. Aranda and G. Venolia, "The Secret Life of Bugs: Going Past the Errors and Omissions in Software Repositories," Proc. 31st Int'l Conf. Software Eng., 2009.
[7] S. Artzi, S. Kim, and M.D. Ernst, "Recrash: Making Software Failures Reproducible by Preserving Object States," Proc. 22nd European Object-Oriented Programming Conf., pp. 542-565, 2008.
[8] B. Ashok, J. Joy, H. Liang, S.K. Rajamani, G. Srinivasa, and V. Vangala, "DebugAdvisor: A Recommender System for Debugging," Proc. Seventh Joint Meeting of the European Software Eng. Conf. and the ACM SIGSOFT Symp. Foundations of Software Eng. on European Software Eng. Conf. and Foundations of Software Eng. Symp., pp. 373-382, 2009.
[9] I. Barker, "What Is Information Architecture? KM Column," http:/www.steptwo.com.au, 2010.
[10] V.R. Basili, F. Shull, and F. Lanubile, "Building Knowledge through Families of Experiments," IEEE Trans. Software Eng., vol. 25, no. 4, pp. 456-473, July 1999.
[11] D. Bertram, A. Voida, S. Greenberg, and R. Walker, "Communication, Collaboration, and Bugs: The Social Nature of Issue Tracking in Small, Collocated Teams," Proc. 2010 ACM Conf. Computer Supported Cooperative Work, pp. 291-300, 2010.
[12] N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann, "Quality of Bug Reports in Eclipse," Proc. 2007 OOPSLA Workshop Eclipse Technology eXchange, pp. 21-25, Oct. 2007.
[13] N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann, "What Makes a Good Bug Report?" Proc. 16th Int'l Symp. Foundations of Software Eng., Nov. 2008.
[14] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim, "Duplicate Bug Reports Considered Harmful... Really?" Proc. 24th IEEE Int'l Conf. Software Maintenance, pp. 337-345, Sept. 2008.
[15] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim, "Extracting Structural Information from Bug Reports," Proc. Fifth Working Conf. Mining Software Repositories, 2008.
[16] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim, "Extracting Structural Information from Bug Reports," Proc. Fifth Int'l Working Conf. Mining Software Repositories, May 2008.
[17] C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu, "Fair and Balanced? Bias in Bug-Fix Data Sets," Proc. European Software Eng. Conf. and ACM SIGSOFT Symp. Foundations of Software Eng., 2009.
[18] S. Breu, R. Premraj, J. Sillito, and T. Zimmermann, "Information Needs in Bug Reports: Improving Cooperation between Developers and Users," Proc. 2010 ACM Conf. Computer Supported Cooperative Work, pp. 301-310, 2010.
[19] G. Canfora and L. Cerulo, "Fine Grained Indexing of Software Repositories to Support Impact Analysis," Proc. Int'l Workshop Mining Software Repositories, pp. 105-111, 2006.
[20] G. Canfora and L. Cerulo, "Supporting Change Request Assignment in Open Source Development," Proc. 2006 ACM Symp. Applied Computing, pp. 1767-1772, 2006.
[21] L. Cherry and W. Vesterman, "Writing Tools—The STYLE and DICTION Programs," technical report, AT&T Laboratories, 1980.
[22] D. Cubranic and G.C. Murphy, "Automatic Bug Triage Using Text Categorization," Proc. 16th Int'l Conf. Software Eng. and Knowledge Eng., pp. 92-97, 2004.
[23] W. Cunningham, "Best of Bugzilla," http://eclipse-projects. blogspot.com/2005/ 12best-of-bugzilla.html, 2005.
[24] B. Dit and A. Marcus, "Improving the Readability of Defect Reports," Proc. 2008 Int'l Workshop Recommendation Systems for Software Eng., pp. 47-49, 2008.
[25] Eclipse Foundation, Callisto Simultaneous Release Bug Finding Contest, http://www.eclipse.org/projects/callisto-files callisto- bug-contest.php, 2010.
[26] M. Fischer, M. Pinzger, and H. Gall, "Analyzing and Relating Bug Report Data for Feature Tracking," Proc. 10th Working Conf. Reverse Eng., pp. 90-101, 2003.
[27] E. Goldberg, "Bug Writing Guidelines," https://bugs.eclipse. org/bugsbugwritinghelp.html , 2010.
[28] P.J. Guo, T. Zimmermann, N. Nagappan, and B. Murphy, "Characterizing and Predicting Which Bugs Get Fixed: An Empirical Study of Microsoft Windows," Proc. 32nd Int'l Conf. Software Eng., May 2010.
[29] L. Hiew, "Assisted Detection of Duplicate Bug Reports," master's thesis, The Univ. of British Columbia, 2006.
[30] P. Hooimeijer and W. Weimer, "Modeling Bug Report Quality," Proc. 22nd IEEE/ACM Int'l Conf. Automated Software Eng., pp. 34-43, 2007.
[31] HOT or NOT, http:/www.hotornot.com/, 2010.
[32] N. Jalbert and W. Weimer, "Automated Duplicate Detection for Bug Tracking Systems," Proc. Conf. Dependable Systems and Networks, pp. 52-61, June 2008.
[33] G. Jeong, S. Kim, and T. Zimmermann, "Improving Bug Triage with Bug Tossing Graphs," Proc. European Software Eng. Conf. and ACM SIGSOFT Symp. Foundations of Software Eng., 2009.
[34] S. Joshi and A. Orso, "SCARPE: A Technique and Tool for Selective Record and Replay of Program Executions," Proc. 23rd IEEE Int'l Conf. Software Maintenance, Oct. 2007.
[35] S. Just, R. Premraj, and T. Zimmermann, "Towards the Next Generation of Bug Tracking Systems," Proc. 2008 IEEE Symp. Visual Languages and Human-Centric Computing, pp. 82-85, Sept. 2008.
[36] M. Kersten and G.C. Murphy, "Using Task Context to Improve Programmer Productivity," Proc. 14th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 1-11, 2006.
[37] S. Kim, MSR Mining Challenge 2008, http://msr.uwaterloo.ca/msr2008challenge /, 2008.
[38] J.P. Kincaid, R.P. Fishburne,Jr., R.L. Rogers, and B.S. Chissom, "Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel," technical report, Research Branch Report 8-75, Naval Technical Training, US Naval Air Station, 1975.
[39] A.J. Ko and P.K. Chilana, "How Power Users Help and Hinder Open Bug Reporting," Proc. ACM Conf. Human Factors in Computing Systems, 2010.
[40] A.J. Ko, B.A. Myers, and D.H. Chau, "A Linguistic Analysis of How People Describe Software Problems," Proc. 2006 IEEE Symp. Visual Languages and Human-Centric Computing, pp. 127-134, 2006.
[41] B. Liblit, M. Naik, A.X. Zheng, A. Aiken, and M.I. Jordan, "Scalable Statistical Bug Isolation," Proc. 2005 ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 15-26, 2005.
[42] R. Likert, "A Technique for the Measurement of Attitudes," Archives of Psychology, vol. 140, pp. 1-55, 1932.
[43] D. MacKenzie, P. Eggert, and R. Stallman, Comparing and Merging Files with GNU Diff and Patch. Network Theory, Ltd., 2003.
[44] T. Menzies and A. Marcus, "Automated Severity Assessment of Software Defect Reports," Proc. 24th IEEE Int'l Conf. Software Maintenance, pp. 346-355, Sept. 2008.
[45] L. Moonen, "Generating Robust Parsers Using Island Grammars," Proc. Eighth Working Conf. Reverse Eng., pp. 13-22, 2001.
[46] Mozilla.org. Mozilla Security Bug Bounty Program, http://www.mozilla.org/securitybug-bounty.html , 2010.
[47] A. Orso, S. Joshi, M. Burger, and A. Zeller, "Isolating Relevant Component Interactions with JINSI," Proc. Fifth Int'l Workshop Dynamic Analysis, May 2006.
[48] A. Page, "Duplicate Bugs," http://blogs.msdn.com/alanpa/archive/2007/ 08/01duplicate-bugs.aspx, Apr. 2008.
[49] A. Page, K. Johnston, and B. Rollison, How We Test Software at Microsoft. Microsoft Press, 2008.
[50] L.D. Panjer, "Predicting Eclipse Bug Lifetimes," Proc. Fourth Int'l Workshop Mining Software Repositories, 2007.
[51] U. Passing and M.J. Shepperd, "An Experiment on Software Project Size and Effort Estimation," Proc. Int'l Symp. Empirical Software Eng., pp. 120-131, 2003.
[52] T. Punter, M. Ciolkowski, B. Freimut, and I. John, "Conducting Online Surveys in Software Engineering," Proc. Int'l Symp. Empirical Software Eng., pp. 80-88, 2003.
[53] Ratemyface.com., http:/www.ratemyface.com/, 2010.
[54] P. Runeson, M. Alexandersson, and O. Nyholm, "Detection of Duplicate Defect Reports Using Natural Language Processing," Proc. 29th Int'l Conf. Software Eng., pp. 499-510, 2007.
[55] A. Schröter, N. Bettenburg, and R. Premraj, "Do Stacktraces Help Developers Fix Bugs?" Proc. 2010 Int'l Working Conf. Mining Software Repositories, 2010.
[56] S. Siegel and N.J. CastellanJr., Nonparametric Statistics for the Behavioral Sciences, second ed. McGraw-Hill, 1988.
[57] J. Singer and N.G. Vinson, "Ethical Issues in Empirical Studies of Software Engineering," IEEE Trans. Software Eng., vol. 28, no. 12, pp. 1171-1180, Dec. 2002.
[58] J. Spolsky, Joel on Software. APress, 2004.
[59] D.W. van Liere, "How Shallow Is a Bug? Open Source Communities as Information Repositories and Solving Software Defects," SSRN eLibrary, http://ssrn.compaper=1507233, 2009.
[60] X. Wang, L. Zhang, T. Xie, J. Anvik, and J. Sun, "An Approach to Detecting Duplicate Bug Reports Using Natural Language and Execution Information," Proc. 30th Int'l Conf. Software Eng., May 2008.
[61] L. Wasserman, All of Statistics: A Concise Course in Statistical Inference, second ed. Springer, 2004.
[62] W. Weimer, "Patches as Better Bug Reports," Proc. Fifth Int'l Conf. Generative Programming and Component Eng., pp. 181-190, 2006.
[63] C. Weiss, R. Premraj, T. Zimmermann, and A. Zeller, "How Long Will It Take to Fix This Bug?" Proc. Fourth Int'l Workshop Mining Software Repositories, 2007.
[64] I.H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, second ed. Morgan Kaufmann, 2005.
[65] G. Xu, A. Rountev, Y. Tang, and F. Qin, "Efficient Checkpointing of Java Software Using Context-Sensitive Capture and Replay," Proc. European Software Eng. Conf. and ACM SIGSOFT Symp. Foundations of Software Eng., pp. 85-94, 2007.
[66] T. Zimmermann, R. Premraj, J. Sillito, and S. Breu, "Improving Bug Tracking Systems," Proc. Companion 31st Int'l Conf. Software Eng., pp. 247-250, 2009.
33 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool