The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - Oct. (2013 vol.39)
pp: 1427-1443
Stephen W. Thomas , Queen's University, Kingston
Meiyappan Nagappan , Queen's University, Kingston
Dorothea Blostein , Queen's University, Kingston
Ahmed E. Hassan , Queen's University, Kingston
ABSTRACT
Bug localization is the task of determining which source code entities are relevant to a bug report. Manual bug localization is labor intensive since developers must consider thousands of source code entities. Current research builds bug localization classifiers, based on information retrieval models, to locate entities that are textually similar to the bug report. Current research, however, does not consider the effect of classifier configuration, i.e., all the parameter values that specify the behavior of a classifier. As such, the effect of each parameter or which parameter values lead to the best performance is unknown. In this paper, we empirically investigate the effectiveness of a large space of classifier configurations, 3,172 in total. Further, we introduce a framework for combining the results of multiple classifier configurations since classifier combination has shown promise in other domains. Through a detailed case study on over 8,000 bug reports from three large-scale projects, we make two main contributions. First, we show that the parameters of a classifier have a significant impact on its performance. Second, we show that combining multiple classifiers--whether those classifiers are hand-picked or randomly chosen relative to intelligently defined subspaces of classifiers--improves the performance of even the best individual classifiers.
INDEX TERMS
classifier combination, Software maintenance, bug localization, information retrieval, VSM, LSI, LDA,
CITATION
Stephen W. Thomas, Meiyappan Nagappan, Dorothea Blostein, Ahmed E. Hassan, "The Impact of Classifier Configuration and Classifier Combination on Bug Localization", IEEE Transactions on Software Engineering, vol.39, no. 10, pp. 1427-1443, Oct. 2013, doi:10.1109/TSE.2013.27
REFERENCES
[1] R. Arnold and S. Bohner, "Impact Analysis—Towards a Framework for Comparison," Proc. Int'l Conf. Software Maintenance, pp. 292-301, 1993.
[2] R. Baeza-Yates and B. Ribeiro-Neto, Modern Information Retrieval, vol. 463, ACM Press, 1999.
[3] C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu, "Fair and Balanced?: Bias in Bug-Fix Data Sets," Proc. Seventh European Software Eng. Conf. and Symp. Foundations of Software Eng., pp. 121-130, 2009.
[4] D.M. Blei and J.D. Lafferty, "Topic Models," Text Mining: Classification, Clustering, and Applications, pp. 71-94. Chapman & Hall, 2009.
[5] D.M. Blei, A.Y. Ng, and M.I. Jordan, "Latent Dirichlet Allocation," J. Machine Learning Research, vol. 3, pp. 993-1022, 2003.
[6] S. Bohner and R. Arnold, Software Change Impact Analysis. IEEE CS Press, 1996.
[7] G. Box and D. Behnken, "Some New Three Level Designs for the Study of Quantitative Variables," Technometrics, vol. 2, no. 4, pp. 455-475, 1960.
[8] C. Carpineto and G. Romano, "A Survey of Automatic Query Expansion in Information Retrieval," ACM Computing Surveys, vol. 44, no. 1, pp. 1-50, Jan. 2012.
[9] J. Chang and D.M. Blei, "Relational Topic Models for Document Networks," Proc. 12th Int'l Conf. Artificial Intelligence and Statistics, pp. 81-88, 2009.
[10] B. Cleary, C. Exton, J. Buckley, and M. English, "An Empirical Analysis of Information Retrieval Based Concept Location Techniques in Software Comprehension," Empirical Software Eng., vol. 14, no. 1, pp. 93-130, 2008.
[11] G.V. Cormack, C.L. Clarke, and S. Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods," Proc. 32nd Int'l Conf. Research and Development in Information Retrieval, pp. 758-759, 2009.
[12] V. Dallmeier and T. Zimmermann, "Extraction of Bug Localization Benchmarks from History," Proc. 22nd Int'l Conf. Automated Software Eng., pp. 433-436, 2007.
[13] M. D'Ambros, M. Lanza, and R. Robbes, "Evaluating Defect Prediction Approaches: A Benchmark and an Extensive Comparison," Empirical Software Eng., vol. 17, no. 4, pp. 531-577, 2012.
[14] S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R. Harshman, "Indexing by Latent Semantic Analysis," J. Am. Soc. Information Science, vol. 41, no. 6, pp. 391-407, 1990.
[15] B. Dit, L. Guerrouj, D. Poshyvanyk, and G. Antoniol, "Can Better Identifier Splitting Techniques Help Feature Location?" Proc. Int'l Conf. Program Comprehension, pp. 11-20, 2011.
[16] "Eclipse Foundation," Eclipse, http:/www.eclipse.org/. 2011.
[17] E. Enslen, E. Hill, L. Pollock, and K. Vijay-Shanker, "Mining Source Code to Automatically Split Identifiers for Software Analysis," Proc. Sixth Int'l Working Conf. Mining Software Repositories, pp. 71-80, 2009.
[18] M. Fischer, M. Pinzger, and H. Gall, "Populating a Release History Database from Version Control and Bug Tracking Systems," Proc. 19th Int'l Conf. Software Maintenance, 2003.
[19] R.L. Glass, Facts and Fallacies of Software Engineering. Addison-Wesley Professional, 2003.
[20] E. Hill, S. Rao, and A. Kak, "On the Use of Stemming for Concern Location and Bug Localization in Java," Proc. 12th Int'l Working Conf. Source Code Analysis and Manipulation, pp. 1-10, 2012.
[21] T. Ho, J. Hull, and S. Srihari, "Decision Combination in Multiple Classifier Systems," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 1 pp. 66-75, Jan. 1994.
[22] IBM. Jazz, http://www-01.ibm.com/software/rationaljazz /, 2011.
[23] E. Keogh, S. Lonardi, and C. Ratanamahatana, "Towards Parameter-Free Data Mining," Proc. 10th Int'l Conf. Knowledge Discovery and Data Mining, pp. 206-215, 2004.
[24] J. Kittler, M. Hatef, R.P. Duin, and J. Matas, "On Combining Classifiers," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226-239, Mar. 1998.
[25] E. Kocaguneli, T. Menzies, and J.W. Keung, "On the Value of Ensemble Effort Estimation," IEEE Trans. Software Eng., vol. 38, no. 6 pp. 1403-1416, Nov./Dec. 2012.
[26] R. Kuehl, Design of Experiments: Statistical Principles of Research Design and Analysis. Brooks/Cole, 2000.
[27] D. Lawrie and D. Binkley, "Expanding Identifiers to Normalize Source Code Vocabulary," Proc. 27th Int'l Conf. Software Maintenance, pp. 113-122, 2011.
[28] S.K. Lukins, N.A. Kraft, and L.H. Etzkorn, "Source Code Retrieval for Bug Localization Using Latent Dirichlet Allocation," Proc. 15th Working Conf. Reverse Eng., pp. 155-164, 2008.
[29] S.K. Lukins, N.A. Kraft, and L.H. Etzkorn, "Bug Localization Using Latent Dirichlet Allocation," Information and Software Technology, vol. 52, no. 9, pp. 972-990, 2010.
[30] N. Madani, L. Guerrouj, M. Di Penta, Y. Gueheneuc, and G. Antoniol, "Recognizing Words from Source Code Identifiers Using Speech Recognition Techniques," Proc. 14th European Conf. Software Maintenance and Reeng., pp. 68-77, 2010.
[31] C.D. Manning, P. Raghavan, and H. Schutze, Introduction to Information Retrieval, vol. 1, Cambridge Univ. Press Cambridge, 2008.
[32] A. Marcus, A. Sergeyev, V. Rajlich, and J.I. Maletic, "An Information Retrieval Approach to Concept Location in Source Code," Proc. 11th Working Conf. Reverse Eng., pp. 214-223, 2004.
[33] A.K. McCallum, "Mallet: A Machine Learning for Language Toolkit," http:/mallet.cs.umass.edu, 2002.
[34] C. McMillan, M. Grechanik, D. Poshyvanyk, Q. Xie, and C. Fu, "Portfolio: Finding Relevant Functions and Their Usage," Proc. 33rd Int'l Conf. Software Eng., pp. 111-120, 2011.
[35] A.T. Misirli, A.B. Bener, and B. Turhan, "An Industrial Case Study of Classifier Ensembles for Locating Software Defects," Software Quality J., vol. 19, no. 3, pp. 515-536, 2011.
[36] R. Moser, W. Pedrycz, and G. Succi, "A Comparative Analysis of the Efficiency of Change Metrics and Static Code Attributes for Defect Prediction," Proc. 30th Int'l Conf. Software Eng., pp. 181-190, 2008.
[37] Mozilla Foundation, Bugzilla. http:/www.bugzilla.org/, 2012.
[38] Mozilla Foundation, Mozilla. http:/www.mozilla.org/, 2012.
[39] R. Myers, A. Khuri, and W. Carter, "Response Surface Methodology: 1966-l988," Technometrics, vol. 31, no. 2 pp. 137-157, 1989.
[40] A.T. Nguyen, T.T. Nguyen, J. Al-Kofahi, H.V. Nguyen, and T.N. Nguyen, "A Topic-Based Approach for Narrowing the Search Space of Buggy Files from a Bug Report," Proc. 26th Int'l Conf. Automated Software Eng., pp. 263-272, 2011.
[41] T.H. Nguyen, B. Adams, and A.E. Hassan, "A Case Study of Bias in Bug-Fix Data Sets," Proc. 17th Working Conf. Reverse Eng., pp. 259-268, 2010.
[42] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, "Where the Bugs Are," Proc. Int'l Symp. Software Testing and Analysis, pp. 86-96, 2004.
[43] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, "Predicting the Location and Number of Faults in Large Software Systems," IEEE Trans. Software Eng., vol. 31, no. 4 pp. 340-355, Apr. 2005.
[44] L. Page, S. Brin, R. Motwani, and T. Winograd, "The PageRank Citation Ranking: Bringing Order to the Web," technical report, Stanford InfoLab, 1999.
[45] M. Porter, "An Algorithm for Suffix Stripping," Program, vol. 14, pp. 130-137, 1980.
[46] D. Poshyvanyk, Y. Gueheneuc, A. Marcus, G. Antoniol, and V. Rajlich, "Feature Location Using Probabilistic Ranking of Methods Based on Execution Scenarios and Information Retrieval," IEEE Trans. Software Eng., vol. 33, no. 6 pp. 420-432, June 2007.
[47] D. Poshyvanyk and A. Marcus, "Combining Formal Concept Analysis with Information Retrieval for Concept Location in Source Code," Proc. 15th Int'l Conf. Program Comprehension, pp. 37-48, 2007.
[48] F. Rahman, D. Posnett, A. Hindle, E. Barr, and P. Devanbu, "BugCache for Inspections: Hit or Miss?" Proc. 19th Symp. and 13th European Conf. Foundations of Software Eng., pp. 322-331, 2011.
[49] S. Rao and A. Kak, "Retrieval from Software Libraries for Bug Localization: A Comparative Study of Generic and Composite Text Models," Proc. Eighth Working Conf. Mining Software Repositories, pp. 43-52, 2011.
[50] M. Revelle, B. Dit, and D. Poshyvanyk, "Using Data Fusion and Web Mining to Support Feature Location in Software," Proc. 18th Int'l Conf. Program Comprehension, pp. 14-23, 2010.
[51] S. Robertson, H. Zaragoza, and M. Taylor, "Simple BM25 Extension to Multiple Weighted Fields," Proc. 13th Int'l Conf. Information and Knowledge Management, pp. 42-49, 2004.
[52] G. Salton, A. Wong, and C.S. Yang, "A Vector Space Model for Automatic Indexing," Comm. ACM, vol. 18, no. 11, pp. 613-620, 1975.
[53] R.W. Selby, "Enabling Reuse-Based Software Development of Large-Scale Systems," IEEE Trans. Software Eng., vol. 31, no. 6 pp. 495-510, June 2005.
[54] J. Sliwerski, T. Zimmerman, and A. Zeller, "When Do Changes Induce Fixes?" Proc. Second Working Conf. Mining Software Repositories, 2005.
[55] S.W. Thomas http://sailhome.cs.queensu.ca/replication/ sthomasTSE2013, 2012.
[56] S.W. Thomas, "Mining Software Repositories with Topic Models," Technical Report 2012-586, School of Computing, Queen's Univ., 2012.
[57] J. Tukey, "The Philosophy of Multiple Comparisons," Statistical Science, vol. 6, no. 1 pp. 100-116, 1991.
[58] J. Tukey and H. Braun, The Collected Works of John W. Tukey: Multiple Comparisons, 1948-1983, vol. 8, Chapman & Hall/CRC, 1994.
[59] M. Van Erp and L. Schomaker, "Variants of the Borda Count Method for Combining Ranked Classifier Hypotheses," Proc. Seventh Int'l Workshop Frontiers in Handwriting Recognition, pp. 443-452, 2000.
[60] H. Wang, T.M. Khoshgoftaar, and A. Napolitano, "Software Measurement Data Reduction Using Ensemble Techniques," Neurocomputing, vol. 92, pp. 124-132, 2012.
[61] X. Wei and W.B. Croft, "LDA-Based Document Models for Ad-Hoc Retrieval," Proc. 29th Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 178-185, 2006.
[62] T. Zimmermann, R. Premraj, and A. Zeller, "Predicting Defects for Eclipse," Proc. Third Int'l Workshop Predictor Models in Software Eng., 2007.
52 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool