Issue No. 07 - July (2017 vol. 43)
Daniel Alencar da Costa , Department of Informatics and Applied Mathematics (DIMAp), Federal University of Rio Grande do Norte, Natal-RN, Brazil
Shane McIntosh , Department of Electrical and Computer Engineering, McGill University, Montreal, QC, Canada
Weiyi Shang , Department of Computer Science and Software Engineering, Concordia University, Montreal, QC, Canada
Uira Kulesza , Department of Informatics and Applied Mathematics (DIMAp), Federal University of Rio Grande do Norte, Natal-RN, Brazil
Roberta Coelho , Department of Informatics and Applied Mathematics (DIMAp), Federal University of Rio Grande do Norte, Natal-RN, Brazil
Ahmed E. Hassan , Software Analysis and Intelligence Lab (SAIL), School of Computing, Queen?s University, Kingston, ON, Canada
The approach proposed by Śliwerski, Zimmermann, and Zeller (SZZ) for identifying bug-introducing changes is at the foundation of several research areas within the software engineering discipline. Despite the foundational role of SZZ, little effort has been made to evaluate its results. Such an evaluation is a challenging task because the ground truth is not readily available. By acknowledging such challenges, we propose a framework to evaluate the results of alternative SZZ implementations. The framework evaluates the following criteria: (1) the earliest bug appearance, (2) the future impact of changes, and (3) the realism of bug introduction. We use the proposed framework to evaluate five SZZ implementations using data from ten open source projects. We find that previously proposed improvements to SZZ tend to inflate the number of incorrectly identified bug-introducing changes. We also find that a single bug-introducing change may be blamed for introducing hundreds of future bugs. Furthermore, we find that SZZ implementations report that at least 46 percent of the bugs are caused by bug-introducing changes that are years apart from one another. Such results suggest that current SZZ implementations still lack mechanisms to accurately identify bug-introducing changes. Our proposed framework provides a systematic mean for evaluating the data that is generated by a given SZZ implementation.
Computer bugs, Software engineering, Electronic mail, Software, Manuals, History, Systematics
D. A. da Costa, S. McIntosh, W. Shang, U. Kulesza, R. Coelho and A. E. Hassan, "A Framework for Evaluating the Results of the SZZ Approach for Identifying Bug-Introducing Changes," in IEEE Transactions on Software Engineering, vol. 43, no. 7, pp. 641-657, 2017.