This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
A Large-Scale Empirical Study of Just-in-Time Quality Assurance
June 2013 (vol. 39 no. 6)
pp. 757-773
Yasutaka Kamei, Kyushu University, Fukuoka
Emad Shihab, Rochester Institute of Technology, Rochester
Bram Adams, École Polytechnique de Montréal, Montréal
Ahmed E. Hassan, Queen's University, Kingston
Audris Mockus, Avaya Labs Research, Basking Ridge
Anand Sinha, Research In Motion, Waterloo
Naoyasu Ubayashi, Kyushu University, Fukuoka
Defect prediction models are a well-known technique for identifying defect-prone files or packages such that practitioners can allocate their quality assurance efforts (e.g., testing and code reviews). However, once the critical files or packages have been identified, developers still need to spend considerable time drilling down to the functions or even code snippets that should be reviewed or tested. This makes the approach too time consuming and impractical for large software systems. Instead, we consider defect prediction models that focus on identifying defect-prone (“risky”) software changes instead of files or packages. We refer to this type of quality assurance activity as “Just-In-Time Quality Assurance,” because developers can review and test these risky changes while they are still fresh in their minds (i.e., at check-in time). To build a change risk model, we use a wide range of factors based on the characteristics of a software change, such as the number of added lines, and developer experience. A large-scale study of six open source and five commercial projects from multiple domains shows that our models can predict whether or not a change will lead to a defect with an average accuracy of 68 percent and an average recall of 64 percent. Furthermore, when considering the effort needed to review changes, we find that using only 20 percent of the effort it would take to inspect all changes, we can identify 35 percent of all defect-inducing changes. Our findings indicate that “Just-In-Time Quality Assurance” may provide an effort-reducing way to focus on the most risky changes and thus reduce the costs of developing high-quality software.
Index Terms:
Measurement,Quality assurance,Predictive models,Software,Entropy,Object oriented modeling,Accuracy,just-in-time prediction,Maintenance,software metrics,mining software repositories,defect prediction
Citation:
Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E. Hassan, Audris Mockus, Anand Sinha, Naoyasu Ubayashi, "A Large-Scale Empirical Study of Just-in-Time Quality Assurance," IEEE Transactions on Software Engineering, vol. 39, no. 6, pp. 757-773, June 2013, doi:10.1109/TSE.2012.70
Usage of this product signifies your acceptance of the Terms of Use.