This Article 
 Bibliographic References 
 Add to: 
Comparing the Defect Reduction Benefits of Code Inspection and Test-Driven Development
May-June 2012 (vol. 38 no. 3)
pp. 547-560
Jerod W. Wilkerson, Pennsylvania State University, Erie, Erie
Jay F. Nunamaker, University of Arizona, Tucson
Rick Mercer, University of Arizona, Tucson
This study is a quasi experiment comparing the software defect rates and implementation costs of two methods of software defect reduction: code inspection and test-driven development. We divided participants, consisting of junior and senior computer science students at a large Southwestern university, into four groups using a two-by-two, between-subjects, factorial design and asked them to complete the same programming assignment using either test-driven development, code inspection, both, or neither. We compared resulting defect counts and implementation costs across groups. We found that code inspection is more effective than test-driven development at reducing defects, but that code inspection is also more expensive. We also found that test-driven development was no more effective at reducing defects than traditional programming methods.

[1] G. Tassey, "The Economic Impact of Inadequate Infrastructure for Software Testing," technical report, Nat'l Inst. of Standards and Tech nology, 2002.
[2] B. George and L. Williams, "A Structured Experiment of Test-Driven Development," Information and Software Technology, vol. 46, no. 5, pp. 337-342, 2004.
[3] E.M. Maximilien and L. Williams, "Assessing Test-Driven Development at IBM," Proc. 25th Int'l Conf. Software Eng., pp. 564-9, 2003.
[4] D.L. Parnas and M. Lawford, "The Role of Inspections in Software Quality Assurance," IEEE Trans. Software Eng., vol. 29, no. 8, pp. 674-676, Aug. 2003.
[5] F. Shull, V.R. Basili, B.W. Boehm, A.W. Brown, P. Costa, M. Lindvall, D. Port, I. Rus, R. Tesoriero, and M. Zelkowitz, "What We Have Learned about Fighting Defects," Proc. Eighth IEEE Symp. Software Metrics, pp. 249-58, 2002.
[6] M.E. Fagan, "Design and Code Inspections to Reduce Errors in Program Development," IBM Systems J., vol. 15, no. 3, pp. 182-211, 1976.
[7] N. Nagappan, M.E. Maximilien, T. Bhat, and L. Williams, "Realizing Quality Improvement through Test Driven Development: Results and Experiences of Four Industrial Teams," Empirical Software Eng., vol. 13, no. 3, pp. 289-302, 2008.
[8] P. Runeson, C. Andersson, T. Thelin, A. Andrews, and T. Berling, "What Do We Know about Defect Detection Methods?" IEEE Software, vol. 23, no. 3, pp. 82-90, May/June 2006.
[9] M.E. Fagan, "Advances in Software Inspections," IEEE Trans. Software Eng., vol. 12, no. 7, pp. 744-51, July 1986.
[10] T. Gilb and D. Graham, Software Inspection. Addison-Wesley, 1993.
[11] O. Laitenberger and J.-M. DeBaud, "An Encompassing Life Cycle Centric Survey of Software Inspection," J. Systems and Software, vol. 50, no. 1, pp. 5-31, 2000.
[12] A. Aurum, H. Petersson, and C. Wohlin, "State-of-the-Art: Software Inspections after 25 Years," Software Testing, Verification and Reliability, vol. 12, no. 3, pp. 133-54, 2002.
[13] A.F. Ackerman, L.S. Buchwald, and F.H. Lewski, "Software Inspections: An Effective Verification Process," IEEE Software, vol. 6, no. 3, pp. 31-36, May 1989.
[14] W.S. Humphrey, A Discipline for Software Eng., ser. the SEI Series in Software Engineering. Addison-Wesley Publishing Company, 1995.
[15] R.C. Linger, "Cleanroom Software Engineering for Zero-Defect Software," Proc. 15th Int'l Conf. Software Eng., pp. 2-13, 1993.
[16] T. Thelin, P. Runeson, and B. Regnell, "Usage-Based Reading—An Experiment to Guide Reviewers with Use Cases," Information and Software Technology, vol. 43, no. 15, pp. 925-38, 2001.
[17] T. Thelin, P. Runeson, and C. Wohlin, "An Experimental Comparison of Usage-Based and Checklist-Based Reading," IEEE Trans. Software Eng., vol. 29, no. 8, pp. 687-704, Aug. 2003.
[18] T. Thelin, P. Runeson, C. Wohlin, T. Olsson, and C. Andersson, "Evaluation of Usage-Based Reading—Conclusions after Three Experiments," Empirical Software Eng., vol. 9, nos. 1/2, pp. 77-110, 2004.
[19] V.R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sørumgård, and M.V. Zelkowitz, "The Empirical Investigation of Perspective-Based Reading," Empirical Software Eng., vol. 1, no. 2, pp. 133-64, 1996.
[20] C. Denger, M. Ciolkowski, and F. Lanubile, "Investigating the Active Guidance Factor in Reading Techniques for Defect Detection," Proc. Third Int'l Symp. Empirical Software Eng., 2004.
[21] O. Laitenberger and J.-M. DeBaud, "Perspective-Based Reading of Code Documents at Robert Bosch GmbH," Information and Software Technology, vol. 39, no. 11, pp. 781-791, 1997.
[22] J. Miller, M. Wood, and M. Roper, "Further Experiences with Scenarios and Checklists," Empirical Software Eng., vol. 3, no. 1, pp. 37-64, 1998.
[23] V.R. Basili, G. Caldiera, F. Lanubile, and F. Shull, "Studies on Reading Techniques," Proc. 21st Ann. Software Eng. Workshop, pp. 59-65, 1996.
[24] V.R. Basili and R.W. Selby, "Comparing the Effectiveness of Software Testing Strategies," IEEE Trans. Software Eng., vol. 13, no. 12, pp. 1278-1296, Dec. 1987.
[25] A.A. Porter and L.G. Votta, "An Experiment to Assess Different Defect Detection Methods for Software Requirements Inspections," Proc. 16th Int'l Conf. Software Eng., pp. 103-12, 1994.
[26] A.A. Porter, L.G. VottaJr, and V.R. Basili, "Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment," IEEE Trans. Software Eng., vol. 21, no. 6, pp. 563-575, June 1995.
[27] F. Shull, F. Lanubile, and V.R. Basili, "Investigating Reading Techniques for Object-Oriented Framework Learning," IEEE Trans. Software Eng., vol. 26, no. 11, pp. 1101-1118, Nov. 2000.
[28] J.F. NunamakerJr, R.O. Briggs, D.D. Mittleman, D.R. Vogel, and P.A. Balthazard, "Lessons from a Dozen Years of Group Support Systems Research: A Discussion of Lab and Field Findings," J. Management Information Systems, vol. 13, no. 3, pp. 163-207, 1997.
[29] J.F. NunamakerJr, A.R. Dennis, J.S. Valacich, D.R. Vogel, and J.F. George, "Electronic Meeting Systems to Support Group Work," Comm. ACM, vol. 34, no. 7, pp. 40-61, 1991.
[30] P.M. Johnson, "An Instrumented Approach to Improving Software Quality through Formal Technical Review," Proc. 16th Int'l Conf. Software Eng., pp. 113-22, 1994.
[31] M. van Genuchten, C. van Dijk, H. Scholten, and D. Vogel, "Using Group Support Systems for Software Inspections," IEEE Software, vol. 18, no. 3, pp. 60-65, May/June 2001.
[32] S. Biffli, P. Grünbacher, and M. Halling, "A Family of Experiments to Investigate the Effects of Groupware for Software Inspection," Automated Software Eng., vol. 13, no. 3, pp. 373-394, 2006.
[33] F. Lanubile, T. Mallardo, and F. Calefato, "Tool Support for Geographically Dispersed Inspection Teams," Software Process: Improvement and Practice, vol. 8, no. 4, pp. 217-231, 2003.
[34] C.K. Tyran and J.F. George, "Improving Software Inspections with Group Process Support," Comm. ACM, vol. 45, no. 9, pp. 87-92, 2002.
[35] M. van Genuchten, W. Cornelissen, and C. van Dijk, "Supporting Inspections with an Electronic Meeting System," J. Management Information Systems, vol. 14, no. 3, pp. 165-78, 1997.
[36] P. Vitharana and K. Ramamurthy, "Computer-Mediated Group Support, Anonymity, and the Software Inspection Process: An Empirical Investigation," IEEE Trans. Software Eng., vol. 29, no. 2, pp. 167-80, Feb. 2003.
[37] B.W. Boehm, Software Eng. Economics, Prentice Hall, 1981.
[38] K. Beck, Test Driven Development: By Example. Addison-Wesley Professional, 2002.
[39] M.M. Müller and O. Hagner, "Experiment about Test-First Programming," IEE Proc. Software, vol. 149, no. 5, pp. 131-136, Oct. 2002.
[40] H. Erdogmus, M. Morisio, and M. Torchiano, "On the Effectiveness of the Test-First Approach to Programming," IEEE Trans. Software Eng., vol. 31, no. 3, pp. 226-37, Mar. 2005.
[41] W.S. Humphrey, Managing the Software Process, ser. The SEI Series in Software Engineering. Addison-Wesley Publishing Company, 1989.
[42] T.L. Rodgers, D.L. Dean, and J.F. Nunamaker Jr, "Increasing Inspection Efficiency through Group Support Systems," Proc. 37th Ann. Hawaii Int'l Conf. System Sciences, 2004.
[43] C. Fox, "Java Inspection Checklist," 1999.
[44] R.G. Ebenau and S.H. Strauss, Software Inspection Process, ser. Systems Design and Implementation. McGraw Hill, 1994.
[45] L.S. Meyers, G. Gamst, and A.J. Guarino, Applied Multivariate Research: Design and Interpretation. Sage Publications, Inc., 2006.
[46] J.F. Hair, B. Black, B. Babin, R.E. Anderson, and R.L. Tatham, Multivariate Data Analysis, sixth ed. Prentice Hall, 2005.
[47] S.R. Searle, F.M. Speed, and G.A. Milliken, "Population Marginal Means in the Linear Model: An Alternative to Least Squares Means," The Am. Statistician, vol. 34, no. 4, pp. 216-221, 1980.

Index Terms:
Agile programming, code inspections and walk throughs, reliability, test-driven development, testing strategies, empirical study.
Jerod W. Wilkerson, Jay F. Nunamaker, Rick Mercer, "Comparing the Defect Reduction Benefits of Code Inspection and Test-Driven Development," IEEE Transactions on Software Engineering, vol. 38, no. 3, pp. 547-560, May-June 2012, doi:10.1109/TSE.2011.46
Usage of this product signifies your acceptance of the Terms of Use.