The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.06 - June (2013 vol.39)
pp: 835-856
Yahya Rafique , Ryerson University, Toronto
Vojislav B. Mišić , Ryerson University, Toronto
ABSTRACT
This paper provides a systematic meta-analysis of 27 studies that investigate the impact of Test-Driven Development (TDD) on external code quality and productivity. The results indicate that, in general, TDD has a small positive effect on quality but little to no discernible effect on productivity. However, subgroup analysis has found both the quality improvement and the productivity drop to be much larger in industrial studies in comparison with academic studies. A larger drop of productivity was found in studies where the difference in test effort between the TDD and the control group's process was significant. A larger improvement in quality was also found in the academic studies when the difference in test effort is substantial; however, no conclusion could be derived regarding the industrial studies due to the lack of data. Finally, the influence of developer experience and task size as moderator variables was investigated, and a statistically significant positive correlation was found between task size and the magnitude of the improvement in quality.
INDEX TERMS
Productivity, Computational modeling, Testing, Process control, Programming, Size measurement, agile software development, Test-driven development, meta-analysis, code quality, programmer productivity
CITATION
Yahya Rafique, Vojislav B. Mišić, "The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis", IEEE Transactions on Software Engineering, vol.39, no. 6, pp. 835-856, June 2013, doi:10.1109/TSE.2012.28
REFERENCES
[1] S. Ambler, "Test-Driven Development Is the Combination of Test First Development and Refactoring," Dr. Dobbs' Agile Newsletter, June 2006.
[2] D. Astels, Test Driven Development: A Practical Guide. Prentice-Hall, 2003.
[3] K. Beck, Test-Driven Development: By Example. Prentice-Hall, 2003.
[4] J. Andrea, "Test-Driven Development: Driving New Standards of Beauty," Beautiful Testing: Leading Professionals Reveal How They Improve Software, pp. 181-194, O'Reilly Media, 2009.
[5] H. Erdogmus, M. Morisio, and M. Torchiano, "On the Effectiveness of the Test-First Approach to Programming," IEEE Trans. Software Eng., vol. 31, no. 3, pp. 226-237, Mar. 2005.
[6] M. Lipsey and D. Wilson, Practical Meta-Analysis. Sage Publications, Inc., 2001.
[7] B. Kitchenham and S. Charters, "Guidelines for Performing Systematic Literature Reviews in Software Engineering," Keele Univ., Technical Report EBSE 2007-001, 2007.
[8] J. Hannay, T. Dybå, E. Arisholm, and D. Sjøberg, "The Effectiveness of Pair Programming: A Meta-Analysis," Information and Software Technology, vol. 51, no. 7, pp. 1110-1122, 2009.
[9] T. Huedo-Medina, J. Sánchez-Meca, F. Marín-Martínez, and J. Botella, "Assessing Heterogeneity in Meta-Analysis: $Q$ statistic or $I^2$ Index?" Psychological Methods, vol. 11, no. 2, pp. 193-206, 2006.
[10] G. Melnik and F. Maurer, "A Cross-Program Investigation of Students' Perceptions of Agile Methods," Proc. 27th Int'l Conf. Software Eng., pp. 481-488, 2005.
[11] S. Kollanus and V. Isomöttönen, "Understanding TDD in Academic Environment: Experiences from Two Experiments," Proc. Eighth Int'l Conf. Computing Education Research, pp. 25-31, 2008.
[12] A. Rendell, "Effective and Pragmatic Test Driven Development," Proc. AGILE, pp. 298-303, 2008.
[13] J. Langr, "Evolution of Test and Code via Test-First Design," Proc. ACM Conf. Object-Oriented Programming, Systems, Languages, and Applications, http://www.objectmentor.com/resources/articles tfd.pdf, Oct. 2001.
[14] D.H. Steinberg, "The Effect of Unit Tests on Entry Points, Coupling and Cohesion in an Introductory Java Programming Course," Proc. XP Universe, Oct. 2001.
[15] P. Abrahamsson, A. Hanhineva, and J. Jäälinoja, "Improving Business Agility through Technical Solutions: A Case Study on Test-Driven Development in Mobile Software Development," Proc. Int'l Working Conf. Int'l Federation for Information Processing, pp. 227-243, 2005.
[16] J. Sanchez, L. Williams, and E. Maximilien, "On the Sustained Use of a Test-Driven Development Practice at IBM," Proc. AGILE, pp. 5-14, 2007.
[17] L. Madeyski, "The Impact of Pair Programming and Test-Driven Development on Package Dependencies in Object-Oriented Design—An Experiment," Proc. Seventh Int'l Conf. Product-Focused Software Process Improvement, pp. 278-289, 2006.
[18] M. Siniaalto, "The Impact of Test-Driven Development on Design Quality," Information Technology for European Advancement, technical report, D.5.2.10 v1.0, 2006.
[19] D. Janzen, C.S. Turner, and H. Saiedian, "Empirical Software Eng. in Industry Short Courses," Proc. 20th Conf. Software Eng. Education and Training, pp. 89-96, 2007.
[20] M. Siniaalto and P. Abrahamsson, "A Comparative Case Study on the Impact of Test-Driven Development on Program Design and Test Coverage," Proc. First Int'l Symp. Empirical Software Eng. and Measurement, pp. 275-284, 2007.
[21] M. Siniaalto and P. Abrahamsson, "Does Test-Driven Development Improve the Program Code? Alarming Results from a Comparative Case Study," Proc. Second IFIP TC 2 Central and East European Conf. Software Eng. Techniques, pp. 143-156, 2007.
[22] D. Janzen and H. Saiedian, "Does Test-Driven Development Really Improve Software Design Quality," IEEE Software, vol. 25, no. 2, pp. 77-84, Mar./Apr. 2008.
[23] L. Madeyski, "The Impact of Test-First Programming on Branch Coverage and Mutation Score Indicator of Unit Tests: An Experiment," Information and Software Technology, vol. 52, no. 2, pp. 169-184, 2010.
[24] R. Ynchausti, "Integrating Unit Testing into a Software Development Teams Process," Proc. Second Int'l Conf. Extreme Programming and Flexible Processes in Software Eng., pp. 79-83, May 2001.
[25] L. Damm, L. Lundberg, and D. Olsson, "Introducing Test Automation and Test-Driven Development: An Experience Report," Electronic Notes in Theoretical Computer Science, vol. 116, pp. 3-15, 2005.
[26] L. Madeyski and Ł. Szała, "The Impact of Test-Driven Development on Software Development Productivity—An Empirical Study," Proc. 14th European Conf. Software Process Improvement, pp. 200-211, 2007.
[27] B. George and L. Williams, "An Initial Investigation of Test Driven Development in Industry," Proc. ACM Symp. Applied Computing, pp. 1135-1139, 2003.
[28] B. George, "Analysis and Quantification of Test-Driven Development Approach," master's thesis, North Carolina State Univ., 2002.
[29] E. Maximilien and L. Williams, "Assessing Test-Driven Development at IBM," Proc. 25th Int'l Conf. Software Eng., pp. 564-569, 2003.
[30] L. Williams, E. Maximilien, and M. Vouk, "Test-Driven Development as a Defect-Reduction Practice," Proc. IEEE Int'l Symp. Software Reliability Eng., pp. 34-45, 2003.
[31] S. Edwards, "Using Software Testing to Move Students from Trial-and-Error to Reflection-in-Action," ACM SIGCSE Bulletin, vol. 36, no. 1, pp. 26-30, 2004.
[32] S. Edwards, "Using Test-Driven Development in the Classroom: Providing Students with Automatic, Concrete Feedback on Performance," Proc. Int'l Conf. Education and Information Systems Technologies and Applications, vol. 3, pp. 421-426, 2003.
[33] B. George and L. Williams, "A Structured Experiment of Test-Driven Development," Information and Software Technology, vol. 46, no. 5, pp. 337-342, 2004.
[34] A. Geras, "The Effectiveness of Test-Driven Development," master's thesis, Univ. of Calgary, Canada, 2004.
[35] A. Geras, M. Smith, and J. Miller, "A Prototype Empirical Evaluation of Test Driven Development," Proc. 10th Int'l Symp. Software Metrics, pp. 405-416, 2004.
[36] A. Hanhineva, "Test-Driven Development in Mobile Java Environment," master's thesis, Univ. of Oulu, Finland, 2004.
[37] T. Bhat and N. Nagappan, "Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng., pp. 356-363, 2006.
[38] N. Nagappan, E. Maximilien, T. Bhat, and L. Williams, "Realizing Quality Improvement through Test Driven Development: Results and Experiences of Four Industrial Teams," Empirical Software Eng., vol. 13, no. 3, pp. 289-302, 2008.
[39] G. Canfora, A. Cimitile, F. Garcia, M. Piattini, and C. Visaggio, "Productivity of Test Driven Development: A Controlled Experiment with Professionals," Proc. Seventh Int'l Conf. Product-Focused Software Process Improvement, pp. 383-388, 2006.
[40] G. Canfora, A. Cimitile, F. Garcia, M. Piattini, and C. Visaggio, "Evaluating Advantages of Test Driven Development: A Controlled Experiment with Professionals," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng., pp. 364-371, 2006.
[41] D. Janzen and H. Saiedian, "On the Influence of Test-Driven Development on Software Design," Proc. 19th Conf. Software Eng. Education and Training, pp. 141-148, Apr. 2006.
[42] D. Janzen, "An Empirical Evaluation of the Impact of Test-Driven Development on Software Quality," PhD dissertation, Univ. of Kansas, 2006.
[43] F. Shull, G. Melnik, B. Turhan, L. Layman, M. Diep, and H. Erdogmus, "What Do We Know about Test-Driven Development?" IEEE Software, vol. 27, no. 6, pp. 16-19, Nov./Dec. 2010.
[44] http:/www.meta-analysis.com/, 2013.
[45] V. Kampenes, T. Dybå, J. Hannay, and D. Sjøberg, "A Systematic Review of Effect Size in Software Engineering Experiments," Information and Software Technology, vol. 49, nos. 11/12, pp. 1073-1086, 2007.
[46] M. Borenstein, L.V. Hedges, J.P. Higgins, and H.R. Rothstein, "A Basic Introduction to Fixed-Effect and Random-Effects Models for Meta-Analysis," Research Synthesis Methods, vol. 1, no. 2, pp. 97-111, 2010.
[47] A. Gupta and P. Jalote, "An Experimental Evaluation of the Effectiveness and Efficiency of the Test Driven Development," Proc. First Int'l Symp. Empirical Software Eng. and Measurement, pp. 285-294, 2007.
[48] L. Madeyski, "Preliminary Analysis of the Effects of Pair Programming and Test-Driven Development on the External Code Quality," Proc. Conf. Software Eng.: Evolution and Emerging Technologies, pp. 113-123, 2005.
[49] M. Pančur and M. Ciglarič, "Impact of Test-Driven Development on Productivity, Code, and Tests: A Controlled Experiment," Information and Software Technology, vol. 53, pp. 557-573, 2011.
[50] C. Desai, D. Janzen, and J. Clements, "Implications of Integrating Test-Driven Development into CS1/CS2 Curricula," ACM SIGCSE Bulletin, vol. 41, no. 1, pp. 148-152, 2009.
[51] T. Flohr and T. Schneider, "Lessons Learned from an XP Experiment with Students: Test-First Needs More Teachings," Product-Focused Software Process Improvement, pp. 305-318, Springer, 2006.
[52] L. Huang and M. Holcombe, "Empirical Investigation Towards the Effectiveness of Test First Programming," Information and Software Technology, vol. 51, no. 1, pp. 182-194, 2009.
[53] R. Kaufmann and D. Janzen, "Implications of Test-Driven Development: A Pilot Study," Proc. 18th Ann. ACM SIGPLAN Conf. Object-Oriented Programming, Systems, Languages, and Applications, pp. 298-299, 2003.
[54] M.M. Müller and O. Hagner, "Experiment about Test-First Programming," Proc. IEE Software, vol. 149, no. 5, pp. 131-136, Oct. 2002.
[55] M. Pančur, M. Ciglarič, M. Trampuš, and T. Vidmar, "Towards Empirical Evaluation of Test-Driven Development in a University Environment," Proc. Int'l Conf. Computer as a Tool, pp. 83-86, 2003.
[56] S. Rahman, "Applying the TBC Method in Introductory Programming Courses," Proc. 37th Ann. Frontiers in Education Conf.-Global Eng.: Knowledge without Borders, Opportunities without Passports, pp. T1E-20-T1E-21, 2007.
[57] J. Vu, N. Frojd, C. Shenkel-Therolf, and D. Janzen, "Evaluating Test-Driven Development in an Industry-Sponsored Capstone Project," Proc. Sixth Int'l Conf. Information Technology: New Generations, pp. 229-234, 2009.
[58] S. Xu and T. Li, "Evaluation of Test-Driven Development: An Academic Case Study," Software Eng. Research, Management and Applications, pp. 229-238, Springer, 2009.
[59] S. Yenduri and L. Perkins, "Impact of Using Test-Driven Development: A Case Study," Software Eng. Research and Practice, pp. 126-129, 2006.
[60] L. Zhang, S. Akifuji, K. Kawai, and T. Morioka, "Comparison between Test Driven Development and Waterfall Development in a Small-Scale Project," Extreme Programming and Agile Processes in Software Engineering, pp. 211-212, Springer, 2006.
[61] T. Dogša and D. Batič, "The Effectiveness of Test-Driven Development: An Industrial Case Study," Software Quality J., vol. 19, pp. 643-661, 2011.
[62] K. Lui and K. Chan, "Test Driven Development and Software Process Improvement in China," Extreme Programming and Agile Processes in Software Eng., pp. 219-222, Springer, 2004.
[63] O. Slyngstad, J. Li, R. Conradi, H. Ronneberg, E. Landre, and H. Wesenberg, "The Impact of Test Driven Development on the Evolution of a Reusable Framework of Components: An Industrial Case Study," Proc. Third Int'l Conf. Software Eng. Advances, pp. 214-223, 2008.
[64] P. Hodgetts, "Refactoring the Development Process: Experiences with the Incremental Adoption of Agile Practices," Proc. Agile Development Conf., pp. 106-113, 2004.
[65] B. Turhan, L. Layman, M. Diep, H. Erdogmus, and F. Shull, "How Effective Is Test Driven Development?" Making Software What Really Works, and Why We Believe It, A. Oram and G. Wilson, eds. pp. 207-219, O'Reilly Media, 2010.
[66] M. Müller and A. Höfer, "The Effect of Experience on the Test-Driven Development Process," Empirical Software Eng., vol. 12, pp. 593-615, 2007.
[67] A. Höfer and M. Philipp, "An Empirical Study on the TDD Conformance of Novice and Expert Pair Programmers," Agile Processes in Software Eng. and Extreme Programming, pp. 33-42, Springer, 2009.
[68] L. Madeyski, "Is External Code Quality Correlated with Programming Experience or Feelgood Factor?" Extreme Programming and Agile Processes in Software Eng., pp. 65-74, Springer, 2006.
[69] M. Siniaalto, "Test Driven Development: Empirical Body of Evidence," Information Technology for European Advancement, Eindhoven, The Netherlands, Agile Deliverable, vol. D.2.7, 2006.
[70] S. Kollanus, "Test-Driven Development—Still a Promising Approach?" Proc. Seventh Int'l Conf. Quality of Information and Comm. Technology, pp. 403-408, Oct. 2010.
[71] L. Damm and L. Lundberg, "Quality Impact of Introducing Component-Level Test Automation and Test-Driven Development," Software Process Improvement, P. Abrahamsson, N. Baddoo, T. Margaria, and R. Messnarz, eds. vol. 4764, pp. 187-199, Springer, 2007.
34 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool