This Article 
 Bibliographic References 
 Add to: 
On the Effectiveness of the Test-First Approach to Programming
March 2005 (vol. 31 no. 3)
pp. 226-237
Maurizio Morisio, IEEE Computer Society
Marco Torchiano, IEEE Computer Society
Test-Driven Development (TDD) is based on formalizing a piece of functionality as a test, implementing the functionality such that the test passes, and iterating the process. This paper describes a controlled experiment for evaluating an important aspect of TDD: In TDD, programmers write functional tests before the corresponding implementation code. The experiment was conducted with undergraduate students. While the experiment group applied a test-first strategy, the control group applied a more conventional development technique, writing tests after the implementation. Both groups followed an incremental process, adding new features one at a time and regression testing them. We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

[1] K. Beck, Test-Driven Development: By Example. Addison Wesley, 2003.
[2] D. Astels, Test Driven Development: A Practical Guide. Prentice Hall, 2003.
[3] K. Beck, Extreme Programming Explained: Embrace Change. Addison-Wesley, 2000.
[4] D. Astels, G. Miller, and M. Novak, A Practical Guide to Extreme Programming. Prentice Hall, 2002.
[5],, 2004.
[6] K. Beck and E. Gamma, “Test Infected: Programmers Love Writing Tests,” Java Report, vol. 3, pp. 51-56, 1998.
[7] R.C. Linger, H.D. Mills, and M. Dyer, “Cleanroom Software Engineering,” IEEE Software, Sept. 1987.
[8] M.M. Müller and W.F. Tichy, “Case Study: Extreme Programming in a University Environment,” Proc. Int'l Conf. Software Eng. (ICSE), 2001.
[9] M.M. Müller and O. Hagner, “Experiment about Test-First Programming,” Proc. Conf. Empirical Assessment in Software Eng. (EASE), 2002.
[10] E.M. Maximilien and L. Williams, “Assessing Test-Driven Development at IBM,” Proc. Int'l Conf. Software Eng. (ICSE), 2003.
[11] S.H. Edwards, “Using Test-Driven Development in the Classroom: Providing Students with Concrete Feedback on Performance,” Proc. Int'l Conf. Education and Information Systems: Technologies and Applications (EISTA '03), Aug. 2003.
[12] B. George and L. Williams, “An Initial Investigation of Test Driven Development in Industry,” Proc. ACM Symp. Applied Computing, 2003.
[13] F. Maurer and S. Martel, “On the Productivity of Agile Software Practices: An Industrial Case Study,” technical report, Univ. of Calgary, Alberta, 2001, pdf .
[14] APA, Publication Manual of the Am. Psychological Association, fourth ed. Washington DC: Am. Psychological Association, 1994.
[15] C. Wohlin, P. Runeson, M. Host, M.C. Ohlsson, B. Regnell, and A. Wesslen, Experimentation in Software Engineering: An Introduction. Kluwer Academic, 2000.
[16] B. Boehm, “Industrial Metrics Top-10 List,” IEEE Software, pp. 84-85, Sept. 1987.
[17] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, second ed. Hillsdale, N.J.: Lawrence Earlbaum and Associates, 1988.
[18] “Eclipse Platform Technical Overview,” vol. 2004, Object Technology Int'l, 2003, eclipseoverview.pdf .
[19] P. Cederqvist, “Version Management with CVS,” Bristol, U.K.: Network Theory Ltd., 1993.
[20] H.B. Mann and D.R. Whitney, “On a Test of Whether One of Two Random Variables is Stochastically Larger than the Other,” Annals of Math. Statistics, 1947.
[21] P. Runeson, “Using Students as Experiment Subjects— An Analysis on Graduate and Freshmen Student Data,” Proc. Seventh Int'l Conf. Empirical Assessment and Evaluation in Software Eng. (EASE '03), 2003.
[22] A. Porter and L. Votta, “Comparing Detection Methods for Software Requirements Inspection: A Replication Using Professional Subjects,” Empirical Software Eng., vol. 3, pp. 355-380, 1995.
[23] M. Höst, B. Regnell, and C. Wohlin, “Using Students as Subjects— A Comparative Study of Students and Professionals in Lead-Time Impact Assessment,” Empirical Software Eng., vol. 5, pp. 201-214, 2000.
[24] E. Arisholm and D.I.K. Sjøberg, “A Controlled Experiment with Professionals to Evaluate the Effect of a Delegated versus Centralized Control Style on the Maintainability of Object-Oriented Software,” Technical Report 2003-6, Simula Research Laboratory, Norway, June 2003, .
[25] H. Erdogmus and Y. Wang, “The Role of Process Measurement in Test-Driven Development,” Proc. XP/Agile Universe Conf. Extreme Programming and Agile Methods, Aug. 2004.

Index Terms:
General programming techniques, coding tools and techniques, testing and debugging, testing strategies, productivity, Software Quality/SQA, software engineering process, programming paradigms.
Hakan Erdogmus, Maurizio Morisio, Marco Torchiano, "On the Effectiveness of the Test-First Approach to Programming," IEEE Transactions on Software Engineering, vol. 31, no. 3, pp. 226-237, March 2005, doi:10.1109/TSE.2005.37
Usage of this product signifies your acceptance of the Terms of Use.