This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Using Version Control Data to Evaluate the Impact of Software Tools: A Case Study of the Version Editor
July 2002 (vol. 28 no. 7)
pp. 625-637

Software tools can improve the quality and maintainability of software, but are expensive to acquire, deploy, and maintain, especially in large organizations. We explore how to quantify the effects of a software tool once it has been deployed in a development environment. We present an effort-analysis method that derives tool usage statistics and developer actions from a project's change history (version control system) and uses a novel effort estimation algorithm to quantify the effort savings attributable to tool usage. We apply this method to assess the impact of a software tool called VE, a version-sensitive editor used in Bell Labs. VE aids software developers in coping with the rampant use of certain preprocessor directives (similar to #if/#endif in C source files). Our analysis found that developers were approximately 40 percent more productive when using VE than when using standard text editors.

[1] A.J. Albrecht and J.R. Gaffney, “Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation,” IEEE Trans. Software Eng., vol. 9, no. 6, pp. 638–648, 1983.
[2] D. Atkins, T. Ball, T. Graves, and A. Mockus, “Using Version Control Data to Evaluate the Effectiveness of Software Tools,” Proc. Int'l Conf. Software Eng., pp. 324–333, 1999.
[3] D. Atkins, A. Mockus, and H. Siy, “Measuring Technology Effects on Software Change Cost,” Bell Labs Technical J., vol. 5, no. 2, pp.  7–18, Apr./June 2000.
[4] D.L. Atkins, “Version Sensitive Editing: Change History as a Programming Tool,” Proc. Eighth Conf. Software Configuration Management (SCM-8), pp. 146–157, 1998.
[5] V. Basili and R. Reiter, “An Investigation of Human Factors in Software Development,” Computer, vol. 12, no. 12, pp. 21–38, Dec. 1979.
[6] B. Boehm, Software Engineering Economics, Prentice Hall, Upper Saddle River, N.J., 1981, pp. 533-535.
[7] A. Brown and K. Wallnau, "A Framework for Evaluating Software Technology," IEEE Software, vol. 13, no. 5, Sept. 1996, pp. 39-49.
[8] T. Bruckhaus, N. Madhavji, I. Janssen, and J. Henshaw, “The Impact of Tools on Software Productivity,” IEEE Software, pp. 29–38, Sept. 1996.
[9] J.O. Coplien, D.L. DeBruler, and M.B. Thompson, “The Delta System: A Nontraditional Approach to Software Version Management,” Int'l Switching Symp., Mar. 1987.
[10] B. Curtis, “Substantiating Programmer Variability,” Proc. IEEE 69, July 1981.
[11] A.P. Dempster, N. Laird, and D.B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. Royal Statistical Soc. B, vol. 39, pp. 1–38, 1977.
[12] H. Gottfried and M. Burnett, “Programming Complex Objects in Spreadsheets: An Empirical Study Comparing Textual Formula Entry with Direct Manipulation and Gestures,” Proc. Seventh Workshop Empirical Studies of Programming, 1997.
[13] T.L. Graves and A. Mockus, ”Inferring Change Effort from Configuration Management Databases,” Proc. Metrics 98, 1998.
[14] T.L. Graves and A. Mockus, “Identifying Productivity Drivers by Modeling Work Units Using Partial Data,” Technometrics, vol. 43, no. 2, pp. 168–179, May 2001.
[15] B. Kitchenham, L. Pickard, and S. Pfleeger, “Case Studies for Method and Tool Evaluation,” IEEE Software, vol. 11, no. 4, pp. 52–62, July 1995.
[16] A. Lawrence, A. Badre, and J. Stasko, “Empirically Evaluating the Use of Animations to Teach Algorithms,” Proc. 1994 IEEE Symp. Visual Languages, pp. 48–54, Oct. 1994.
[17] J. Lyle and M. Weiser, "Experiments on Slicing-Based Debugging Tools," Proc. First Conf. Empirical Studies of Programming, pp. 187-197, June 1986,.
[18] P. McCullagh and J.A. Nelder, Generalized Linear Models, second ed. New York: Chapman and Hall, 1989.
[19] A.K. Midha, “Software Configuration Management for the 21st Century,” Bell Labs Technical J., vol. 2, no. 1, Winter 1997.
[20] A. Mockus and L.G. Votta, Identifying Reasons for Software Change Using Historic Databases Proc. Int'l Conf. Software Maintenance, pp. 120-130, 2000.
[21] A. Mockus and D.M. Weiss, “Predicting Risk of Software Changes,” Bell Labs Technical J., vol. 5, no. 2, pp. 169–180, Apr./June 2000.
[22] V. Mosley, “How to Assess Tools Efficiently and Quantitatively,” IEEE Software, pp. 29–32 May 1992.
[23] T. Omerod and L. Ball, “An Empirical Evaluation of TEd, a Techniques Editor for Prolog Programming,” Proc. Sixth Workshop Empirical Studies of Programming, 1996.
[24] A. Pal and M. Thompson, “An Advanced Interface to a Switching Software Version Management System,” Seventh Int'l Conf. Software Eng. for Telecomm. Switching Systems, July 1989.
[25] R. Poston and M. Sexton, “Evaluating and Selecting Testing Tools,” IEEE Software, pp. 33–42, May 1992.
[26] M. Rochkind, “The Source Code Control System,” IEEE Trans. Software Eng., vol. 1, no. 4, pp. 364–370, 1975.
[27] M.J. Shepperd and C. Schofield, “Estimating Software Project Effort Using Analogies,” IEEE Trans. Software Eng., vol. 23, pp. 736-743, 1997.
[28] B. Shneiderman, Designing the User Interface, second ed. Addison-Wesley, 1991.
[29] H. Siy and A. Mockus, “Measuring Domain Engineering Effects on Software Coding Cost,” Metrics 99: Sixth Int'l Symp. Software Metrics, pp. 304–311, Nov. 1999.
[30] H. Siy and A. Mockus, “Measuring Domain Engineering Effects on Software Coding Cost,” Metrics 99: Sixth Int'l Symp. Software Metrics, pp. 304–311, Nov. 1999.
[31] G. Snelting, “Reengineering of Configurations Based on Mathematical Concept Analysis,” ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 146-189, Apr. 1996.
[32] E.B. Swanson, "The Dimensions of Maintenance," Proc. Second Int'l Conf. Software Eng., pp. 492-497, 1976.

Index Terms:
Software tools, version control system, effort analysis.
Citation:
David L. Atkins, Thomas Ball, Todd L. Graves, Audris Mockus, "Using Version Control Data to Evaluate the Impact of Software Tools: A Case Study of the Version Editor," IEEE Transactions on Software Engineering, vol. 28, no. 7, pp. 625-637, July 2002, doi:10.1109/TSE.2002.1019478
Usage of this product signifies your acceptance of the Terms of Use.