This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Does Software Process Improvement Reduce the Severity of Defects? A Longitudinal Field Study
July-Aug. 2012 (vol. 38 no. 4)
pp. 810-827
Donald E. Harter, Syracuse University, Syracuse
Chris F. Kemerer, University of Pittsburgh, Pittsburgh and King Abdul Aziz University, Saudi Arabia
Sandra A. Slaughter, Georgia Institute of Technology, Atlanta
As firms increasingly rely on information systems to perform critical functions, the consequences of software defects can be catastrophic. Although the software engineering literature suggests that software process improvement can help to reduce software defects, the actual evidence is equivocal. For example, improved development processes may only remove the “easier” syntactical defects, while the more critical defects remain. Rigorous empirical analyses of these relationships have been very difficult to conduct due to the difficulties in collecting the appropriate data on real systems from industrial organizations. This field study analyzes a detailed data set consisting of 7,545 software defects that were collected on software projects completed at a major software firm. Our analyses reveal that higher levels of software process improvement significantly reduce the likelihood of high severity defects. In addition, we find that higher levels of process improvement are even more beneficial in reducing severe defects when the system developed is large or complex, but are less beneficial in development when requirements are ambiguous, unclear, or incomplete. Our findings reveal the benefits and limitations of software process improvement for the removal of severe defects and suggest where investments in improving development processes may have their greatest effects.

[1] The Economic Impacts of Inadequate Infrastructure for Software Testing, US Dept. of Commerce, Nat'l Inst. of Standards and Tech nology, 2002.
[2] C. Mann, "Why Software Is So Bad...and What's Being Done to Fix It," MIT Technology Rev., vol. 105, pp. 33-38, 2002.
[3] J. Herbsleb, D. Zubrow, D. Goldenson, W. Hayes, and M. Paulk, "Software Quality and the Capability Maturity Model," Comm. ACM, vol. 40, no. 6, pp. 30-40, 1997.
[4] M.S. Krishnan, C.H. Kriebel, S. Kekre, and T. Mukhopadhyay, "An Empirical Analysis of Productivity and Quality in Software Products," Management Science, vol. 46, no. 6, pp. 745-759, 2000.
[5] D.E. Harter, M.S. Krishnan, and S.A. Slaughter, "Effects of Process Maturity on Quality, Cycle Time and Effort in Software Product Development," Management Science, vol. 46, pp. 451-466, 2000.
[6] P.B. Crosby, Quality Is Free. McGraw-Hill, 1979.
[7] C. Jones, "Measuring Defect Potentials and Defect Removal Efficiency," CrossTalk, http://www.stsc.hill.af.mil/crosstalk/2008/ 060806Jones.html, June 2008.
[8] K. Belson and M.L. Wald, "'03 Blackout Is Recalled, Amid Lessons Learned," New York Times, Aug. 2008.
[9] G. Prophet, "A Hard Look at Software Reliability," Global EDN Report, http://www.edn.com/articleCA608882.html? industryid=23439 , June 2005.
[10] B.W. Boehm, Software Engineering Economics. Prentice Hall, 1981.
[11] M. Lipow, "Prediction of Software Failures," The J. Systems and Software, vol. 1, no. 1, pp. 71-75, 1979.
[12] J.A. Hager, "Software Cost Reduction Methods in Practice," IEEE Trans. Software Eng., vol. 15, no. 12, pp. 1638-1644, Dec. 1989.
[13] T. Gyimothy, R. Ference, and L. Siket, "Empirical Validation of Object-Oriented Metrics on Open Source Software for Fault Predictions," IEEE Trans. Software Eng., vol. 31, no. 10, pp. 897-910, Oct. 2005.
[14] M. Alshayeb and L. Wei, "An Empirical Validation of Object-Oriented Metrics in Two Different Iterative Software Processes," IEEE Trans. Software Eng., vol. 29, no. 11, pp. 1043-1049, Nov. 2003.
[15] K. El Emam, S. Benlarbi, N. Goel, and S.N. Raj, "The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics," IEEE Trans. Software Eng., vol. 27, no. 7, pp. 630-650, July 2001.
[16] J. Lo and C. Huang, "An Integration of Fault Detection and Correction Processes in Software Reliability Analysis," The J. Systems and Software, vol. 79, no. 9, pp. 1312-1323, 2006.
[17] K.-Y. Cai, "On Estimating the Number of Defects Remaining in Software," The J. Systems and Software, vol. 40, no. 2, pp. 93-114, 1998.
[18] R.B. Grady, Practical Software Metrics for Project Management and Process Improvement. Prentice Hall, 1992.
[19] S. Zahran, Software Process Improvement: Practical Guidelines for Business Success. Addison Wesley Longman Ltd., 1998.
[20] R. Chillarege, "Orthogonal Defect Classification," Handbook of Software Reliability Eng., pp. 359-400, IEEE CS Press, 1995.
[21] W.S. Humphrey, Managing the Software Process. Addison-Wesley Publishing, 1989.
[22] M. Donnelly, B. Everett, J. Musa, and G. Wilson, "Best Current Practice of Software Reliability Engineering (SRE)," Handbook of Software Reliability Eng., pp. 219-254, IEEE CS Press, 1995.
[23] L.H. Putnam and W. Myers, Industrial Strength Software: Effective Management Using Measurement, p. 44. IEEE CS Press, 1997.
[24] C. Jones, Applied Software Measurement: Assuring Productivity and Quality. McGraw-Hill, 1996.
[25] ANSI/IEEE Std 729-1983, Glossary of Software Eng. Terminology, IEEE, 1987.
[26] J.S. Poulin, Measuring Software Reuse, Principles, Practices, and Economic Models. Addison-Wesley, 1997.
[27] J. Gao, H.J. Tsao, and Y. Wu, Testing and Quality Assurance for Component-Based Software. Artech House, 2003.
[28] Quality Standards Defect Measurement Manual. United Kingdom Software Metrics Assoc. (UKSMA), Oct. 2000.
[29] F. Tsui, Managing Software Projects. Jones & Bartlett Publishing, 2004.
[30] Y. Zhou and H. Leung, "Empirical Analysis of Object-Oriented Design Metrics for Predicting High and Low Severity Faults," IEEE Trans. Software Eng., vol. 32, no. 10, pp. 771-789, Oct. 2006.
[31] T.J. Ostrand and E.J. Weyuker, "The Distribution of Faults in a Large Industrial Software System," Proc. Int'l Symp. Software Testing and Analysis, pp. 55-64, 2002.
[32] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, "Predicting the Location and Number of Faults in Large Software Systems," IEEE Trans. Software Eng., vol. 31, no. 4, pp. 340-355, Apr. 2005.
[33] R.B. Grady, Successful Software Process Improvement. Prentice Hall, 1997.
[34] T. Dyba, "An Empirical Investigation of the Key Factors for Success in Software Process Improvement," IEEE Trans. Software Eng., vol. 31, no. 5, pp. 410-424, May 2005.
[35] J.M. Juran, "A Note on Economics of Quality," Industrial Quality Control, pp. 20-23, 1959.
[36] J.M. Juran, Juran on Quality by Design: The New Steps for Planning Quality into Goods and Services. Free Press, 1992.
[37] W.E. Deming, Out of Crisis. MIT Center for Advanced Eng. Study, 1992.
[38] W.S. Humphrey, "Characterizing the Software Process: A Maturity Framework," IEEE Software, vol. 5, no. 3, pp. 73-79, May 1988.
[39] M.C. Paulk, B. Curtis, M.B. Chrissis, and C.V. Weber, "Capability Maturity Model, Version 1.1," IEEE Software, vol. 10, no. 4, pp. 18-27, July 1993.
[40] G. Li and S. Rajagopalan, "Process Improvement, Quality and Learning Effects," Management Science, vol. 44, pp. 1517-1532, 1998.
[41] M.S. Krishnan and M.I. Kellner, "Measuring Process Consistency: Implications for Reducing Software Defects," Management Science, vol. 25, pp. 800-815, 1999.
[42] N. Ramasubbu, S. Mithas, M.S. Krishnan, and C.F. Kemerer, "Work Dispersion, Process-Based Learning and Offshore Software Development Performance," MIS Quarterly, vol. 32, no. 2, pp. 437-458, June 2008.
[43] H. Wohlwend and S. Rosenbaum, "Schlumberger's Software Improvement Program," IEEE Trans. Software Eng., vol. 20, no. 11, pp. 833-839, Nov. 1994.
[44] M. Diaz and J. Sligo, "How Software Process Improvement Helped Motorola," IEEE Software, vol. 14, no. 5, pp. 75-81, Sept./Oct. 1997.
[45] M. Agrawal and K. Chari, "Software Effort, Quality and Cycle Time: A Study of CMM Level 5 Projects," IEEE Trans. Software Eng., vol. 33, no. 3, pp. 145-156, Mar. 2007.
[46] M.C. Paulk, C.V. Weber, B. Curtis, and M.B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process. Addison-Wesley Publishing Company, 1994.
[47] E. Barry, C.F. Kemerer, and S. Slaughter, "How Software Process Automation Affects Software Evolution: A Longitudinal Empirical Analysis," J. Software Maintenance and Evolution, vol. 19, no. 1, pp. 1-31, Jan./Feb. 2007.
[48] C. Andersson and P. Runeson, "A Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems," IEEE Trans. Software Eng., vol. 33, no. 5, pp. 273-286, May 2007.
[49] D.E. Harter and S.A. Slaughter, "The Cascading Effect of Process Maturity on Software Quality," Proc. Int'l Conf. Information Systems, 2000.
[50] C.F. Kemerer, "Software Complexity and Software Maintenance: A Survey of Empirical Research," Annals of Software Eng., vol. 1, no. 1, pp. 1-22, Aug. 1995.
[51] J.W. Cangussu, R.A. DeCarlo, and A.P. Mathur, "A Formal Model of the Software Test Process," IEEE Trans. Software Eng., vol. 28, no. 8, pp. 782-796, Aug. 2002.
[52] B.K. Clark, "Quantifying the Effects of Process Improvement on Effort," IEEE Software, vol. 17, no. 6, pp. 65-70, Nov./Dec. 2000.
[53] R.D. Banker and S.A. Slaughter, "The Moderating Effects of Structure on Volatility and Complexity in Software Enhancement," Information Systems Research, vol. 11, no. 3, pp. 219-240, 2000.
[54] R.D. Banker, S.M. Datar, C.F. Kemerer, and D. Zweig, "Software Complexity and Software Maintenance Costs," Comm. ACM, vol. 36, no. 11, pp. 81-94, 1993.
[55] R.D. Banker, G.B. Davis, and S.A. Slaughter, "Software Development Practices, Software Complexity and Software Maintenance Performance: A Field Study," Management Science, vol. 44, pp. 433-450, 1998.
[56] J.E. Gaffney, "Estimating the Number of Faults in Code," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 459-465, July 1984.
[57] V.Y. Shen, T. Yu, and S.M. Thebut, "Identifying Error-Prone Software—An Empirical Study," IEEE Trans. Software Eng., vol. 11, no. 4, pp. 317-324, Apr. 1985.
[58] R.D. Banker and C.F. Kemerer, "Scale Economies in New Software Development," IEEE Trans. Software Eng., vol. 15, no. 10, pp. 1199-1205, Oct. 1989.
[59] R. Subramanyam and M.S. Krishnan, "Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects," IEEE Trans. Software Eng., vol. 29, no. 4, pp. 297-310, Apr. 2003.
[60] F. Brooks, The Mythical Man Month, anniversary ed. Pearson Education India 1995.
[61] E.T.G. Wang, P. Ju, J.J. Jiang, and G. Klein, "The Effects of Change Control and Management Review on Software Flexibility and Project Performance," Information and Management, vol. 45, no. 7, pp. 438-443, 2008.
[62] D. Damian and J. Chisan, "An Empirical Study of the Complex Relationships between Requirements Engineering Processes and Other Processes that Lead to Payoffs in Productivity, Quality, and Risk Management," IEEE Trans. Software Eng., vol. 32, no. 7, pp. 433-453, July 2006.
[63] K. El Emam, The ROI from Software Quality. Auerbach Publications, 2005.
[64] D.E. Harter and S.A. Slaughter, "Quality Improvement and Infrastructure Activity Costs in Software Development: A Longitudinal Analysis," Management Science, vol. 49, no. 6, pp. 784-800, 2003.
[65] T.F. Liao, Interpreting Probability Models: Logit, Probit, and Other Generalized Linear Models. Sage Publications, 1994.
[66] W.H. Greene, Econometric Analysis, third ed. MacMillan Publishing Company, 1997.
[67] C. Ai and E.C. Norton, "Interaction Terms in Logit and Probit Models," Economics Letters, vol. 80, pp. 123-129, 2003.
[68] E.C. Norton, H. Wang, and C. Ai, "Computing Interaction Effects and Standard Errors in Logit and Probit Models," The Stata J., vol. 4, no. 2, pp. 154-167, 2004.

Index Terms:
Software complexity, defect severity, requirements ambiguity, software process, CMM
Citation:
Donald E. Harter, Chris F. Kemerer, Sandra A. Slaughter, "Does Software Process Improvement Reduce the Severity of Defects? A Longitudinal Field Study," IEEE Transactions on Software Engineering, vol. 38, no. 4, pp. 810-827, July-Aug. 2012, doi:10.1109/TSE.2011.63
Usage of this product signifies your acceptance of the Terms of Use.