This Article 
 Bibliographic References 
 Add to: 
Measuring Process Consistency: Implications for Reducing Software Defects
November/December 1999 (vol. 25 no. 6)
pp. 800-815

Abstract—In this paper, an empirical study that links software process consistency with product defects is reported. Various measurement issues such as validity, reliability, and other challenges in measuring process consistency at the project level are discussed. A measurement scale for software process consistency is introduced. An empirical study that uses this scale to measure consistency in achieving the CMM goal questions in various key process areas (KPAs) in 45 projects at a leading software vendor is reported. The results of this analysis indicate that consistent adoption of practices specified in the CMM is associated with a lower number of defects. Even a relatively modest improvement in the consistency of implementing these practices is associated with a significant reduction in field defects.

[1] S. Bandinelli, A. Fuggetta, C. Ghezzi, and L. Lavazza, “SPADE: An Environment for Software Process Analysis, Design, and Enactment,” Software Process Modelling and Technology, A. Finkelstein, J. Kramer, and B. Nuseibeh, eds., 1994.
[2] V.R. Basili, “The Experience Factory and Its Relationship to Other Quality Approaches,” Advances in Computers, M.V. Zelkowitz, ed., vol. 41, pp. 65-82, Boston: Academic Press, 1995.
[3] V.R. Basili, M. Zelkowitz, F. McGarry, J. Page, S. Waligora, and R. Pajerski, “SEL's Software Process Improvement Program,” IEEE Software, vol. 12, no. 6, pp. 83-87, Nov. 1995.
[4] B.W. Boehm, Software Eng. Economics. Englewood Cliffs, N.J.: Prentice Hall, 1981.
[5] T.B. Bollinger and C. McGowen, “A Critical Look at Software Capability Evaluations,” IEEE Software, pp. 25-41, July 1991.
[6] A.M. Christie, Software Process Automation: The Technology and Its Adoption. Berlin: Springer-Verlag, 1995.
[7] B.K. Clark, “The Effects of Software Process Maturity on Software Development Effort,” Univ. of Southern Calif., 1997. unpublished
[8] S.D. Conte, H. E. Dunsmore, and V. Y. Shen, Software Engineering Metrics and Models, Benjamin/Cummings, Menlo Park, Calif., 1986.
[9] J.E. Cook, L.G. Votta, and A.L. Wolf, “Cost-Effective Analysis of In-Place Software Process,” IEEE Trans. Software Eng., 1998. to appear
[10] R.D. Cook and S. Weisberg, Residuals and Influence in Regression. London: Chapman&Hall, 1982.
[11] B. Curtis, “The Factor Structure of the CMM and Other Latent Issues,” Proc. 1996 SEPG Conf., Atlantic City, N.J., 1996.
[12] B. Curtis, M. Kellner, and J. Over, “Process Modeling,” Comm. ACM, pp. 75-90, Sept. 1992.
[13] G. Cugola, E. Di Nitto, C. Ghezzi, and M. Mantione, “How to Deal with Deviations During Process Model Enactment,” Proc. 17th Int'l Conf. Software Eng., Seattle, New York: ACM, 1995.
[14] G. Cugola, E. Di Nitto, A. Fuggetta, and C. Ghezzi, “A Framework for Formalizing Inconsistencies and Deviations in Human-Centered Systems,” ACM Trans. Software Eng. and Methodology, vol. 5, no. 3, pp. 191-230, July 1996.
[15] F.D. Davis, “Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information Technology,” MIS Quarterly, vol. 13, no. 3, pp. 319-340, Sept. 1989.
[16] K. El Emam and D.R. Goldenson, “Some Initial Results from the International SPICE Trials,” Software Process Newsletter, Technical Council on Software Eng., vol. 6,IEEE Computer Society, Spring 1996.
[17] K. El Emam and N.H. Madhavji, “Reliability of Measuring Organizational Maturity,” Software Process—Improvements and Practice, vol. 1, pp. 3-25, 1995.
[18] P. Ferguson et al., "Results of Applying the Personal Software Process," Computer, Vol. 30 No. 5, May 1997, pp. 24-31.
[19] A. Finkelstein, J. Kramer, and B. Nuseibeh, Software Process Modeling and Technology. Research Studies Press, U.K., 1994.
[20] C. Fox and W. Frakes, “The Quality Approach: Is it Delivering?” Comm. ACM, vol. 40, no. 6, pp. 25-29, 1997.
[21] R.L. Glass, “The Software-Research Crisis,” IEEE Software, Nov. 1994, pp. 42-47.
[22] D. Goldenson, “A Multiple Response Scale for Process Measurement,” working report, Software Eng. Inst., Carnegie Mellon Univ., Pittsburgh, 1994.
[23] A. Gopal, T. Mukhopadhyay, and M.S. Krishnan, “The Role of Software Process and Communication in Offshore Software Development,” Comm. ACM, 1998.
[24] J. Hatz et al., Trillium Method, Method Issue 1.0, NortelBNR Corporate Standard 133.21
[25] W. Hayes and J.W. Over, “The Personal Software Process (PSP): An Empirical Study of the Impact of PSP on Individual Engineers,” Technical Report, CMU/SEI-97-TR-001, Software Eng. Inst., Pittsburgh, 1997.
[26] T.G. Heineman, J.E. Botsford, G. Caldiera, G.E. Kaiser, M.I. Kellner, and N.H. Madhavji, “Emerging Technologies that Support a Software Process Life Cycle,” IBM Systems J., vol. 33, no. 3, pp. 501-529, 1994.
[27] J. Herbsleb, D. Zubrow, D. Goldenson, W. Hayes, and M. Paulk, “Software Quality and the Capability Maturity Model,” Comm. ACM, vol. 40, no. 6, pp. 30-40, 1997.
[28] W.S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, Mass., 1989.
[29] W.S. Humphrey, A Discipline for Software Engineering, Addison-Wesley, Reading, Mass., 1995.
[30] Industry Surveys, “Software Industry: A Huge Market, Getting Bigger All the Time,” pp. C108, Nov.
[31] C. Inwood, “Standards May Solve User Frustration,” Computing Canada, vol. 20,no, 2, 1994.
[32] C. Jones, Applied Software Measurement, McGraw-Hill, New York, 1991.
[33] M. Keil, “Pulling the Plug: Software Project Management and the Problem of Project Escalation,” MIS Quarterly, vol. 19, no. 4, pp. 421-447, Dec. 1995.
[34] M.I. Kellner, L.C. Briand, and J.W. Over, “A Method for Designing, Defining, and Evolving Software Processes,” Proc. Fourth Int'l Conf. Software Process: Improvement and Practice, Brighton, England U.K., pp. 37-48, IEEE CS Press, Dec. 1996.
[35] S. Kekre, M.S. Krishnan, and K. Srinivasan, “Drivers of Customer Satisfaction for Software Products: Implications for Design and Service Support,” Management Science, vol. 41, no. 9, pp. 1,456-1,470, 1995.
[36] C.F. Kemerer, Software Project Management Readings and Cases. McGraw-Hill, 1997.
[37] J. Kmenta, Elements of Econometrics. New York: Macmillan, 1986.
[38] M.S. Krishnan, “Cost and Quality Considerations in Software Product Management,” Carnegie Mellon Univ., Pittsburgh, 1996. unpublished doctoral dissertation,
[39] M.S. Krishnan and M.I. Kellner, “Software Process Measurement: Implications for Improving Software Quality,” working paper, Univ. Michigan Business School, Ann Arbor, 1998.
[40] M.S. Krishnan, “The Role of Team Factors in Software Cost and Quality: An Empirical Analysis,” Information Technology and People, vol. 11, no. 1, 1998.
[41] P. Kuvaja, J. Similä, L.L. Krzanik, A. Bicego, S. Saukkonen, and G. Koch, Software Process Assessment&Improvement: The Bootstrap Approach. U.K.: Blackwell, 1994.
[42] G.C. Moore and I. Benbasat, “Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation,” Information Systems Research, vol. 2, no. 3, pp. 192-221, 1991.
[43] A.M. Newfelder, Ensuring Software Reliability. New York: Marcel Dekker, Inc., 1993.
[44] J.C. Nunnally, Psychometric Theory. McGraw-Hill, 1978.
[45] G. Pall, Quality Process Management. Englewood Cliffs, N.J.: Prentice Hall, 1987.
[46] M. Paulk, “How ISO 9001 Compares with the CMM,” IEEE Software, pp 74-83, Jan. 1995.
[47] M. Paulk, B. Curtis, M.B. Chrissis, and C. Weber, “Capability Maturity Model,”version 1.1, Technical Report, CMU/SEI-93-TR-24, Software Eng. Inst., Pittsburgh, 1993.
[48] D.E. Perry, N.A. Staudenmayer, and L.G. Votta, "People, Organizations, and Process Improvement." IEEE Software, vol. 11, no. 4, pp. 36-45, 1994.
[49] D.M. Raffo and M.I. Kellner, “Modeling Software Processes Quantitatively and Evaluating the Performance of Process Alternatives,” Elements of Software Process Assessment and Improvement, ch. 16, pp. 297-341, K. El Emam, and H.N. Madhavji, eds., IEEE CS Press, 1999.
[50] H. White, “Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,” Econometrica, vol. 48, no. 5, pp. 817-838, 1980.
[51] R. Whitney, E. Nowrocki,W. Hayes, and J. Siegel, “Interim Profile Development and Trial of a Method to Rapidly Measure Software Engineering Maturity Status,” Technical Report CMU/SEI-TR-4, Software Eng. Inst., Pittsburgh, Mar. 1994.
[52] D. Zubrow, W. Hayes, J. Siegel, and D. Goldenson, “Maturity Questionnaire,” Technical Report CMU/SEI-94-SR-007, Software Eng. Inst., Pittsburgh, 1994.
[53] D.A. Belsley, E. Kuh, and R. Ewelsch, Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. John Wiley&Sons, 1980.

Index Terms:
Software process consistency, software process measurement, CMM, software defects, empirical model.
M.s. Krishnan, Marc I. Kellner, "Measuring Process Consistency: Implications for Reducing Software Defects," IEEE Transactions on Software Engineering, vol. 25, no. 6, pp. 800-815, Nov.-Dec. 1999, doi:10.1109/32.824401
Usage of this product signifies your acceptance of the Terms of Use.