This Article 
 Bibliographic References 
 Add to: 
Cost-Effective Analysis of In-Place Software Processes
August 1998 (vol. 24 no. 8)
pp. 650-663

Abstract—Process studies and improvement efforts typically call for new instrumentation on the process in order to collect the data they have deemed necessary. This can be intrusive and expensive, and resistance to the extra workload often foils the study before it begins. The result is neither interesting new knowledge nor an improved process. In many organizations, however, extensive historical process and product data already exist. Can these existing data be used to empirically explore what process factors might be affecting the outcome of the process? If they can, organizations would have a cost-effective method for quantitatively, if not causally, understanding their process and its relationship to the product. We present a case study that analyzes an in-place industrial process and takes advantage of existing data sources. In doing this, we also illustrate and propose a methodology for such exploratory empirical studies. The case study makes use of several readily available repositories of process data in the industrial organization. Our results show that readily available data can be used to correlate both simple aggregate metrics and complex process metrics with defects in the product. Through the case study, we give evidence supporting the claim that exploratory empirical studies can provide significant results and benefits while being cost effective in their demands on the organization.

[1] V.R. Basili and D.M. Weiss, "A Methodology for Collecting Valid Software Engineering Data," IEEE Trans. Software Eng., vol. 10, no. 6, pp. 728-737, 1984.
[2] I. Bhandari, M. Halliday, E. Tarver, D. Brown, J. Chaar, and R. Chillarege, "A Case Study of Software Process Improvement During Development," IEEE Trans. Software Eng., vol. 19, no. 12, pp. 1,157-1,170, Dec. 1993.
[3] M.G. Bradac, D.E. Perry, and L.G. Votta, "Prototyping a Process Monitoring Experiment," IEEE Trans. Software Eng., pp. 774-784, Oct. 1994.
[4] L.J. Chmura, A.F. Norcio, and T.J. Wicinski, "Evaluating Software Design Process by Analyzing Change Data Over Time," IEEE Trans. Software Eng., vol. 16, no. 7, pp. 729-739, July 1990.
[5] J.E. Cook, "Process Discovery and Validation Through Event-Data Analysis," Technical Report CU-CS-817-96, Dept. of Computer Science, Univ. of Colorado, Nov. 1996.
[6] J.E. Cook and A.L. Wolf, "Toward Metrics for Process Validation," Proc. Third Int'l Conf. Software Process, pp. 33-44, IEEE Computer Society, Oct. 1994.
[7] J.E. Cook and A.L. Wolf, "Software Process Validation: Quantitatively Measuring the Correspondence of a Process to a Model Using Event-Based Data," Technical Report CU-CS-820-96, Dept. of Computer Science, Univ. of Colorado, Nov. 1996.
[8] J.L. Devore, Probability and Statistics for Engineering and the Sciences third edition. Pacific Grove, Calif.: Brooks/Cole, 1991.
[9] C.M. Judd, E.R. Smith, and L.H. Kidder, Research Methods in Social Relations, sixth edition. Ft. Worth: Holt, Rinehart and Winston, Inc., 1991.
[10] A. Kouchakdjian, S. Green, and V.R. Basili, "Evaluation of the Cleanroom Methodology in the Software Engineering Laboratory," Proc. 14th Software Eng. Workshop, NASA Goddard Space Flight Center, Greenbelt, Md, 1989.
[11] R.W. Selby, A.A. Porter, D.C. Schmidt, and J. Berney, "Metric-Driven Analysis and Feedback Systems for Enabling Empirically Guided Software Development," Proc. 13th Int'l Conf. Software Eng., pp. 288-298. IEEE CS Press, May 1991.
[12] S.W. Sherer, A. Kouchakdjian, and P.G. Arnold, "Experience Using Cleanroom Software Engineering," IEEE Software, vol. 13, no. 3, May 1996.
[13] S.A. Lee, “A Scientific Methodology for MIS Case Studies,” MIS Quarterly, vol. 13, no. 1, pp. 33–50, 1989.
[14] Statistical Methods, eighth edition. G.W. Snedecor and W.G. Cochran, eds., Iowa State Univ. Press, 1989.
[15] L.G. Votta and M.L. Zajac, "Design Process Improvement Case Study Using Process Waiver Data," Proc. Fifth European Software Eng. Conf., ESEC'95, pp. 44-58. Springer-Verlag, Sept. 1995.
[16] A.L. Wolf and D.S. Rosenblum, "A Study in Software Process Data Capture and Analysis," Proc. Second Int'l Conf. Software Process, pp. 115-124. IEEE Computer Society, Feb. 1993.
[17] R. Yin, Case Study Research: Design and Method, second edition. Thousand Oaks, Calif.: SAGE Publications, 1994.

Index Terms:
Software process, process improvement, retrospective case study, empirical case study, process measurement, process model validation.
Jonathan E. Cook, Lawrence G. Votta, Alexander L. Wolf, "Cost-Effective Analysis of In-Place Software Processes," IEEE Transactions on Software Engineering, vol. 24, no. 8, pp. 650-663, Aug. 1998, doi:10.1109/32.707700
Usage of this product signifies your acceptance of the Terms of Use.