This Article 
 Bibliographic References 
 Add to: 
Validation of an Approach for Improving Existing Measurement Frameworks
June 2000 (vol. 26 no. 6)
pp. 484-499

Abstract—Software organizations are in need of methods to understand, structure, and improve the data they are collecting. We have developed an approach for use when a large number of diverse metrics are already being collected by a software organization [1], [2]. The approach combines two methods. One looks at an organization's measurement framework in a top-down goal-oriented fashion and the other looks at it in a bottom-up data-driven fashion. The top-down method is based on a measurement paradigm called Goal-Question-Metric (GQM). The bottom-up method is based on a data mining technique called Attribute Focusing (AF). A case study was executed to validate this approach and to assess its usefulness in an industrial environment. The top-down and bottom-up methods were applied in the customer satisfaction measurement framework at the IBM Toronto Laboratory. The top-down method was applied to improve the customer satisfaction (CUSTSAT) measurement from the point of view of three data user groups. It identified several new metrics for the interviewed groups, and also contributed to better understanding the data user needs. The bottom-up method was used to gain new insights into the existing CUSTSAT data. Unexpected associations between key variables prompted new business insights, and revealed problems with the process used to collect and analyze the CUSTSAT data. This paper uses the case study and its results to qualitatively compare our approach against current ad hoc practices used to improve existing measurement frameworks.

[1] M.G. Mendonça, V.R. Basili, I.S. Bhandari, and J. Dawson, “An Approach to Improving Existing Measurement Frameworks,” IBM Systems J., vol. 37, no. 4, pp. 484–501, Nov. 1998.
[2] M.G. Mendonça, “An Approach to Improving Existing Measurement Frameworks in Software Development Organizations,” PhD thesis, University of Maryland, College Park, MD, also available as CS-TR-3852 and UMIACS-TR-97-82, Dec. 1997.
[3] T. DeMarco, Why Does Software Cost So Much?, Chapter 2: Mad About Measurement. Dorset Housing Publishing, pp. 11–25, 1995.
[4] B.W. Boehm, J.R. Brown, and M. Lipow, “Quantitative Evaluation of Software Quality,” Second Int'l Conf. Software Eng., pp. 592–605, Oct. 1976.
[5] R.A. DeMillo and R.J. Lipton, “Software Project Forecasting,” Software Metrics, A.J. Perlis, F.G. Sayard, and M. Shaw, eds., Cambridge Mass.: MIT Press, 1981.
[6] N. Fenton and A. Melton,“Deriving structurally based software measures,” J. Systems and Software, vol. 12, no. 3, pp. 177-187, 1990.
[7] A.C. Melton et al., "Mathematical Perspective of Software Measures Research," Software Eng. J., Vol. 5, No. 5, 1990, pp. 246-254.
[8] H. Zuse, Software Complexity: Measures and Methods. deGruyter, 1990.
[9] A.J. Albrech and J.E. Gaffney, “Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation,” IEEE Trans. Software Eng., vol. 9, no. 6, pp. 639–647, Nov. 1983.
[10] L.C. Briand, S. Morasca, and V.R. Basili, "Property-Based Software Engineering Measurement," IEEE Trans. Software Eng., vol. 22, no. 1, pp. 68-85, Jan. 1996.
[11] M. Shepperd, “Algebraic Models and Metric Validation,” Formal Aspects of Measurement, I. Somerville and M. Paul, eds., Springer Verlag, 1992.
[12] J. Tian and M.V. Zelkowitz,“A formal model of program complexity and its application,” J. Systems and Software, vol. 17, no. 3, pp. 253-266, 1992.
[13] E.J. Weyuker, "Evaluating Software Complexity Measures," IEEE Trans. Software Eng., Vol. 14, No. 9, 1988, pp. 1357-1365.
[14] B.W. Boehm, “Software Engineering Economics,” IEEE Trans. Software Eng., vol. 10, no. 1, pp. 4–21, Jan. 1984.
[15] L.C. Briand, V.R. Basili, and W.M. Thomas, "A Pattern Recognition Approach for Software Engineering Data Analysis," IEEE Trans. Software Eng., vol. 18, no. 11, pp. 931-942, 1992.
[16] C. Kemerer, "An Empirical Validation of Software Cost Estimation Models," Comm. ACM, vol. 30, pp. 416-429, May 1987.
[17] C.E. Walston and C.P. Felix, “A Method of Programming Measurement and Estimation,” IBM Systems J., vol. 16, no. 1, pp. 54–73, Jan. 1977.
[18] V.R. Basili and R.W. Reiter, Jr., “Evaluating Automatable Measures of Software Development,” Proc. Workshop Quantitative Software Models, Oct. 1979.
[19] V.R. Basili and D.H. Hutchens, “An Empirical Study of a Syntactic Complexity Family,” IEEE Trans. Software Eng., vol. 9, no. 6, pp. 664–672, Nov. 1983.
[20] V.R. Basili, R.W. Selby, and T.Y. Phillips, “Metric Analysis and Validation Across FORTRAN Projects,” IEEE Trans. Software Eng., vol. 9, no. 6, pp. 652–663, Nov. 1983.
[21] T.M. Khoshgoftaar, E.B. Allen, and D.L. Lanning, “An Information Theory-Based Approach to Quantify the Contribution of a Software Metric,” J. Systems and Software, vol. 36, no. 2, pp. 103–113, Feb. 1997.
[22] N. Schneidewind, "Methodology for Validating Software Metrics," IEEE Trans. Software Eng., vol. 18, pp. 410-421, May 1992.
[23] B.A. Kitchenham, S.L. Pfleeger, and N. Fenton, “Towards a Framework for Software Measurement Validation,” IEEE Trans. Software Eng., vol. 21, no. 12, pp. 929-944, Dec. 1995.
[24] M.K. Daskalantonakis, "A Practical View of Software Management and Implementation Experiences within Motorola," IEEE Trans. Software Eng. Vol. 18, No. 11, 1992, pp. 998-1009.
[25] R.B. Grady, “Successfully Applying Software Metrics,” IEEE Computer, vol. 27, no. 9, pp. 18–25, Sept. 1994.
[26] S.L. Pfleeger, "Lessons Learned in Building a Corporate Metrics Program," IEEE Software, May 1993, pp. 67-74.
[27] S. Rifkin and C. Cox, “Measurement in Practice,” Technical report CMU/SEI-91-TR-16, SEI, 1991.
[28] E.F. Weller, "Using Metrics to Manage Software Projects," Computer, Sept. 1994, pp. 27-34.
[29] T. Hall and N. Fenton, “Implementing Effective Software Metrics Programs,” IEEE Software, pp. 55-65, Mar./Apr. 1997.
[30] R.J. Offen and R. Jeffery, “Establishing Software Measurement Programs,” IEEE Software, pp. 45-53, Mar./Apr. 1997.
[31] W. Klösgen and J.M. Zytkow, “Knowledge Discovery in Databases Terminology,” Advances in Knowledge Discovery and Data Mining, U.M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, eds., Cambridge, Mass.: AAAI Press/The MIT Press, 1996.
[32] N. Fenton, "Software Measurement: A Necessary Scientific Bias," IEEE Trans. Software Eng., vol. 20, pp. 199-206, Mar. 1994.
[33] V.R. Basili and H.D. Rombach, "The TAME Project: Towards Improvement-Oriented Software Environments," IEEE Trans. Software Eng., Vol. 14, No. 6, 1988, pp. 758-773.
[34] V.R. Basili and D.M. Weiss, “A Methodology for Collecting Valid Software Engineering Data,” IEEE Trans. Software Eng., vol. 10, no. 6, pp. 728–738, Nov. 1984.
[35] V. Basili and S. Green, "Software Process Evolution at the SEL," IEEE Software, pp. 58-66, 1994.
[36] R.B. Grady, Practical Software Metrics for Project Management and Process Improvement, Prentice Hall, Englewood Cliffs, N.J., 1992.
[37] J. Barnard and A. Price, “Managing Code Inspection Information,” IEEE Software, vol. 11, no. 2, pp. 59–69, Mar. 1994.
[38] V.R. Basili, M. Daskalantonakis, and R. Yacobellis, Technology Transfer at Motorola IEEE Software, pp. 70-76, Mar. 1994.
[39] K. El Eman, N. Moukheiber, and N.H. Madhavji, “An Empirical Evaluation of the GQM Method,” Proc. Center for Advanced Studies Conf. (CASCON 93), pp. 265–289, Nov. 1993.
[40] I.S. Bhandari, M.J. Halliday, J. Chaar, R. Chillarege, K. Jones, J.S. Atkinson, C. Lepori-Costello, P.Y. Jasper, E.D. Tarver, C.C. Lewis, and M. Yonezawa, “In-Process Improvement through Defect Data Interpretation,” IBM Systems J., vol. 33, no. 1, Jan. 1994.
[41] I.S. Bhandari, M.J. Halliday, E. Tarver, D. Brown, J. Chaar, and R. Chillarege, “A Case Study of Software Process Improvement during Development,” IEEE Trans. Software Eng., vol. 19, no. 12, pp. 1,157–1,170, Dec. 1993.
[42] I.S. Bhandari, B. Ray, M.Y. Wong, D. Choi, A. Watanabe, R. Chillarege, M. Halliday, A. Dooley, and J. Chaar, “An Inference Structure for Process Feedback: Technique and Implementation,” Software Quality J., vol. 3, no. 3, pp. 167–189, 1994.
[43] I.S. Bhandari, M.G. Mendonça, and J. Dawson, “On the Use of Machine-Assisted Knowledge Discovery to Analyze and Reengineer Measurement Frameworks,” Proc. of Center for Advanced Studies Conf. (CASCON'95), pp. 275–284, Nov. 1995.
[44] I.S. Bhandari, E. Colet, J. Parker, Z. Pines, R. Pratap, and K. Ramanujam, “Advanced Scout: Data Mining and Knowledge Discovery in the NBA Data,” Data Mining and Knowledge Discovery, vol. 1, no. 1, pp. 121–125, Jan. 1997.
[45] E. Colet and I.S. Bhandari, “Statistical Issues in the Application of Data Mining to the NBA Using Attribute Focusing,” Proc. Am. Statistical Assoc. Section on Statistics in Sports, Joint Statistical Meetings, pp. 1–6, Aug. 1997.
[46] Y.S. Lincoln and E.G. Guba, “Naturalistic Inquiry,” Sage, 1985.
[47] L.C. Briand,V.R. Basili,, and C.J. Hetmanski,“Developing interpretable models with optimized set reduction for identifying high-risk software components,” IEEE Transactions on Software Engineering, vol. 19, no. 11, pp. 1,028-1,044, Nov. 1993.
[48] D. Potier, J.L. Albin, R. Ferreol, A, Bilodeau, "Experiments with Computer Software Complexity and Reliability," Proc. Sixth Int'l Conf. Software Eng., pp. 94-103, 1982.
[49] R.W. Selby and A.A. Porter,“Learning from examples: Generation and evaluation of decision trees for software resource analysis,” IEEE Trans. Software Engineering, vol. 14, no. 12, pp. 1,743-1,757, Dec. 1988.
[50] K. Srinivasan and D. Fisher, “Machine Learning Approaches to Estimating Software Development Effort,” IEEE Trans. Software Eng., vol. 21, no. 2, pp. 126–137, Feb. 1995.
[51] J. Tian and J. Palma, “Analyzing and Improving Reliability: A Tree-Based Approach,” IEEE Software, vol. 15, no. 2, pp. 97–104, Mar. 1998.
[52] I.S. Bhandari, “Attribute Focusing: Machine-Assisted Knowledge Discovery Applied to Software Production Process Control,” Knowledge Acquisition J., vol. 6, no. 3, pp. 271–294, Sept. 1994.
[53] M. Zelkowitz and D. Wallace, “Experimental Models for Validating Technology,” Computer, vol. 31, no. 5, pp. 23–31, May 1998.
[54] B. Kitchenham, L. Pickard, and S. Pfleeger, “Case Studies for Method and Tool Evaluation,” IEEE Software, vol. 11, no. 4, pp. 52–62, July 1995.
[55] M. Ajmone Marshan, A. Bianco, E. Leonardi, and F. Neri, “Topologies for Wavelength-Routing All-Optical Networks,” IEEE/ACM Trans. Networking, vol. 1, pp. 534-546, Oct. 1993.

Index Terms:
Software metrics, goal-oriented measurement, GQM, data mining, knowledge discovery, AF, experimental validation, method evaluation, case study.
Manoel G. Mendonça, Victor R. Basili, "Validation of an Approach for Improving Existing Measurement Frameworks," IEEE Transactions on Software Engineering, vol. 26, no. 6, pp. 484-499, June 2000, doi:10.1109/32.852739
Usage of this product signifies your acceptance of the Terms of Use.