This Article 
 Bibliographic References 
 Add to: 
Software Quality Measurement Based on Fault-Detection Data
September 1994 (vol. 20 no. 9)
pp. 665-676

We develop a methodology to measure the quality levels of a number of releases of a software product in its evolution process. The proposed quality measurement plan is based on the faults detected in field operation of the software. We describe how fault discovery data can be analyzed and reported in a framework very similar to that of the QMP (quality measurement plan) proposed by B. Hoadley (1986). The proposed procedure is especially useful in situations where one has only very little data from the latest release. We present details of implementation of solutions to a class of models on the distribution of fault detection times. The conditions under which the families: exponential, Weibull, or Pareto distributions might be appropriate for fault detection times are discussed. In a variety of typical data sets that we investigated one of these families was found to provide a good fit for the data. The proposed methodology is illustrated with an example involving three releases of a software product, where the fault detection times are exponentially distributed. Another example for a situation where the exponential fit is not good enough is also considered.

[1] R. E. Barlow and N. D. Singpurawalla, "Assessing the reliability of computer software and computer networks: An opportunity for partnership with computer scientists,"The American Statistician, vol. 39, pp. 88-94, 1985.
[2] B. Efron, "The Jackknife, the bootstrap and other resampling plans," CBMS-NSF Regional Conf. Series in Applied Mathematics, 1972.
[3] A. L. Goel, "Software reliability models: assumptions, limitations, and applicability,"IEEE Trans. on Software Eng., vol. SE-11, pp. 1411-1423, 1985.
[4] A. L. Goel and K. Okumoto, "Time-dependent error-detection rate model for software reliability and other performance measures,"IEEE Trans. Reliability, vol. R-28, pp. 750-757, 1979.
[5] B. Hoadley, "The Quality Measurement Plan (QMP),"Bell Syst. Tech. J., vol. 60, no. 2, pp. 215-273, 1981.
[6] B. Hoadley, "Quality Measurement Plan (QMP),"Encyclopedia of Statistical Sciences, pp. 393-398, Kotz, Johnson, and Read, Eds. New York: John Wiley, 1986.
[7] H. Joe, "Statistical inference for general-order-statistics and nonhomogeneous-Poisson-process software reliability models,"IEEE Trans. Software Eng., vol. SE-15, pp. 1485-1490, 1989.
[8] K. D. Levin and O. Yadid, "Optimal release time of improved versions of software packages,"Inform. and Software Tech., vol. 32, pp. 65-70, 1990.
[9] B. Littlewood and J. L. Verrall, "A Bayesian reliability growth model for computer software,"J. Roy. Stat. Soc., Series C, vol. 22, pp. 332-346, 1973.
[10] D. R. Miller, "Exponential order statistic models of software reliability growth,"IEEE Trans. Software Eng., vol. SE-12, no. 1, pp. 12-24, Jan. 1986.
[11] R. G. Miller, "An unbalanced jackknife,"The Annals of Statistics, vol. 2, pp. 880-891, 1974.
[12] J. D. Musa, A. Iannino, and K. Okumoto,Software Reliability. New York: McGraw-Hill, 1990.
[13] K. Okumoto, "A statistical method for software quality control,"IEEE Trans. Software Eng., vol. SE-11, pp. 1424-1430, 1985.
[14] B. W. Silverman,Density Estimation for Statistics and Data Analysis. New York: Chapman and Hall, 1986.
[15] J. Tukey, "Bias and confidence in not quite large samples,"Annals of Mathematical Statistics, vol. 29, 1958.
[16] M. Xie,Software Reliability Modeling. Singapore: World Scientific Publisher, 1991.
[17] S. Yamada and S. Osaki, "Software reliability growth modeling: Models and applications,"IEEE Trans. Software Eng., vol. SE-11, pp. 1431-1437, 1985.
[18] "Reliability and quality measurements for telecommunications systems (RQMS)," Bellcore Tech. Ref. TR-TSY-000929, iss. 1, June 1990.

Index Terms:
software quality; software metrics; software reliability; software quality measurement; fault-detection data; quality levels; software product; quality measurement plan; field operation; fault discovery data; QMP; exponential; Weibull; Pareto distribution; data sets
S. Weerahandi, R.E. Hausman, "Software Quality Measurement Based on Fault-Detection Data," IEEE Transactions on Software Engineering, vol. 20, no. 9, pp. 665-676, Sept. 1994, doi:10.1109/32.317425
Usage of this product signifies your acceptance of the Terms of Use.