This Article 
 Bibliographic References 
 Add to: 
A Decision-Analytic Stopping Rule for Validation of Commercial Software Systems
September 2000 (vol. 26 no. 9)
pp. 907-918

Abstract—The decision about when to release a software product commercially is not a question of when the software has attained some objectively justifiable degree of correctness. It is, rather, a question of whether the software achieves a reasonable balance among engineering objectives, market demand, customer requirements, and marketing directives of the software organization. In this paper, we present a rigorous framework for addressing this important decision. Conjugate distributions from statistical decision theory provide an attractive means of modeling the cost and rate of bugs given information acquired during software testing, as well as prior information provided by software engineers about the fidelity of the software before testing begins. In contrast to methods such as [1] and [15], the stopping analysis presented here yields a computationally simple rule for deciding when to release a commercial software product based on information revealed to engineers during software testing—complicated numerical procedures are not needed. Our method has the added benefits that it is sequential: It measures explicitly the costs of customer dissatisfaction associated with bugs as well as the costs of declining market position while the testing process continues; and it incorporates a practical framework for cost-criticality assessment that makes sense to professional software developers. A probabilistic model of catastrophic bugs provides another useful way of characterizing and measuring the software's expected performance after commercial release. Taken together, these tools provide a software organization with a clearer basis for making decisions about when to release a commercial software product.

[1] S. Campodónico and N. Singpurwalla, “A Bayesian Analysis of the Logarithmic-Poisson Execution Time Model Based on Expert Opinion and Failure Data,” IEEE Tran. Software Eng., vol. 20, no. 9, 1994.
[2] S. Dalal and C. Mallows, “When Should One Stop Testing Software?” J. American Statistical Assoc., vol. 83, pp. 872–979, 1988.
[3] S. Dalal and A. McIntosh, “When to Stop Testing for Large Software Systems with Changing Code,” IEEE Trans. Software Eng., vol. 20, no. 4, 1994.
[4] M. deGroot, Optimal Statistical Decisions. McGraw-Hill, 1970.
[5] D. Heckerman, “A Tutorial on Learning with Bayesian Networks,” Microsoft Technical Report 95-06, Mar. 1995.
[6] R. Howard and J. Matheson, “Influence Diagrams,” The Principles and Applications of Decision Analysis, Strategic Decisions Group, Menlo Park, Calif., 1983.
[7] R. Howard, “Knowledge Maps,” Management Science, vol. 35, no. 8, 1989.
[8] G.A. Moore, Crossing the Chasm. Harper-Collins, 1991.
[9] M.G. Granger and M. Henrion, Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge Univ. Press, 1990.
[10] J.D. Musa,A. Iannino,, and K. Okumoto,Software Reliability: Measurement, Prediction and Application.New York: McGraw-Hill, 1987.
[11] H. Raiffa and J. Schlaifer, Applied Statistical Decision Theory. MIT Press, 1961.
[12] H. Raiffa, Decision Analysis. Addison-Wesley, 1968.
[13] S. Ross, Stochastic Processes. Wiley, 1983.
[14] S. Ross, “Software Reliability: The Stopping Rule Problem,” IEEE Trans. Software Eng., vol. 11, no. 12, 1985.
[15] D.A. Seaver, D.V. von Winterfeldt, and W. Edwards, “Eliciting Subjective Probability Distributions on Continuous Variables,” Organizational Behavior and Human Performance, vol 21, pp. 379-391, 1978.
[16] N. Singpurwalla, “Determining an Optimal Time Interval for Testing and Debugging Software,” IEEE Trans. Software Eng., vol. 17, no. 4., Apr. 1991.
[17] N. Singpurwalla, “The Failure Rate of Software: Does It Exist?” IEEE Trans. Reliability, vol. 44, no. 3, 1995.
[18] C.S. Spetzler and S.l von Holstein, “Probability Encoding in Decision Analysis,” Management Science, vol. 22, no. 3, 1975.
[19] J. von Neumannand, O. Morgenstern, Theory of Games and Economic Behavior. Princeton, NJ: Princeton Univ. Press, 1947.
[20] R.L. Winkler, “The Assessment of Prior Distributions in Bayesian Analysis,” J. American Statistical Association, vol. 62, 1967.

Index Terms:
Software testing, reliability, decision analysis, Bayesian statistics, commercial software development.
Tom Chávez, "A Decision-Analytic Stopping Rule for Validation of Commercial Software Systems," IEEE Transactions on Software Engineering, vol. 26, no. 9, pp. 907-918, Sept. 2000, doi:10.1109/32.877849
Usage of this product signifies your acceptance of the Terms of Use.