This Article 
 Bibliographic References 
 Add to: 
When to Stop Testing for Large Software Systems with Changing Code
April 1994 (vol. 20 no. 4)
pp. 318-323

Developers of large software systems must decide how long software should be tested before releasing it. A common and usually unwarranted assumption is that the code remains frozen during testing. We present a stochastic and economic framework to deal with systems that change as they are tested. The changes can occur because of the delivery of software as it is developed, the way software is tested, the addition of fixes, and so on. Specifically, we report the details of a real time trial of a large software system that had a substantial amount of code added during testing. We describe the methodology, give all of the relevant details, and discuss the results obtained. We pay particular attention to graphical methods that are easy to understand, and that provide effective summaries of the testing process. Some of the plots found useful by the software testers include: the Net Benefit Plot, which gives a running chart of the benefit; the Stopping Plot, which estimates the amount of additional time needed for testing; and diagnostic plots. To encourage other researchers to try out different models, all of the relevant data are provided.

[1] W. S. Cleveland, "Robust locally weighted regression and smoothing scatterplots,"J. Am. Statist. Assoc., vol. 74, pp. 829-836, 1979.
[2] S. R. Dalal and C. L. Mallows, "Buying with exacting confidence,"Annals of Applied Probability, vol. 2, pp. 752-765, Mar. 1992.
[3] S. R. Dalal and C. L. Mallows, "Some graphical aids for deciding when to stop testing software,"IEEE J. Select. Areas Commun., Special Issue on Telecommunications Software Quality and Productivity, vol. 8, pp. 169-175, Feb. 1990.
[4] S. R. Dalal and C. L. Mallows, "When should one stop testing software?"J. Am. Statist. Assoc., vol. 83, pp. 872-879, 1988.
[5] S. R. Dalal and C. L. Mallows, "When to stop testing software? Some exact and asymptotic results," inBayesian Analysis in Statistics and Econometrics, P. Goel, Ed. New York: Springer-Verlag, 1990, pp. 267-276.
[6] S. R. Dalal and A. A. McIntosh, "Reliability modeling and when to stop testing for large software systems in the presence of code churn," Bellcore internal memorandum, 1992.
[7] W. Ehrlich, B. Prasanna, J. Stampfel, and J. Wu, "Determining the cost of a stop-test decision,"IEEE Software, pp. 33-42, 1993.
[8] D. M. Gay, "Algorithm 611: Subroutines for unconstrained minimization using a model/trust-region approach,"ACM Trans. Mathematical Sofware, vol. 9, no. 4, pp. 503-524, 1983.
[9] A.L. Goel and K. Okumoto, "Time-dependent error-detection rate model for software reliability and other performance measures,"IEEE Trans. Reliability, vol. R-28, pp. 206-211, 1979.
[10] Z. Jelinski and P. B. Moranda, "Software reliability research," inStatistical Computer Performance Evaluation, W. Frieberger, Ed. New York: Academic, 1972, pp. 465-484.
[11] W. Kremer, "Birth-death and bug counting,"IEEE Trans. Reliability, vol. R-32, no. 1, pp. 37-47, Jan. 1983.
[12] N. Langberg and N. D. Singpurwalla, "A unification of some software reliability models,"SIAM J. Sci. Stat. Comput., vol. 36, no. 1, pp. 781-790, Jan. 1985.
[13] B. Littlewood, "Stochastic reliability-growth: A model for fault-removal in computer programs and hardware design,"IEEE Trans. Reliability, vol. R-30, no. 4, pp. 313-320, Apr. 1981.
[14] J. D. Musaet al., Software Reliability Measurement, Prediction, Application. New York: McGraw-Hill International, 1987.
[15] J. D. Musa, "A theory of software reliability and its applications,"IEEE Trans. Software Eng., vol. SE-1, pp. 312-327, 1975.
[16] M. C. J. van Pul, "Statistical Analysis of Software Reliability Models," Chapter 7, Ph.D. dissertation, Univ. of Utrecht, 1993.
[17] M. C. J. Van Pul, "Statistical models in software reliability,"Proc. 23rd Symp. Interface, 1991, pp. 106-109.
[18] N. Singpurwalla, "Determining an optimal time interval for testing and debugging software,"IEEE Trans. Sofware Eng., vol. 17, pp. 313-319, Apr. 1991.

Index Terms:
program testing; software metrics; software reliability; computer graphics; large software systems; changing code; economic framework; real time trial; software testers; Net Benefit Plot; Stopping Plot; diagnostic plots; optimal stopping rule; graphical methods; software metrics; statistical inference; software reliability model; software fault detection
S.R. Dalal, A.A. McIntosh, "When to Stop Testing for Large Software Systems with Changing Code," IEEE Transactions on Software Engineering, vol. 20, no. 4, pp. 318-323, April 1994, doi:10.1109/32.277579
Usage of this product signifies your acceptance of the Terms of Use.