This Article 
 Bibliographic References 
 Add to: 
A Markov Chain Model for Statistical Software Testing
October 1994 (vol. 20 no. 10)
pp. 812-824

Statistical testing of software establishes a basis for statistical inference about a software system's expected field quality. This paper describes a method for statistical testing based on a Markov chain model of software usage. The significance of the Markov chain is twofold. First, it allows test input sequences to be generated from multiple probability distributions, making it more general than many existing techniques. Analytical results associated with Markov chains facilitate informative analysis of the sequences before they are generated, indicating how the test is likely to unfold. Second, the test input sequences generated from the chain and applied to the software are themselves a stochastic model and are used to create a second Markov chain to encapsulate the history of the test, including any observed failure information. The influence of the failures is assessed through analytical computations on this chain. We also derive a stopping criterion for the testing process based on a comparison of the sequence generating properties of the two chains.

[1] K. Agrawal and J. A. Whittaker, "Experiences in applying statistical testing to a real-time embedded software system,"Proc. Pacific Northwest Software Quality Conf., 1993, pp. 154-170.
[2] R. Ash,Information Theory and Coding. New York: McGraw-Hill, 1963.
[3] R. C. Cheung, "A user-oriented software reliability model,"IEEE Trans. Software Eng., vol. SE-6, Mar. 1980.
[4] P. A. Currit, M. Dyer, and H. D. Mills, "Certifying the reliability of software,"IEEE Trans. Software Eng., vol. SE-12, no. 1, pp. 3-11, Jan. 1986.
[5] H. Cramer,The Elements Probability Theory. Huntington, NY: Robert E. Krieger, 1955.
[6] J. L. Doob,Stochastic Processes. New York: Wiley, 1953.
[7] J. W. Duran and S. C. Ntafos, "An evaluation of random testing,"IEEE Trans. Software Eng., vol. SE-10, no. 4, pp. 438-444, July 1984.
[8] J. W. Duran and J. J. Wiorkowski, "Quantifying software validity by sampling,"IEEE Trans. Reliability, vol. R-29, no. 2, pp. 141-144, June 1980.
[9] W. Feller,An Introduction to Probability Theory and Its Applications, vol. 1. New York: Wiley, 1950.
[10] R. Hamlet, "Testing software for software reliability," Tech. Rep. TR- 91-2, rev. 1, Dept. of Comput. Sci., Portland State Univ., Portland, OR, USA, Mar. 1992.
[11] R. Hamlet and R. Taylor, "Partition testing does not inspire confidence,"IEEE Trans. Software Eng., vol. 16, pp. 1402-1411, Dec. 1990.
[12] J.E. Hopcroft and J.D. Ullman,Introduction to Automata Theory, Languages, and Computation, Addison-Wesley, Reading, Mass., 1979.
[13] B. H. Juang and L. R. Rabiner, "A probabilistic distance measure for hidden Markov models,"AT&T Tech. J., vol. 64, no. 2, pp. 391-408, Feb. 1985.
[14] J. G. Kemeny and J. L. Snell,Finite Markov Chains. New York: Springer-Verlag, 1976.
[15] S. Kullback,Information Theory and Statistics. New York: Wiley, 1958.
[16] K. W. Miller, L. J. Morrell, R. E. Noonan, S. K. Park, D. M. Nicol, B. W. Murrill, and J. M. Voas, "Estimating the probability of failure when testing reveals no failures,"IEEE Trans. Software Eng., vol. 18, pp. 33-43, Jan. 1992.
[17] H. D. Mills, "The new math of computer programming,"CACM, vol. 18, no. 1, pp. 43-48, Jan. 1975.
[18] J. D. Musa, "A theory of software reliability and its application,"IEEE Trans. Software Eng., vol. SE-1, pp. 312-321, Aug. 1975.
[19] G. J. Myers,The Art of Software Testing. New York: Wiley, 1979.
[20] D.L. Parnas, A.J. van Shouwen, and S.P. Kwan, "Evaluation of Safety-Critical Software,"Comm. ACM, Vol. 33, No. 6, June 1990, pp. 636-648.
[21] E. Parzen,Stochastic Processes. San Francisco, CA: Holden-Day, 1962.
[22] M. L. Shooman,Software Engineering: Design, Reliability and Management. New York: McGraw-Hill, 1983.
[23] C. E. Shannon, "A mathematical theory of communication,"Bell Syst. Tech. J., vol. 27, pp. 379-423, 623-656, 1948.
[24] K. Siegrist, "Reliability of systems with Markov transfer of control,"IEEE Trans. Software Eng., vol. 14, pp. 1049-1053, July 1988.
[25] M. G. Thomason, "Generating functions for stochastic context-free grammars, "Int. J. Patt. Recognition Art. Intell., vol. 4, pp. 553-572, Apr. 1990.
[26] J. A. Whittaker, "Markov chain techniques for software testing and reliability analysis," Ph.D. dissertation, Dept. of Comput. Sci., Univ. of Tennessee, Knoxville, USA, 1992.
[27] J. A. Whittaker and J. H. Poore, "Markov analysis of software specifications,"ACM Trans. Software Eng. Methodology, vol. 2, pp. 93-106, Jan. 1993.
[28] D. M. Wolt, "Realistic expectations of random testing," CRL Rep. 246, McMaster Univ., Hamilton, ON, Canada, May 1992.

Index Terms:
Markov processes; program testing; software quality; probability; Markov chain model; statistical software testing; statistical inference; software field quality; software usage; test input sequences; multiple probability distributions; stochastic model; software failures; testing process; sequence generating properties
J.A. Whittaker, M.G. Thomason, "A Markov Chain Model for Statistical Software Testing," IEEE Transactions on Software Engineering, vol. 20, no. 10, pp. 812-824, Oct. 1994, doi:10.1109/32.328991
Usage of this product signifies your acceptance of the Terms of Use.