This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
On Comparisons of Random, Partition, and Proportional Partition Testing
October 2001 (vol. 27 no. 10)
pp. 949-960

Abstract—Early studies of random versus partition testing used the probability of detecting at least one failure as a measure of test effectiveness and indicated that partition testing is not significantly more effective than random testing. More recent studies have focused on proportional partition testing because a proportional allocation of the test cases (according to the probabilities of the subdomains) can guarantee that partition testing will perform at least as well as random testing. In this paper, we show that this goal for partition testing is not a worthwhile one. Guaranteeing that partition testing has at least as high a probability of detecting a failure comes at the expense of decreasing its relative advantage over random testing. We then discuss other problems with previous studies and show that failure to include important factors (cost, relative effectiveness) can lead to misleading results.

[1] B. Beizer, Software Testing Techniques, second ed. Boston, Mass.: Int'l Thomson Computer Press, 1990.
[2] F. Chan, T. Chen, I. Mak, and Y. Yu, “Proportional Sampling Strategy: Guidelines for Software Test Practitioners,” Information and Software Technology, vol. 38, no. 12, pp. 775-782, 1996.
[3] T. Chen and Y. Yu, “On the Relationship Between Partition and Random Testing,” IEEE Trans. Software Eng., vol. 20, no. 12, pp. 977-980, Dec. 1994.
[4] T. Chen and Y. Yu, “On the Expected Number of Failures Detected by Subdomain Testing and Random Testing,” IEEE Trans. Software Eng., vol. 22, no. 2, pp. 109-119, Feb. 1996.
[5] T. Chen and Y. Yu, “A More General Sufficient Condition for Partition Testing to be Better than Random Testing,” Information Processing Letters, vol. 57, no. 3, pp. 145-149, 1996.
[6] T. Chen and Y. Yu, “Constraints for Safe Partition Testing Strategies,” Computer J., vol. 39, no. 7, pp. 619-625, 1996.
[7] R. Cobb and H. Mills, “Engineering Software Under Statistical Quality Control,” IEEE Software, pp. 44-54, Nov. 1990.
[8] W. Cochran, Sampling Techniques. Wiley, 1977.
[9] R. DeMillo, R.R. Lipton, and F. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” IEEE Computer, vol. 11, no. 4, pp. 34-41, 1978.
[10] J. Duran and S. Ntafos, “An Evaluation of Random Testing,” IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.
[11] P. Frankl, D. Hamlet, B. Littlewood, and L. Strigini, “Choosing a Testing Method to Deliver Reliability,” Proc. 19th Int'l Conf. Software Eng., pp. 68-78, May 1997.
[12] P. Frankl and O. Iakounenko, “Further Empirical Studies of Test Effectiveness,” Proc. ACM SIGSOFT Sixth Symp. Foundations of Software Eng., SIGSOFT Software Eng. Notes, vol. 23, no. 6, pp. 153-162, Nov. 1998.
[13] P. Frankl, S. Weiss, and C. Hu, “All-Uses versus Mutation Testing: An Experimental Comparison of Effectiveness,” J. Systems and Software, vol. 38, no. 3, pp. 235-253, Sept. 1997.
[14] W. Gutjahr, “Optimal Test Distributions for Software Failure Cost Estimation,” IEEE Trans. Software Eng., vol. 21, no. 3, pp. 219-228, Mar. 1995.
[15] D. Hamlet and R. Taylor, "Partition Testing Does Not Inspire Confidence," IEEE Trans. Software Eng., vol. 16, pp. 1,402-1,412, Dec. 1990.
[16] H. Hecht, “Rare Conditions—An Important Cause of Failures,” Proc. Ann. Conf. Computer Assurance, (COMPASS-93), pp. 81-85, 1993.
[17] W. Howden, “A Functional Approach to Program Testing and Analysis,” IEEE Trans. Software Eng., vol. 12, pp. 997-1005, 1986.
[18] B. Jeng, “Toward an Integration of Data Flow and Domain Testing,” J. Systems and Software, vol. 45, no. 1, pp. 19-30, Feb. 1999.
[19] B. Mitchell and S. Zeil, “An Experiment in Estimating Reliability Growth Under Both Representative and Directed Testing,” Proc. ISSTA-98 ACM Software Eng. Notes, vol. 23, no. 2, pp. 32-41, Mar. 1998.
[20] G.J. Myers,The Art of Software Testing.New York: Wiley, 1979.
[21] S. Ntafos, “The Cost of Software Failures,” Proc. Int'l Assoc. of Science and Technology for Development, Int'l Conf. Software Eng., pp. 53-57, Nov. 1997.
[22] A. Pudgurski and C. Yang, “Partition Testing, Stratified Sampling and Cluster Analysis,” Proc. SIGSOFT Symp. Foundations of Software Eng., pp. 169-181, 1993.
[23] D.S. Rosenblum and E.J. Weyuker, “Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies,” IEEE Trans. Software Eng., vol. 23, no. 3, pp. 146-156, Mar. 1997.
[24] S. Sherer, “Measuring Software Failure Risk: Methodology and an Example,” J. Systems and Software, vol. 25, pp. 257-269, 1994.
[25] S. Sherer, “A Cost-Effective Approach to Testing,” IEEE Software, pp. 34-40, 1991.
[26] M. Tsoukalas, M.J. Duran, and S. Ntafos, “On Some Reliability Estimation Problems in Random and Partition Testing,” IEEE Trans. Software Eng., vol. 19, no. 7, pp. 687-697, July 1993.
[27] E. Weyuker, “Using Failure Cost Information for Testing and Reliability Assessment,” ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 87-98, Apr. 1996.
[28] E.J. Weyuker and B. Jeng,“Analyzing partition testing strategies,” IEEE Trans. Software Engineering, vol. 17, pp. 703-711, 1991.
[29] E. Weyuker and T. Ostrand, “Theories of Program Testing and the Application of Revealing Subdomains,” IEEE Trans. Software Eng., vol. 6, pp. 236-246, 1980.
[30] L. White and E. Cohen, “A Domain Strategy for Computer Program Testing,” IEEE Trans. Software Eng., vol. 6, pp. 247-257, 1980.

Index Terms:
Program testing, random testing, partition testing, proportional partition testing.
Citation:
Simeon C. Ntafos, "On Comparisons of Random, Partition, and Proportional Partition Testing," IEEE Transactions on Software Engineering, vol. 27, no. 10, pp. 949-960, Oct. 2001, doi:10.1109/32.962563
Usage of this product signifies your acceptance of the Terms of Use.