This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Random Testing: Theoretical Results and Practical Implications
March-April 2012 (vol. 38 no. 2)
pp. 258-277
ASCII Text
x 
 
Andrea Arcuri, Muhammad Zohaib Iqbal, Lionel Briand, "Random Testing: Theoretical Results and Practical Implications," IEEE Transactions on Software Engineering, vol. 38, no. 2, pp. 258-277, March-April, 2012.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2011.121,
author = {Andrea Arcuri and Muhammad Zohaib Iqbal and Lionel Briand},
title = {Random Testing: Theoretical Results and Practical Implications},
journal ={IEEE Transactions on Software Engineering},
volume = {38},
number = {2},
issn = {0098-5589},
year = {2012},
pages = {258-277},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2011.121},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - Random Testing: Theoretical Results and Practical Implications
IS - 2
SN - 0098-5589
SP258
EP277
EPD - 258-277
A1 - Andrea Arcuri,
A1 - Muhammad Zohaib Iqbal,
A1 - Lionel Briand,
PY - 2012
KW - Coupon collector
KW - random testing
KW - theory
KW - Schur function
KW - predictability
KW - partition testing
KW - adaptive random testing.
VL - 38
JA - IEEE Transactions on Software Engineering
ER -
 
Andrea Arcuri, Simula, Oslo
Muhammad Zohaib Iqbal, Simula Research Laboratory, Lysaker
Lionel Briand, Simula Research Laboratory, Lysaker
A substantial amount of work has shed light on whether random testing is actually a useful testing technique. Despite its simplicity, several successful real-world applications have been reported in the literature. Although it is not going to solve all possible testing problems, random testing appears to be an essential tool in the hands of software testers. In this paper, we review and analyze the debate about random testing. Its benefits and drawbacks are discussed. Novel results addressing general questions about random testing are also presented, such as how long does random testing need, on average, to achieve testing targets (e.g., coverage), how does it scale, and how likely is it to yield similar results if we rerun it on the same testing problem (predictability). Due to its simplicity that makes the mathematical analysis of random testing tractable, we provide precise and rigorous answers to these questions. Results show that there are practical situations in which random testing is a viable option. Our theorems are backed up by simulations and we show how they can be applied to most types of software and testing criteria. In light of these results, we then assess the validity of empirical analyzes reported in the literature and derive guidelines for both practitioners and scientists.

[1] J.H. Andrews, A. Groce, M. Weston, and R.G. Xu, "Random Test Run Length and Effectiveness," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 19-28, 2008.
[2] A. Arcuri, "Theoretical Analysis of Local Search in Software Testing," Proc. Symp. Stochastic Algorithms, Foundations and Applications, pp. 156-168, 2009.
[3] A. Arcuri, "A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage," IEEE Trans. Software Eng., http://doi.ieeecomputersociety. org/10.1109 TSE.2011.44, 2011.
[4] A. Arcuri and L. Briand, "Adaptive Random Testing: An Illusion of Effectiveness?" Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 265-275, 2011.
[5] A. Arcuri and L. Briand, "Formal Analysis of The Probability of Interaction Fault Detection using Random Testing," IEEE Trans. Software Eng., doi:10.1109/TSE.2011.85, 2011.
[6] A. Arcuri and L. Briand, "A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering," Proc. ACM/IEEE Int'l Conf. Software Eng., pp. 1-10, 2011.
[7] A. Arcuri, M.Z. Iqbal, and L. Briand, "Black-Box System Testing of Real-Time Embedded Systems Using Random and Search-Based Testing," Proc. IFIP Int'l Conf. Testing Software and Systems, pp. 95-110, 2010.
[8] A. Arcuri, M.Z. Iqbal, and L. Briand, "Formal Analysis of the Effectiveness and Predictability of Random Testing," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 219-229, 2010.
[9] P.J. Boland, H. Singh, and B. Cukic, "Comparing Partition and Random Testing via Majorization and Schur Functions," IEEE Trans. Software Eng., vol. 29, no. 1, pp. 88-94, Jan. 2003.
[10] T.Y. Chen, F.C. Kuo, and R. Merkel, "On the Statistical Properties of Testing Effectiveness Measures," J. Systems and Software, vol. 79, no. 5, pp. 591-601, 2006.
[11] T.Y. Chen and Y.T. Yu, "On the Relationship between Partition and Random Testing," IEEE Trans. Software Eng., vol. 20, no. 12, pp. 877-980, Dec. 1994.
[12] T.Y. Chen and Y.T. Yu, "On the Expected Number of Failures Detected by Subdomain Testing and Random Testing," IEEE Trans. Software Eng., vol. 22, no. 2, pp. 109-119, Feb. 1996.
[13] I. Ciupa, A. Pretschner, M. Oriol, A. Leitner, and B. Meyer, "On the Number and Nature of Faults Found by Random Testing," Software Testing, Verification and Reliability, vol. 21, pp. 3-28, 2009.
[14] T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein, Introduction to Algorithms, second ed. MIT Press and McGraw-Hill, 2001.
[15] R.A. DeMillo, R.J. Lipton, and F. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
[16] J.W. Duran and S.C. Ntafos, "An Evaluation of Random Testing," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.
[17] W. Feller, An Introduction to Probability Theory and Its Applications, third ed., vol. 1. Wiley, 1968.
[18] G. Fraser and A. Arcuri, "Evolutionary Generation of Whole Test Suites," Proc. Int'l Conf. Quality Software, pp. 31-40, 2011.
[19] P. Godefroid, N. Klarlund, and K. Sen, "Dart: Directed Automated Random Testing," Proc. ACM Conf. Programming Language Design and Implementation, pp. 213-223, 2005.
[20] A. Groce, G. Holzmann, and R. Joshi, "Randomized Differential Testing as a Prelude to Formal Verification," Proc. ACM/IEEE Int'l Conf. Software Eng., pp. 621-631, 2007.
[21] W.J. Gutjahr, "Optimal Test Distributions for Software Failure Cost Estimation," IEEE Trans. Software Eng., vol. 21, no. 3, pp. 219-228, Mar. 1995.
[22] W.J. Gutjahr, "Partition Testing vs. Random Testing: The Influence of Uncertainty," IEEE Trans. Software Eng., vol. 25, no. 5, pp. 661-674, Sept./Oct. 1999.
[23] D. Hamlet and R. Taylor, "Partition Testing Does Not Inspire Confidence," IEEE Trans. Software Eng., vol. 16, no. 12, pp. 1402-1411, Dec. 1990.
[24] R. Hamlet, "Random Testing," Encyclopedia of Software Eng,. pp. 970-978, Wiley, 1994.
[25] M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. Future of Software Eng., pp. 342-357, 2007.
[26] M. Harman, F. Islam, T. Xie, and S. Wappler, "Automated Test Data Generation for Aspect-Oriented Programs," Proc. Eighth ACM Int'l Conf. Aspect-Oriented Software Development, pp. 185-196, 2009.
[27] M. Harman, S.G. Kim, K. Lakhotia, P. McMinn, and S. Yoo, "Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem," Proc. Int'l Workshop Search-Based Software Testing, 2010.
[28] M. Harman and P. McMinn, "A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search," IEEE Trans. Software Eng., vol. 36, no. 2, pp. 226-247, Mar./Apr. 2010.
[29] L. Holst, "Extreme Value Distributions for Random Coupon Collector and Birthday Problems," Extremes, vol. 4, no. 2, pp. 129-145, 2001.
[30] M.Z. Iqbal, A. Arcuri, and L. Briand, "Environment Modeling with UML/MARTE to Support Black-Box System Testing for Real-Time Embedded Systems: Methodology and Industrial Case Studies," Proc. ACM/IEEE Int'l Conf. Model Driven Eng. Languages and Systems, pp. 286-300, 2010.
[31] D. Knuth, The Art of Computer Programming, vol. 3. Addison Wesley 1973.
[32] M. Lyu, Handbook of Software Reliability Engineering. McGraw-Hill, Inc., 1996.
[33] M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-Dimensionally Equidistributed Uniform Pseudo-Random Number Generator," ACM Trans. Modeling and Computer Simulation, vol. 8, no. 1, pp. 23-30, 1998.
[34] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[35] M. Motwani and P. Raghavan, Randomized Algorithms. Cambridge Univ. Press, 1995.
[36] G. Myers, The Art of Software Testing. Wiley, 1979.
[37] C. Pacheco, S.K. Lahiri, M.D. Ernst, and T. Ball, "Feedback-Directed Random Test Generation," Proc. ACM/IEEE Int'l Conf. Software Eng., pp. 75-84, 2007.
[38] R Development Core Team, R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, 2008.
[39] J.A. Rice, Mathematical Statistics and Data Analysis, second ed. Duxbury Press, 1994.
[40] B. Rosén, "Asymptotic Normality in a Coupon Collector's Problem," Probability Theory and Related Fields, vol. 13, no. 3, pp. 256-279, 1969.
[41] R. Sharma, M. Gligoric, A. Arcuri, G. Fraser, and D. Marinov, "Testing Container Classes: Random or Systematic?" Proc. Fundamental Approaches to Software Eng., 2011.
[42] S. Shioda, "Some Upper and Lower Bounds on the Coupon Collector Problem," J. Computational and Applied Math., vol. 200, no. 1, pp. 154-167, 2007.
[43] P. Thévenod-Fosse and H. Waeselynck, "An Investigation of Statistical Software Testing," Software Testing, Verification and Reliability, vol. 1, no. 2, pp. 15-25, 1991.
[44] N. Tillmann and N.J. de Halleux, "Pex—White Box Test Generation for .NET," Proc. Int'l Conf. Tests And Proofs, pp. 134-253, 2008.
[45] P. Tonella, "Evolutionary Testing of Classes," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 119-128, 2004.
[46] M.Z. Tsoukalas, J.W. Duran, and S.C. Ntafos, "On Some Reliability Estimation Problems in Random and Partition Testing," IEEE Trans. Software Eng., vol. 19, no. 7, pp. 687-697, July 1993.
[47] T. Vos, A. Baars, F. Lindlar, P. Kruse, A. Windisch, and J. Wegener, "Industrial Scaled Automated Structural Testing with the Evolutionary Testing Tool," Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 175-184, 2010.
[48] J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing," Information and Software Technology, vol. 43, no. 14, pp. 841-854, 2001.
[49] E.J. Weyuker and B. Jeng, "Analyzing Partition Testing Strategies," IEEE Trans. Software Eng., vol. 17, no. 7, pp. 703-711, July 1991.
[50] E.J. Weyuker and T.J. Ostrand, "Theories of Program Testing and the Application of Revealing Subdomains," IEEE Trans. Software Eng., vol. 6, no. 3, pp. 236-246, May 1980.
[51] D. White, A. Arcuri, and J. Clark, "Evolutionary Improvement of Programs," IEEE Trans. Evolutionary Computation, vol. 15, no. 4, pp. 515-538, Aug. 2011.
[52] Y. Zhan and J.A. Clark, "A Search-Based Framework for Automatic Testing of Matlab/Simulink Models," J. Systems and Software, vol. 81, no. 2, pp. 262-285, 2008.

Index Terms:
Coupon collector, random testing, theory, Schur function, predictability, partition testing, adaptive random testing.
Citation:
Andrea Arcuri, Muhammad Zohaib Iqbal, Lionel Briand, "Random Testing: Theoretical Results and Practical Implications," IEEE Transactions on Software Engineering, vol. 38, no. 2, pp. 258-277, March-April 2012, doi:10.1109/TSE.2011.121
Usage of this product signifies your acceptance of the Terms of Use.