Subscribe

Issue No.02 - Feb. (2013 vol.39)

pp: 163-183

A. Shahbazi , Dept. of Electr. & Comput. Eng., Univ. of Alberta, Edmonton, AB, Canada

A. F. Tappenden , Dept. of Comput. Sci., King's Univ. Coll., Edmonton, AB, Canada

J. Miller , Dept. of Electr. & Comput. Eng., Univ. of Alberta, Edmonton, AB, Canada

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TSE.2012.18

ABSTRACT

Although Random Testing (RT) is low cost and straightforward, its effectiveness is not satisfactory. To increase the effectiveness of RT, researchers have developed Adaptive Random Testing (ART) and Quasi-Random Testing (QRT) methods which attempt to maximize the test case coverage of the input domain. This paper proposes the use of Centroidal Voronoi Tessellations (CVT) to address this problem. Accordingly, a test case generation method, namely, Random Border CVT (RBCVT), is proposed which can enhance the previous RT methods to improve their coverage of the input space. The generated test cases by the other methods act as the input to the RBCVT algorithm and the output is an improved set of test cases. Therefore, RBCVT is not an independent method and is considered as an add-on to the previous methods. An extensive simulation study and a mutant-based software testing investigation have been performed to demonstrate the effectiveness of RBCVT against the ART and QRT methods. Results from the experimental frameworks demonstrate that RBCVT outperforms previous methods. In addition, a novel search algorithm has been incorporated into RBCVT reducing the order of computational complexity of the new approach. To further analyze the RBCVT method, randomness analysis was undertaken demonstrating that RBCVT has the same characteristics as ART methods in this regard.

INDEX TERMS

Subspace constraints, Software testing, Generators, Software algorithms, Power capacitors, Runtime, test strategies, Adaptive random testing, centroidal Voronoi tessellation, P-measure, random testing, software testing, test case generation

CITATION

A. Shahbazi, A. F. Tappenden, J. Miller, "Centroidal Voronoi Tessellations- A New Approach to Random Testing",

*IEEE Transactions on Software Engineering*, vol.39, no. 2, pp. 163-183, Feb. 2013, doi:10.1109/TSE.2012.18REFERENCES

- [1] V. Agrawal, "When to Use Random Testing,"
IEEE Trans. Computers, vol. 27, no. 11, pp. 1054-1055, Nov. 1978.- [2] J.W. Duran and S.C. Ntafos, "An Evaluation of Random Testing,"
IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.- [3] P. Loo and W. Tsai, "Random Testing Revisited,"
Information and Software Technology, vol. 30, no. 7, pp. 402-417, 1988.- [4] P. Schneck, "Comment on 'When to Use Random Testing',"
IEEE Trans. Computers, vol. 28, no. 8, pp. 580-581, Aug. 1979.- [5] D.R. Slutz, "Massive Stochastic Testing of SQl,"
Proc. 24th Int'l Conf. Very Large Data Bases, pp. 618-622, 1998.- [6] H. Bati, L. Giakoumakis, S. Herbert, and A. Surna, "A Genetic Approach for Random Testing of Database Systems,"
Proc. 33th Int'l Conf. Very Large Data Bases, pp. 1243-1251, 2007.- [7] T. Yoshikawa, K. Shimura, and T. Ozawa, "Random Program Generator for Java Jit Compiler Test System,"
Proc. Third Int'l Conf. Quality Software, pp. 20-23, 2003.- [8] C. Pacheco, S.K. Lahiri, and T. Ball, "Finding Errors in .Net with Feedback-Directed Random Testing,"
Proc. Int'l Symp. Software Testing and Analysis, pp. 87-96, 2008.- [9] P. Godefroid, A. Kiezun, and M.Y. Levin, "Grammar-Based Whitebox Fuzzing,"
Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 206-215, 2008.- [10] J.E. Forrester and B.P. Miller, "An Empirical Study of the Robustness of Windows NT Applications Using Random Testing,"
Proc. Fourth Conf. USENIX Windows Systems Symp., vol. 4, pp. 6-6, 2000.- [11] B.P. Miller, G. Cooksey, and F. Moore, "An Empirical Study of the Robustness of Macos Applications Using Random Testing,"
Proc. First Int'l Workshop Random Testing, pp. 46-54, 2006.- [12] S. Lipner and M. Howard, "The Trustworthy Computing Security Development Lifecycle Document (SDL)," http://msdn.microsoft. com/en-us/library ms995349.aspx, 2005.
- [13] P. Godefroid, "Random Testing for Security: Blackbox vs. Whitebox Fuzzing,"
Proc. Second Int'l Workshop Random Testing: Co-Located with IEEE/ACM 22nd Int'l Conf. Automated Software Eng., p. 1, 2007.- [14] M. Marre and A. Bertolino, "Using Spanning Sets for Coverage Testing,"
IEEE Trans. Software Eng., vol. 29, no. 11, pp. 974-984, Nov. 2003.- [15] A. Bertolino and M. Marre, "Automatic Generation of Path Covers Based on the Control Flow Analysis of Computer Programs,"
IEEE Trans. Software Eng., vol. 20, no. 12, pp. 885-899, Dec. 1994.- [16] L. White and E. Cohen, "A Domain Strategy for Computer Program Testing,"
IEEE Trans. Software Eng., vol. 6, no. 3, pp. 247-257, May 1980.- [17] P. Ammann and J. Knight, "Data Diversity: An Approach to Software Fault Tolerance,"
IEEE Trans. Computers, vol. 37, no. 4, pp. 418-425, Apr. 1988.- [18] G.B. Finelli, "NASA Software Failure Characterization Experiments,"
Reliability Eng. and System Safety, vol. 32, nos. 1/2, pp. 155-169, 1991.- [19] P. Bishop, "The Variation of Software Survival Time for Different Operational Input Profiles (or Why You Can Wait a Long Time for a Big Bug to Fail),"
23rd Int'l Symp. Fault-Tolerant Computing Digest of Papers, pp. 98-107, june 1993.- [20] C. Schneckenburger and J. Mayer, "Towards the Determination of Typical Failure Patterns,"
Proc. Fourth Int'l Workshop Software Quality Assurance: in Conjunction with the Sixth ESEC/FSE Joint Meeting, pp. 90-93, 2007.- [21] F.T. Chan, T.Y. Chen, I.K. Mak, and Y.T. Yu, "Proportional Sampling Strategy: Guidelines for Software Testing Practitioners,"
Information and Software Technology, vol. 38, no. 12, pp. 775-782, 1996.- [22] C. Jones, "Software Quality in 2011: A Survey of the State of the Art," http://www.asq509.org/ht/a/GetDocumentAction/ id62711, 2011.
- [23] T. Chen, D.H. Huang, T. Tse, and Z. Yang, "An Innovative Approach to Tackling the Boundary Effect in Adaptive Random Testing,"
Proc. 40th Ann. Hawaii Int'l Conf. System Sciences, p. 262a, 2007.- [24] F.-C. Kuo, "An Indepth Study of Mirror Adaptive Random Testing,"
Proc. Ninth Int'l Conf. Quality Software, pp. 51-58, 2009.- [25] T. Chen, H. Leung, and I. Mak, "Adaptive Random Testing,"
Proc. Ninth Asian Computing Science Conf.: Advances in Computer Science, M. Maher, ed., pp. 3156-3157, 2005.- [26] T. Chen and R. Merkel, "Efficient and Effective Random Testing Using the Voronoi Diagram,"
Proc. Australian Software Eng. Conf., pp. 300-308, 2006.- [27] T. Chen, R. Merkel, P. Wong, and G. Eddy, "Adaptive Random Testing through Dynamic Partitioning,"
Proc. Fourth Int'l Conf. Quality Software, pp. 79-86, 2004.- [28] T.Y. Chen, T.H. Tse, and Y.T. Yu, "Proportional Sampling Strategy: A Compendium and Some Insights,"
J. Systems and Software, vol. 58, no. 1, pp. 65-81, 2001.- [29] T. Chen and D. Huang, "Adaptive Random Testing by Localization,"
Proc. 11th Asia-Pacific Software Eng. Conf., pp. 292-298, 2004.- [30] T. Chen, D. Huang, and Z. Zhou, "Adaptive Random Testing Through Iterative Partitioning,"
Proc. 11th Ada-Europe Int'l Conf. Reliable Software Technologies, L. Pinho and M. González Harbour, eds., pp. 155-166, 2006.- [31] J. Mayer, "Lattice-Based Adaptive Random Testing,"
Proc. IEEE/ACM 20th Int'l Conf. Automated Software Eng., pp. 333-336, 2005.- [32] J. Mayer, "Adaptive Random Testing by Bisection and Localization,"
Proc. Fifth Int'l Conf. Formal Approaches to Software Testing, W. Grieskamp and C. Weise, eds., pp. 72-86, 2006.- [33] A. Tappenden and J. Miller, "A Novel Evolutionary Approach for Adaptive Random Testing,"
IEEE Trans. Reliability, vol. 58, no. 4, pp. 619-633, Dec. 2009.- [34] I. Ciupa, A. Leitner, M. Oriol, and B. Meyer, "Artoo: Adaptive Random Testing for Object-Oriented Software,"
Proc. ACM/IEEE 30th Int'l Conf. Software Eng., pp. 71-80, May 2008.- [35] T.Y. Chen and R. Merkel, "Quasi-Random Testing,"
IEEE Trans. Reliability, vol. 56, no. 3, pp. 562-568, Sept. 2007.- [36] H. Chi and E.L. Jones, "Computational Investigations of Quasirandom Sequences in Generating Test Cases for Specification-Based Tests,"
Proc. 38th Conf. Winter Simulation, pp. 975-980, 2006.- [37] K.P. Chan, T.Y. Chen, F.-C. Kuo, and D. Towey, "A Revisit of Adaptive Random Testing by Restriction,"
Proc. 28th Ann. Int'l Computer Software and Applications Conf., vol. 1, pp. 78-85, 2004.- [38] P. LEcuyer and C. Lemieux, "Recent Advances in Randomized Quasi-Monte Carlo Methods,"
Modeling Uncertainty, pp. 419-474, Springer, 2005.- [39] M. Li, X. Chen, X. Li, B. Ma, and P. Vitanyi, "The Similarity Metric,"
IEEE Trans. Information Theory, vol. 50, no. 12, pp. 3250-3264, Dec. 2004.- [40] M. Li and P. Vitanyi,
An Introduction to Kolmogorov Complexity and Its Applications. Springer-Verlag, 2008.- [41] K. Chan, T. Chen, and D. Towey, "Restricted Random Testing,"
Proc. Seventh Int'l Conf. Software Quality, J. Kontio and R. Conradi, eds., pp. 321-330, 2006.- [42] T.Y. Chen, F.-C. Kuo, and H. Liu, "Adaptive Random Testing Based on Distribution Metrics,"
J. Systems and Software, vol. 82, no. 9, pp. 1419-1433, 2009.- [43] J. Mayer and C. Schneckenburger, "An Empirical Analysis and Comparison of Random Testing Techniques,"
Proc. ACM/IEEE Int'l Symp. Empirical Software Eng., pp. 105-114, 2006.- [44] K.P. Chan, T. Chen, and D. Towey, "Forgetting Test Cases,"
Proc. 30th Ann. Int'l Computer Software and Applications Conf., vol. 1, pp. 485-494, Sept. 2006.- [45] I.M. Sobol, "Uniformly Distributed Sequences with Additional Uniformity Properties,"
J. Computational Math. and Math. Physics, vol. 16, pp. 236-242, 1976.- [46] J.H. Halton, "Algorithm 247: Radical-Inverse Quasi-Random Point Sequence,"
Comm. ACM, vol. 7, pp. 701-702, Dec. 1964.- [47] H. Niederreiter, "Low-Discrepancy and Low-Dispersion Sequences,"
J. Number Theory, vol. 30, no. 1, pp. 51-70, 1988.- [48] H. Faure, "Discrépance de Suites Associées à un Systeme de numération (en Dimension Un),"
Bull. Soc. Math. France, vol. 109, no. 2, pp. 143-182, 1981.- [49] P. Peart, "The Dispersion of the Hammersley Sequence in the Unit Square,"
Monatshefte für Mathematik, vol. 94, pp. 249-261, 1982.- [50] C. Schlier, "On Scrambled Halton Sequences,"
Applied Numerical Math., vol. 58, no. 10, pp. 1467-1478, 2008.- [51] B.L. Fox, "Algorithm 647: Implementation and Relative Efficiency of Quasirandom Sequence Generators,"
ACM Trans. Math. Software, vol. 12, pp. 362-376, Dec. 1986.- [52] H. Everett, D. Lazard, S. Lazard, and M. Safey El Din, "The Voronoi Diagram of Three Lines,"
Discrete and Computational Geometry, vol. 42, pp. 94-130, 2009.- [53] Q. Du, V. Faber, and M. Gunzburger, "Centroidal Voronoi Tessellations: Applications and Algorithms,"
SIAM Rev., vol. 41, no. 4, pp. 637-676, 1999.- [54] L. Ju, Q. Du, and M. Gunzburger, "Probabilistic Methods for Centroidal Voronoi Tessellations and Their Parallel Implementations,"
Parallel Computing, vol. 28, no. 10, pp. 1477-1500, 2002.- [55] A. Okabe, B. Boots, K. Sugihara, and S.N. Chiu,
Spatial Tessellations: Concepts and Applications of Voronoi Diagrams, second ed. John Wiley & Sons, Inc., 2008.- [56] N. Beckmann, H.-P. Kriegel, R. Schneider, and B. Seeger, "The $\rm r^{\ast}$ -Tree: An Efficient and Robust Access Method for Points and Rectangles,"
SIGMOD Record, vol. 19, pp. 322-331, May 1990.- [57] A. Arcuri and L. Briand, "Adaptive Random Testing: An Illusion of Effectiveness?"
Proc. Int'l Symp. Software Testing and Analysis, pp. 265-275, 2011.- [58] T.Y. Chen, F.-C. Kuo, and R. Merkel, "On the Statistical Properties of Testing Effectiveness Measures,"
J. Systems and Software, vol. 79, no. 5, pp. 591-601, 2006.- [59] C. Pacheco, S. Lahiri, M. Ernst, and T. Ball, "Feedback-Directed Random Test Generation,"
Proc. 29th Int'l Conf. Software Eng., pp. 75-84, May 2007.- [60] P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing,"
Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 213-223, 2005.- [61] Y.-S. Ma, J. Offutt, and Y.R. Kwon, "Mujava: An Automated Class Mutation System,"
Software Testing, Verification, and Reliability, vol. 15, no. 2, pp. 97-133, 2005.- [62] J. Cohen, "A Power Primer,"
Psychological Bull., vol. 112, no. 1, pp. 155-159, 1992.- [63] J. Cohen,
Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum, 1988.- [64] L.A. Becker, "Effect Size (ES)," no. 1993, http://web.uccs.edu/lbecker/Psy590es.htm , 2000.
- [65] M.J. Meyer, "Martingale Java Stochastic Library," http://martingale.berlios.deMartingale.html , 2012.
- [66] K. MorseJr, "Compression Tools Compared,"
Linux J., vol. 2005, no. 137, pp. 62-66, 2005.- [67] T. Chen, F. Kuo, and Z. Zhou, "On Favourable Conditions for Adaptive Random Testing,"
Int'l J. Software Eng. and Knowledge Eng., vol. 17, no. 6, pp. 805-825, 2007.- [68] D. Salomon,
Data Compression: The Complete Reference, vol. 10. Springer-Verlag, 2007. |