The Community for Technology Leaders
RSS Icon
Issue No.01 - January/February (2011 vol.37)
pp: 80-94
James H. Andrews , University of Western Ontario, London, Ont., Canada
Tim Menzies , West Virginia University, Morgantown, WV, USA
Felix C.H. Li , University of Western Ontario, London, Ont., Canada
Randomized testing is an effective method for testing software units. The thoroughness of randomized unit testing varies widely according to the settings of certain parameters, such as the relative frequencies with which methods are called. In this paper, we describe Nighthawk, a system which uses a genetic algorithm (GA) to find parameters for randomized unit testing that optimize test coverage. Designing GAs is somewhat of a black art. We therefore use a feature subset selection (FSS) tool to assess the size and content of the representations within the GA. Using that tool, we can reduce the size of the representation substantially while still achieving most of the coverage found using the full representation. Our reduced GA achieves almost the same results as the full system, but in only 10 percent of the time. These results suggest that FSS could significantly optimize metaheuristic search-based software engineering tools.
Software testing, randomized testing, genetic algorithms, feature subset selection, search-based optimization, testing tools.
James H. Andrews, Tim Menzies, Felix C.H. Li, "Genetic Algorithms for Randomized Unit Testing", IEEE Transactions on Software Engineering, vol.37, no. 1, pp. 80-94, January/February 2011, doi:10.1109/TSE.2010.46
[1] B.P. Miller, L. Fredriksen, and B. So, "An Empirical Study of the Reliability of UNIX Utilities," Comm. ACM, vol. 33, no. 12, pp. 32-44, Dec. 1990.
[2] J.H. Andrews, S. Haldar, Y. Lei, and C.H.F. Li, "Tool Support for Randomized Unit Testing," Proc. First Int'l Workshop Randomized Testing, pp. 36-45, July 2006.
[3] C. Pacheco, S.K. Lahiri, M.D. Ernst, and T. Ball, "Feedback-Directed Random Test Generation," Proc. 29th Int'l Conf. Software Eng., pp. 75-84, May 2007.
[4] A. Groce, G.J. Holzmann, and R. Joshi, "Randomized Differential Testing as a Prelude to Formal Verification," Proc. 29th Int'l Conf. Software Eng., pp. 621-631, May 2007.
[5] C.C. Michael, G. McGraw, and M.A. Schatz, "Generating Software Test Data by Evolution," IEEE Trans. Software Eng., vol. 27, no. 12, pp. 1085-1110, Dec. 2001.
[6] W. Visser, C.S. Păsăreanu, and R. Pelánek, "Test Input Generation for Java Containers Using State Matching," Proc. Int'l Symp. Software Testing and Analysis, pp. 37-48, July 2006.
[7] R.-K. Doong and P.G. Frankl, "The ASTOOT Approach to Testing Object-Oriented Programs," ACM Trans. Software Eng. and Methodology, vol. 3, no. 2, pp. 101-130, Apr. 1994.
[8] S. Antoy and R.G. Hamlet, "Automatically Checking an Implementation against its Formal Specification," IEEE Trans. Software Eng., vol. 26, no. 1, pp. 55-69, Jan. 2000.
[9] K. Claessen and J. Hughes, "QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs," Proc. Fifth ACM SIGPLAN Int'l Conf. Functional Programming, pp. 268-279, Sept. 2000.
[10] K. Sen, D. Marinov, and G. Agha, "CUTE: A Concolic Unit Testing Engine for C," Proc. 13th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 263-272, Sept. 2005.
[11] J. Andrews, F. Li, and T. Menzies, "Nighthawk: A Two-Level Genetic-Random Unit Test Data Generator," Proc. 22nd IEEE/ACM Int'l Conf. Automated Software Eng., , 2007.
[12] J. Andrews and T. Menzies, "On the Value of Combining Feature Subset Selection with Genetic Algorithms: Faster Learning of Coverage Models," Proc. Fifth Int'l Conf. Predictor Models in Software Eng.,, 2009.
[13] Program Test Methods, W.C. Hetzel, ed., Prentice-Hall, 1973.
[14] R. Hamlet, "Random Testing," Encyclopedia of Software Eng., Wiley, pp. 970-978, 1994.
[15] E.J. Weyuker, "On Testing Non-Testable Programs," The Computer J., vol. 25, no. 4, pp. 465-470, Nov. 1982.
[16] C. Csallner and Y. Smaragdakis, "JCrasher: An Automatic Robustness Tester for Java," Software Practice and Experience, vol. 34, no. 11, pp. 1025-1050, 2004.
[17] I. Ciupa, A. Leitner, M. Oriol, and B. Meyer, "Artoo: Adaptive Random Testing for Object-Oriented Software," Proc. 30th ACM/IEEE Int'l Conf. Software Eng., pp. 71-80, May 2008.
[18] M.D. Ernst, J. Cockrell, W.G. Griswold, and D. Notkin, "Dynamically Discovering Likely Program Invariants to Support Program Evolution," IEEE Trans. Software Eng., vol. 27, no. 2, pp. 99-123, Feb. 2001.
[19] T. Ball, "A Theory of Predicate-Complete Test Coverage and Generation," Proc. Third Int'l Symp. Formal Methods for Components and Objects, pp. 1-22, Nov. 2004.
[20] J.H. Andrews and Y. Zhang, "General Test Result Checking with Log File Analysis," IEEE Trans. Software Eng., vol. 29, no. 7, pp. 634-648, July 2003.
[21] L.A. Clarke, "A System to Generate Test Data and Symbolically Execute Programs," IEEE Trans. Software Eng., vol. 2, no. 3, pp. 215-222, Sept. 1976.
[22] J.C. King, "Symbolic Execution and Program Testing," Comm. ACM, vol. 19, no. 7, pp. 385-394, 1976.
[23] B. Korel, "Automated Software Test Generation," IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.
[24] N. Gupta, A.P. Mathur, and M.L. Soffa, "Automated Test Data Generation Using an Iterative Relaxation Method," Proc. Sixth Int'l Symp. Foundations of Software Eng., pp. 224-232, Nov. 1998.
[25] W.K. Leow, S.C. Khoo, and Y. Sun, "Automated Generation of Test Programs from Closed Specifications of Classes and Test Cases," Proc. 26th Int'l Conf. Software Eng., pp. 96-105, May 2004.
[26] W. Visser, C.S. Păsăreanu, and S. Khurshid, "Test Input Generation with Java PathFinder," Proc. ACM/SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 97-107, July 2004.
[27] P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 213-223, June 2005.
[28] D. Owen and T. Menzies, "Lurch: A Lightweight Alternative to Model Checking," Proc. 15th Int'l Conf. Software Eng. and Knowledge Eng., pp. 158-165, July 2003.
[29] J.H. Holland, Adaptation in Natural and Artificial Systems. Univ. of Michigan Press, 1975.
[30] D.E. Goldberg, Genetic Algorithm in Search, Optimization, and Machine Learning. Addison-Wesley, 1989.
[31] L. Rela, "Evolutionary Computing in Search-Based Software Engineering," master's thesis, Lappeenranta Univ. of Tech nology, 2004.
[32] R.P. Pargas, M.J. Harrold, and R.R. Peck, "Test-Data Generation Using Genetic Algorithms," J. Software Testing, Verification and Reliability, vol. 9, pp. 263-282, Dec. 1999.
[33] Q. Guo, R.M. Hierons, M. Harman, and K. Derderian, "Computing Unique Input/Output Sequences Using Genetic Algorithms," Proc. Third Int'l Workshop Formal Approaches to Testing of Software, pp. 164-177, 2004.
[34] P. Tonella, "Evolutionary Testing of Classes," Proc. ACM/SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 119-128, July 2004.
[35] A. Watkins and E.M. Hufnagel, "Evolutionary Test Data Generation: A Comparison of Fitness Functions," Software Practice and Experience, vol. 36, pp. 95-116, Jan. 2006.
[36] P.G. Frankl and S.N. Weiss, "An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing," IEEE Trans. Software Eng., vol. 19, no. 8, pp. 774-787, Aug. 1993.
[37] A. Siami Namin and J. Andrews, "The Influence of Size and Coverage on Test Suite Effectiveness," Proc. Int'l Symp. Software Testing and Analysis, pp. 57-68, 2009.
[38] G. Rothermel, M.J. Harrold, J. Ostrin, and C. Hong, "An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites," Proc. Int'l Conf. Software Maintenance, pp. 34-43, Nov. 1998.
[39] D.R. Cok, "Adapting JML to Generic Types and Java 1.6," Proc. Seventh Int'l Workshop Specification and Verification of Component-Based Systems, pp. 27-34, Nov. 2008.
[40] K. Havelund and T. Pressburger, "Model Checking Java Programs Using Java PathFinder," Int'l J. Software Tools for Technology Transfer, vol. 2, no. 4, pp. 366-381, 2000.
[41] K.A. DeJong and W.M. Spears, "An Analysis of the Interacting Roles of Population Size and Crossover in Genetic Algorithms," Proc. First Workshop Parallel Problem Solving from Nature, pp. 38-47, 1990.
[42] Cobertura Development Team "Cobertura Web Site," cobertura., Feb. 2007.
[43] L. Burdy, Y. Cheon, D.R. Cok, M.D. Ernst, J.R. Kiniry, and G.T. Leavens, "An Overview of JML Tools and Applications," Int'l J. Software Tools for Technology Transfer, vol. 7, no. 3, pp. 212-232, June 2005.
[44] M. Hall and G. Holmes, "Benchmarking Attribute Selection Techniques for Discrete Class Data Mining," IEEE Trans. Knowledge and Data Eng., vol. 15, no. 6, pp. 1437-1447, Nov./Dec. 2003.
[45] A. Miller, Subset Selection in Regression, second ed., Chapman & Hall, 2002.
[46] K. Kira and L. Rendell, "A Practical Approach to Feature Selection," Proc. Ninth Int'l Conf. Machine Learning, pp. 249-256, 1992.
[47] I. Kononenko, "Estimating Attributes: Analysis and Extensions of Relief," Proc. Seventh European Conf. Machine Learning. pp. 171-182, 1994.
[48] S. Cornett, "Minimum Acceptable Code Coverage," http://www.bullseye.comminimum.html, 2006.
[49] S. Berner, R. Weber, and R.K. Keller, "Enhancing Software Testing by Judicious use of Code Coverage Information," Proc. 29th Int'l Conf. Software Eng., pp. 612-620, May 2007.
23 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool