Subscribe

Issue No.06 - November/December (2010 vol.36)

pp: 763-777

Simon Poulding , University of York, York

John A. Clark , University of York, York

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TSE.2010.24

ABSTRACT

Statistical testing has been shown to be more efficient at detecting faults in software than other methods of dynamic testing such as random and structural testing. Test data are generated by sampling from a probability distribution chosen so that each element of the software's structure is exercised with a high probability. However, deriving a suitable distribution is difficult for all but the simplest of programs. This paper demonstrates that automated search is a practical method of finding near-optimal probability distributions for real-world programs, and that test sets generated from these distributions continue to show superior efficiency in detecting faults in the software.

INDEX TERMS

Software/program verification, testing strategies, test coverage of code, optimization.

CITATION

Simon Poulding, John A. Clark, "Efficient Software Verification: Statistical Testing Using Automated Search",

*IEEE Transactions on Software Engineering*, vol.36, no. 6, pp. 763-777, November/December 2010, doi:10.1109/TSE.2010.24REFERENCES

- [1] P. Thévenod-Fosse and H. Waeselynck, "An Investigation of Statistical Software Testing,"
J. Software Testing, Verification and Reliability, vol. 1, no. 2 pp. 5-26, 1991.- [2] P. Thévenod-Fosse, "Software Validation by Means of Statistical Testing: Retrospect and Future Direction,"
Dependable Computing for Critical Applications, A. Avizienis and J. C. Laprie, eds., pp. 23-50, Springer, 1991.- [3] P. Thévenod-Fosse, H. Waeselynck, and Y. Crouzet, "Software Statistical Testing," Technical Report 95178, Laboratoire d'Analyse et d'Architecture des Systèmes du CNRS (LAAS), 1995.
- [4] P. Thévenod-Fosse and H. Waeselynck, "Statemate Applied to Statistical Testing,"
Proc. Int'l Symp. Software Testing and Analysis, pp. 99-109, 1993.- [5] M. Harman and B.F. Jones, "Search-Based Software Engineering,"
Information and Software Technology, vol. 43, pp. 833-839, 2001.- [6] J. Clark, J.J. Dolado, M. Harman, R. Hierons, B. Jones, M. Lumkin, B. Mitchell, S. Mancoridis, K. Rees, M. Roper, and M. Shepperd, "Reformulating Software Engineering as a Search Problem,"
IEE Proc. Software, vol. 150, no. 3, pp. 161-175, 2003.- [7] Y. Zhang, M. Harman, and S.A. Mansouri, "The Multi-Objective Next Release Problem,"
Proc. Genetic and Evolutionary Computation Conf., pp. 1129-1137, 2007.- [8] E. Alba and F. Chicano, "Software Project Management with GAs,"
Information Sciences, vol. 177, no. 11, pp. 2380-2401, 2007.- [9] P. Emberson and I. Bate, "Minimising Task Migrations and Priority Changes in Mode Transitions,"
Proc. 13th IEEE Real-Time and Embedded Technology and Applications Symp., pp. 158-167, 2007.- [10] M. Harman and L. Tratt, "Pareto Optimal Search Based Refactoring at the Design Level,"
Proc. Genetic and Evolutionary Computation Conf., pp. 1106-1113, 2007.- [11] C. Hao, J.A. Clark, and J.L. Jacob, "Synthesising Efficient and Effective Security Protocols,"
Proc. Workshop Automated Reasoning for Security Protocol Analysis, pp. 25-40, 2004.- [12] P. McMinn, "Search-Based Software Test Data Generation: A Survey,"
Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.- [13] D. White, J.A. Clark, J. Jacob, and S. Poulding, "Evolving Software in the Presence of Resource Constraints,"
Proc. Genetic and Evolutionary Computation Conf., pp. 1775-1782, 2008.- [14] M. Harman, "The Current State and Future of Search Based Software Engineering,"
Proc. 29th Int'l Conf. Software Eng. Future of Software Eng., pp. 342-357, 2007.- [15] J. Duran and S. Ntafos, "An Evaluation of Random Testing,"
IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.- [16] D. Hamlet, "When Only Random Testing Will Do,"
Proc. First Int'l Workshop Random Testing, pp. 1-9, 2006.- [17] H. Zhu, P.A.V. Hall, and J.H.R. May, "Software Unit Test Coverage and Adequacy,"
ACM Computing Surveys, vol. 29, no. 4, pp. 366-427, 1997.- [18] M.J. Harrold, R. Gupta, and M.L. Soffa, "A Methodology for Controlling the Size of a Test Suite,"
ACM Trans. Software Eng. and Methodology, vol. 2, no. 3, pp. 270-285, 1993.- [19] J.A. Jones and M.J. Harrold, "Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage,"
IEEE Trans. Software Eng., vol. 29, no. 3, pp. 195-209, Mar. 2003.- [20] S. Ntafos, "On Random and Partition Testing,"
ACM SIGSOFT Software Eng. Notes, vol. 23, no. 2, pp. 42-48, 1998.- [21]
DO-178B, Software Considerations in Airborne Systems and Equipment Certification, Radio Technical Commission for Aeronautics, 1992.- [22]
Defence Standard 00-55(PART2)/Issue 2, Requirements for Safety Related Software in Defence Equipment, Part 2: Guidance, Ministry of Defence, 1997.- [23] P. Thévenod-Fosse and H. Waeselynck, "Towards a Statistical Approach to Testing Object-Oriented Programs,"
Proc. IEEE Ann. Int'l Symp. Fault Tolerant Computing, pp. 99-108, 1997.- [24] A. Denise, M.-C. Gaudel, and S.-D. Gouraud, "A Generic Method for Statistical Testing," Technical Report 1386, Laboratoire de Recherche en Informatique (LRI), CNRS—Univ. de Paris Sud, Apr. 2004.
- [25] N. Baskiotis, M. Sebag, M.-C. Gaudel, and S. Gouraud, "A Machine Learning Approach for Statistical Software Testing,"
Proc. Int'l Joint Conf. Artificial Intelligence, pp. 2274-2279, 2007.- [26] R.P. Pargas, M.J. Harrold, and R.R. Peck, "Test-Data Generation Using Genetic Algorithms,"
Software Testing, Verification and Reliability, vol. 9, pp. 263-282, 1999.- [27] B. Korel, "Automatic Software Test Data Generation,"
IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.- [28] N. Tracey, J. Clark, and K. Mander, "Automated Program Flaw Finding Using Simulated Annealing,"
ACM SIGSOFT Software Eng. Notes, vol. 23, no. 2, pp. 73-81, 1998.- [29] N.J. Tracey, "A Search-Based Automated Test-Data Generation Framework for Safety-Critical Software," PhD dissertation, Dept. of Computer Science, Univ. of York, 2000.
- [30] J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing,"
Information and Software Technology, vol. 43, pp. 841-854, 2001.- [31] S. Forrest and M. Mitchell, "Relative Building-Block Fitness and the Building Block Hypothesis,"
Foundations of Genetic Algorithms 2, D.L. Whitley, ed., pp. 109-126, Morgan Kaufmann, 1993.- [32] M. Galassi et al.,
GNU Scientific Library Reference Manual, second ed. Network Theory, 2006.- [33] "JDK 1.4 Demo Applets." http://java.sun.com/applets/jdk/1.4index.html , 2010.
- [34] "WCET Analysis Project." http://www.mrtc.mdh.se/projects/wcetbenchmarks.html , 2010.
- [35] M.E. Delamaro and J.C. Maldonado, "Proteum/IM 2.0: An Integrated Mutation Testing Environment,"
Mutation Testing for the New Century, pp. 91-101, Kluwer Academic Publishers, 2001.- [36] T.J. DiCiccio and B. Efron, "Bootstrap Confidence Intervals,"
Statistical Science, vol. 11, no. 3, pp. 189-212, 1996.- [37] N.L. Leech and A.J. Onwuegbuzie, "A Call for Greater Use of Nonparametric Statistics," technical report, US Dept. of Education, Educational Resources Information Center, 2002.
- [38] F. Wilcoxon, "Individual Comparisons by Ranking Methods,"
Biometrics Bull., vol. 1, no. 6, pp. 80-83, 1945.- [39] A. Vargha and H. Delaney, "A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong,"
J. Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101-132, 2000.- [40] E.W. Weisstein, "'Bernoulli Distribution.' From MathWorld—A Wolfram Web Resource." http://mathworld.wolfram.com BernoulliDistribution.html , 2010.
- [41] E.W. Weisstein, "'Geometric Distribution.' From MathWorld—A Wolfram Web Resource." http://mathworld.wolfram.com GeometricDistribution.html , 2010.
- [42] A.E. Eiben, Z. Michalewicz, M. Schoenauer, and J.E. Smith, "Parameter Control in Evolutionary Algorithms,"
Parameter Setting in Evolutionary Algorithms, F.G. Lobo, C.F. Lima, and Z. Michalewicz, eds., vol. 54, pp. 19-46, Springer, 2007.- [43] W.J. Gutjahr and G.C. Pflug, "Simulated Annealing for Noisy Cost Functions,"
J. Global Optimization, vol. 8, no. 1, pp. 1-13, 1996.- [44] S. Rana, D.L. Whitley, and R. Cogswell, "Searching in the Presence of Noise,"
Parallel Problem Solving from Nature, pp. 198-207, Springer, 1996.- [45] H.-G. Beyer, "Evolutionary Algorithms in Noisy Environments: Theoretical Issues and Guidelines for Practice,"
Computer Methods in Applied Mechanics and Eng., vol. 186, pp. 239-267, 2000.- [46] S. Markon, D.V. Arnold, T. Bäck, T. Beielstein, and H.-G. Beyer, "Thresholding—A Selection Operator for Noisy ES,"
Proc. 2001 IEEE Congress on Evolutionary Computation, pp. 465-472, 2001.- [47] S. Yoo and M. Harman, "Pareto Efficient Multi-Objective Test Case Selection,"
Proc. 2007 Int'l Symp. Software Testing and Analysis, pp. 140-150, 2007.- [48] K. Lakhotia, M. Harman, and P. McMinn, "Handling Dynamic Data Structures in Search Based Testing,"
Proc. Genetic and Evolutionary Computation Conf., pp. 1759-1766, 2008. |