This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage
May-June 2012 (vol. 38 no. 3)
pp. 497-519
ASCII Text
x 
 
Andrea Arcuri, "A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage," IEEE Transactions on Software Engineering, vol. 38, no. 3, pp. 497-519, May-June, 2012.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2011.44,
author = {Andrea Arcuri},
title = {A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage},
journal ={IEEE Transactions on Software Engineering},
volume = {38},
number = {3},
issn = {0098-5589},
year = {2012},
pages = {497-519},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2011.44},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage
IS - 3
SN - 0098-5589
SP497
EP519
EPD - 497-519
A1 - Andrea Arcuri,
PY - 2012
KW - Evolutionary testing
KW - object-oriented software
KW - state problem
KW - search-based software engineering
KW - software testing
KW - length
KW - test sequence.
VL - 38
JA - IEEE Transactions on Software Engineering
ER -
 
Andrea Arcuri, Simula Research Laboratory, Lysaker
In the presence of an internal state, often a sequence of function calls is required to test software. In fact, to cover a particular branch of the code, a sequence of previous function calls might be required to put the internal state in the appropriate configuration. Internal states are not only present in object-oriented software, but also in procedural software (e.g., static variables in C programs). In the literature, there are many techniques to test this type of software. However, to the best of our knowledge, the properties related to the choice of the length of these sequences have received only a little attention in the literature. In this paper, we analyze the role that the length plays in software testing, in particular branch coverage. We show that, on “difficult” software testing benchmarks, longer test sequences make their testing trivial. Hence, we argue that the choice of the length of the test sequences is very important in software testing. Theoretical analyses and empirical studies on widely used benchmarks and on an industrial software are carried out to support our claims.

[1] J.H. Andrews, A. Groce, M. Weston, and R.G. Xu, "Random Test Run Length and Effectiveness," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 19-28, 2008.
[2] A. Arcuri, "Full Theoretical Runtime Analysis of Alternating Variable Method on the Triangle Classification Problem," Proc. Int'l Symp. Search Based Software Eng., pp. 113-121, 2009.
[3] A. Arcuri, "Insight Knowledge in Search Based Software Testing," Proc. Genetic and Evolutionary Computation Conf., pp. 1649-1656, 2009.
[4] A. Arcuri, "Theoretical Analysis of Local Search in Software Testing," Proc. Symp. Stochastic Algorithms, Foundations and Applications, pp. 156-168, 2009.
[5] A. Arcuri, "It Does Matter How You Normalise the Branch Distance in Search Based Software Testing," Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 205-214, 2010.
[6] A. Arcuri, "Longer Is Better: On the Role of Test Sequence Length in Software Testing," Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 469-478, 2010.
[7] A. Arcuri, M.Z. Iqbal, and L. Briand, "Black-Box System Testing of Real-Time Embedded Systems Using Random and Search-Based Testing," Proc. IFIP Int'l Conf. Testing Software and Systems, 2010.
[8] A. Arcuri, M.Z. Iqbal, and L. Briand, "Formal Analysis of the Effectiveness and Predictability of Random Testing," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 219-229, 2010.
[9] A. Arcuri, P. Lehre, and X. Yao, "Theoretical Runtime Analysis in Search Based Software Engineering," Technical Report CSR-09-04, Univ. of Birmingham, 2009.
[10] A. Arcuri, P.K. Lehre, and X. Yao, "Theoretical Runtime Analyses of Search Algorithms on the Test Data Generation for the Triangle Classification Problem," Proc. Int'l Workshop Search-Based Software Testing, pp. 161-169, 2008.
[11] A. Arcuri and X. Yao, "Search Based Software Testing of Object-Oriented Containers," Information Sciences, vol. 178, no. 15, pp. 3075-3095, 2008.
[12] L. Baresi, P.L. Lanzi, and M. Miraz, "Testful: An Evolutionary Test Approach for Java," Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 185-194, 2010.
[13] T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein, Introduction to Algorithms, second ed. MIT Press and McGraw-Hill, 2001.
[14] J.W. Duran and S.C. Ntafos, "An Evaluation of Random Testing," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.
[15] G. Fraser and A. Gargantini, "Experiments on the Test Case Length in Specification Based Test Case Generation," Proc. Int'l Workshop Automation in Software Test, 2009.
[16] M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. Future of Software Eng., pp. 342-357, 2007.
[17] M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, and J. Wegener, "The Impact of Input Domain Reduction on Search-Based Test Data Generation," Proc. European Software Eng. Conf. and the ACM SIGSOFT Symp. Foundations of Software Eng., pp. 155-164, 2007.
[18] M. Harman, S.A. Mansouri, and Y. Zhang, "Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications," Technical Report TR-09-03, King's College, 2009.
[19] M. Harman and P. McMinn, "A Theoretical & Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 73-83, 2007.
[20] M. Harman and P. McMinn, "A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search," IEEE Trans. Software Eng., vol. 36, no. 2, pp. 226-247, Mar./Apr. 2010.
[21] K. Inkumsah and T. Xie, "Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 297-306, 2008.
[22] K. Lakhotia, M. Harman, and P. McMinn, "Handling Dynamic Data Structures in Search Based Testing," Proc. Genetic and Evolutionary Computation Conf., pp. 1759-1766, 2008.
[23] P. Lehre and X. Yao, "Crossover Can Be Constructive When Computing Unique Input Output Sequences," Proc. Int'l Conf. Simulated Evolution and Learning, pp. 595-604, 2008.
[24] P.K. Lehre and X. Yao, "Runtime Analysis of ($1{+}1$ )EA on Computing Unique Input Output Sequences," Proc. IEEE Congress Evolutionary Computation, pp. 1882-1889, 2007.
[25] P.K. Lehre and X. Yao, "Runtime Analysis of Search Heuristics on Software Engineering Problems," Frontiers of Computer Science in China, vol. 3, no. 1, pp. 64-72, 2009.
[26] P.K. Lehre and X. Yao, "Runtime Analysis of the ($1{+}1$ )EA on Computing Unique Input Output Sequences," Information Sciences, pp. 1882-1889, 2010.
[27] A. Leitner, M. Oriol, A. Zeller, I. Ciupa, and B. Meyer, "Efficient Unit Test Case Minimization," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 417-420, 2007.
[28] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[29] M. Motwani and P. Raghavan, Randomized Algorithms. Cambridge Univ. Press, 1995.
[30] G. Myers, The Art of Software Testing. Wiley, 1979.
[31] P.S. Oliveto, J. He, and X. Yao, "Time Complexity of Evolutionary Algorithms for Combinatorial Optimization: A Decade of Results," Int'l J. Automation and Computing, vol. 4, no. 3, pp. 281-293, 2007.
[32] J.C.B. Ribeiro, M.A. Zenha-Rela, and F.F. de Vega, "Test Case Evaluation and Input Domain Reduction Strategies for the Evolutionary Testing of Object-Oriented Software," Information and Software Technology, vol. 51, no. 11, pp. 1534-1548, 2009.
[33] J.C.B. Ribeiro, M.A. Zenha-Rela, and F.F. de Vega, "Enabling Object Reuse on Genetic Programming-Based Approaches to Object-Oriented Evolutionary Testing," Proc. European Conf. Genetic Programming, pp. 220-231, 2010.
[34] P. Tonella, "Evolutionary Testing of Classes," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 119-128, 2004.
[35] W. Visser, C.S. Pasareanu, and S. Khurshid, "Test Input Generation with Java Pathfinder," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 97-107, 2004.
[36] W. Visser, C.S. Pasareanu, and R. Pelànek, "Test Input Generation for Java Containers Using State Matching," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 37-48, 2006.
[37] S. Wappler and J. Wegener, "Evolutionary Unit Testing of Object-Oriented Software Using Strongly-Typed Genetic Programming," Proc. Genetic and Evolutionary Computation Conf., pp. 1925-1932, 2006.
[38] D. Whitley, "The Genitor Algorithm and Selective Pressure: Why Rank-Based Allocation of Reproductive Trials Is Best," Proc. Third Int'l Conf. Genetic Algorithms, pp. 116-121, 1989.
[39] T. Xie, D. Marinov, and D. Notkin, "Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 196-205, 2004.
[40] T. Xie, D. Marinov, W. Schulte, and D. Notkin, "Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution," Proc. 11th Int'l Conf. Tools and Algorithms for the Construction and Analysis of Systems, pp. 365-381, 2005.

Index Terms:
Evolutionary testing, object-oriented software, state problem, search-based software engineering, software testing, length, test sequence.
Citation:
Andrea Arcuri, "A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage," IEEE Transactions on Software Engineering, vol. 38, no. 3, pp. 497-519, May-June 2012, doi:10.1109/TSE.2011.44
Usage of this product signifies your acceptance of the Terms of Use.