Subscribe

Issue No.05 - Sept.-Oct. (2012 vol.38)

pp: 1088-1099

Andrea Arcuri , Simula Research Laboratory, Lysaker

Lionel Briand , Simula Research Laboratory, Lysaker

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TSE.2011.85

ABSTRACT

Modern systems are becoming highly configurable to satisfy the varying needs of customers and users. Software product lines are hence becoming a common trend in software development to reduce cost by enabling systematic, large-scale reuse. However, high levels of configurability entail new challenges. Some faults might be revealed only if a particular combination of features is selected in the delivered products. But testing all combinations is usually not feasible in practice, due to their extremely large numbers. Combinatorial testing is a technique to generate smaller test suites for which all combinations of t features are guaranteed to be tested. In this paper, we present several theorems describing the probability of random testing to detect interaction faults and compare the results to combinatorial testing when there are no constraints among the features that can be part of a product. For example, random testing becomes even more effective as the number of features increases and converges toward equal effectiveness with combinatorial testing. Given that combinatorial testing entails significant computational overhead in the presence of hundreds or thousands of features, the results suggest that there are realistic scenarios in which random testing may outperform combinatorial testing in large systems. Furthermore, in common situations where test budgets are constrained and unlike combinatorial testing, random testing can still provide minimum guarantees on the probability of fault detection at any interaction level. However, when constraints are present among features, then random testing can fare arbitrarily worse than combinatorial testing. As a result, in order to have a practical impact, future research should focus on better understanding the decision process to choose between random testing and combinatorial testing, and improve combinatorial testing in the presence of feature constraints.

INDEX TERMS

Software, Context, Fault detection, Feature extraction, Scalability, Benchmark testing, lower bound, Combinatorial testing, random testing, interaction testing, theory, constraint, feature diagram

CITATION

Andrea Arcuri, Lionel Briand, "Formal Analysis of the Probability of Interaction Fault Detection Using Random Testing",

*IEEE Transactions on Software Engineering*, vol.38, no. 5, pp. 1088-1099, Sept.-Oct. 2012, doi:10.1109/TSE.2011.85REFERENCES

- [1] A. Arcuri, M.Z. Iqbal, and L. Briand, "Black-Box System Testing of Real-Time Embedded Systems Using Random and Search-Based Testing,"
Proc. IFIP Int'l Conf. Testing Software and Systems, pp. 95-110, 2010.- [2] A. Arcuri, M.Z. Iqbal, and L. Briand, "Formal Analysis of the Effectiveness and Predictability of Random Testing,"
Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 219-229, 2010.- [3] J. Bach and P. Schroeder, "Pairwise Testing: A Best Practice That Isn't,"
Proc. 22nd Pacific Northwest Software Quality Conf., pp. 180-196, 2004.- [4] A. Calvagna and A. Gargantini, "A Formal Logic Approach to Constrained Combinatorial Testing,"
J. Automated Reasoning, vol. 45, pp. 1-28, 2010.- [5] M. Chateauneuf and D. Kreher, "On the State of Strength-Three Covering Arrays,"
J. Combinatorial Designs, vol. 10, no. 4, pp. 217-238, 2002.- [6] D. Cohen, S. Dalal, J. Parelius, and G. Patton, "The Combinatorial Design Approach to Automatic Test Generation,"
IEEE Software, vol. 13, no. 5, pp. 83-88, Sept. 1996.- [7] M. Cohen, C. Colbourn, and A. Ling, "Augmenting Simulated Annealing to Build Interaction Test Suites,"
Proc. 14th Int'l Symp. Software Reliability Eng., pp. 394-405, 2003.- [8] M. Cohen, M. Dwyer, and J. Shi, "Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach,"
IEEE Trans. Software Eng., vol. 34, no. 5, pp. 633-650, Sept./Oct. 2008.- [9] M. Cohen, P. Gibbons, W. Mugridge, and C. Colbourn, "Constructing Test Suites for Interaction Testing,"
Proc. ACM/IEEE Int'l Conf. Software Eng., pp. 38-48, 2003.- [10] C. Colbourn, "Covering Array Tables," http://www.public. asu.edu/~ccolbou/src/ tabbycatable.html, 28 Sept., 2009.
- [11] C. Colbourn, "Combinatorial Aspects of Covering Arrays,"
Le Matematiche, vol. 58, pp. 121-167, 2004.- [12] T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein,
Introduction to Algorithms, second ed. MIT Press, 2001.- [13] J.W. Duran and S.C. Ntafos, "An Evaluation of Random Testing,"
IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.- [14] M. Ellims, D. Ince, and M. Petre, "The Effectiveness of T-Way Test Data Generation,"
Proc. 27th Int'l Conf. Computer Safety, Reliability, and Security, pp. 16-29, 2008.- [15] W. Feller,
An Introduction to Probability Theory and Its Applications, vol. 1, third ed. Wiley, 1968.- [16] S. Fouché, M. Cohen, and A. Porter, "Incremental Covering Array Failure Characterization in Large Configuration Spaces,"
Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 177-188, 2009.- [17] B. Garvin, M. Cohen, and M. Dwyer, "Evaluating Improvements to a Meta-Heuristic Search for Constrained Interaction Testing,"
J. Empirical Software Eng., vol. 16, pp. 1-42, 2010.- [18] A. Godbole, D. Skipper, and R. Sunley, "T-Covering Arrays: Upper Bounds and Poisson Approximations,"
Combinatorics, Probability and Computing, vol. 5, no. 2, pp. 105-117, 1996.- [19] M. Grindal, J. Offutt, and S. Andler, "Combination Testing Strategies: A Survey,"
Software Testing, Verification and Reliability, vol. 15, no. 3, pp. 167-199, 2005.- [20] H. Hemmati, A. Arcuri, and L. Briand, "Reducing the Cost of Model-Based Testing through Test Case Diversity,"
Proc. IFIP Int'l Conf. Testing Software and Systems, pp. 63-78, 2010.- [21] H. Hemmati, A. Arcuri, and L. Briand, "Empirical Investigation of the Effects of Test Suite Properties On Similarity-Based Test Case Selection,"
Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 327-336, 2011.- [22] D. Kuhn, R. Kacker, and Y. Lei, "Random vs. Combinatorial Methods for Discrete Event Simulation of a Grid Computer Network,"
Proc. Mod Sim World, pp. 14-17, 2009.- [23] D. Kuhn, D. Wallace, and A. GalloJR, "Software Fault Interactions and Implications for Software Testing,"
IEEE Trans. Software Eng., vol. 30, no. 6, pp. 418-421, June 2004.- [24] R. Kuhn, Y. Lei, and R. Kacker, "Practical Combinatorial Testing: Beyond Pairwise,"
IT Professional, vol. 10, pp. 19-23, 2008.- [25] Y. Lei, R. Kacker, D. Kuhn, V. Okun, and J. Lawrence, "IPOG/IPOG-D: Efficient Test Generation for Multi-Way Combinatorial Testing,"
Software Testing, Verification and Reliability, vol. 18, no. 3, pp. 125-148, 2008.- [26] Y. Lei and K. Tai, "In-Parameter-Order: A Test Generation Strategy for Pairwise Testing,"
Proc. High-Assurance Systems Eng. Symp., pp. 254-261, 1998.- [27] J. McGregor, D. Muthig, K. Yoshimura, and P. Jensen, "Guest Editors' Introduction: Successful Software Product Line Practices,"
IEEE Software, vol. 27, no. 3, pp. 16-21, May/June 2010.- [28] D. Mitrinović and P. Vasić,
Analytic Inequalities. Springer, 1970.- [29] M. Motwani and P. Raghavan,
Randomized Algorithms. Cambridge Univ. Press, 1995.- [30] C. Nie and H. Leung, "A Survey of Combinatorial Testing,"
ACM Computing Surveys, vol. 43, no. 2, pp. 1-29, 2011.- [31] G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. Le Traon, "Automated and Scalable T-Wise Test Case Generation Strategies for Software Product Lines,"
Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 459-468, 2010.- [32] A. Pretschner, T. Mouelhi, and Y. Le Traon, "Model-Based Tests for Access Control Policies,"
Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 338-347, 2008.- [33] M. Sinnema and S. Deelstra, "Classifying Variability Modeling Techniques,"
Information and Software Technology, vol. 49, no. 7, pp. 717-739, 2007.- [34] B. Stevens, L. Moura, and E. Mendelsohn, "Lower Bounds for Transversal Covers,"
Designs, Codes and Cryptography, vol. 15, no. 3, pp. 279-299, 1998.- [35] V. Sugumaran, S. Park, and K.C. Kang, "Introduction,"
Comm. ACM, vol. 49, no. 12, pp. 28-32, 2006.- [36] K. Tai and Y. Lei, "A Test Generation Strategy for Pairwise Testing,"
IEEE Trans. Software Eng., vol. 28, no. 1, pp. 109-111, Jan. 2002.- [37] C. Yilmaz, M. Cohen, and A. Porter, "Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces,"
IEEE Trans. Software Eng., vol. 32, no. 1, pp. 20-34, Jan. 2006. |