Subscribe
Issue No.05 - May (2013 vol.39)
pp: 684-706
Cemal Yilmaz , Sabanci University, Istanbul
ABSTRACT
The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations by using a battery of test cases. Traditional covering arrays, while taking system-wide interoption constraints into account, do not provide a systematic way of handling test case-specific interoption constraints. The basic justification for $(t)$-way covering arrays is that they can cost effectively exercise all system behaviors caused by the settings of $(t)$ or fewer options. In this paper, we hypothesize, however, that in the presence of test case-specific interoption constraints, many such behaviors may not be tested due to masking effects caused by the overlooked test case-specific constraints. For example, if a test case refuses to run in a configuration due to an unsatisfied test case-specific constraint, none of the valid option setting combinations appearing in the configuration will be tested by that test case. To account for test case-specific constraints, we introduce a new combinatorial object, called a test case-aware covering array. A $(t)$-way test case-aware covering array is not just a set of configurations, as is the case in traditional covering arrays, but a set of configurations, each of which is associated with a set of test cases such that all test case-specific constraints are satisfied and that, for each test case, each valid combination of option settings for every combination of $(t)$ options appears at least once in the set of configurations that the test case is associated with. We furthermore present three algorithms to compute test case-aware covering arrays. Two of the algorithms aim to minimize the number of configurations required (one is fast, but produces larger arrays, the other is slower, but produces smaller arrays), whereas the remaining algorithm aims to minimize the number of test runs required. The results of our empirical studies conducted on two widely used highly configurable software systems suggest that test case-specific constraints do exist in practice, that traditional covering arrays suffer from masking effects caused by ignorance of such constraints, and that test case-aware covering arrays are better than other approaches in handling test case-specific constraints, thus avoiding masking effects.
INDEX TERMS
Testing, Servers, Software algorithms, Software systems, Systematics, Computational modeling, covering arrays, Software quality assurance, combinatorial interaction testing
CITATION
Cemal Yilmaz, "Test Case-Aware Combinatorial Interaction Testing", IEEE Transactions on Software Engineering, vol.39, no. 5, pp. 684-706, May 2013, doi:10.1109/TSE.2012.65
REFERENCES
 [1] Advanced Combinatorial Testing System (ACTS), http://csrc.nist.gov/groups/SNS/acts/documents comparison-report. html, 2012. [2] C. Baral, Knowledge Representation, Reasoning, and Declarative Problem Solving. Cambridge Univ. Press, 2003. [3] R. Brownlie, J. Prowse, and M.S. Phadke, "Robust Testing of AT&T PMX/StarMAIL Using OATS," AT&T Technical J., vol. 71, no. 3, pp. 41-47, 1992. [4] R.C. Bryce and C.J. Colbourn, "Constructing Interaction Test Suites with Greedy Algorithms," Proc. 20thIEEE/ACM Int'l Conf. Automated Software Eng., pp. 440-443, 2005. [5] R.C. Bryce and C.J. Colbourn, "Prioritized Interaction Testing for Pair-Wise Coverage with Seeding and Constraints," Information and Software Technology, vol. 48, no. 10, pp. 960-970, 2006. [6] R.C. Bryce and C.J. Colbourn, "The Density Algorithm for Pairwise Interaction Testing: Research Articles," Software Testing and Verification Reliability, vol. 17, pp. 159-182, Sept. 2007. [7] R.C. Bryce and C.J. Colbourn, "One-Test-at-a-Time Heuristic Search for Interaction Test Suites," Proc. Ninth Ann. Conf. Genetic and Evolutionary Computation, pp. 1082-1089, 2007. [8] R.C. Bryce and C.J. Colbourn, "A Density-Based Greedy Algorithm for Higher Strength Covering Arrays," Software Testing and Verification Reliability, vol. 19, pp. 37-53, Mar. 2009. [9] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, "The AETG System: An Approach to Testing Based on Combinatorial Design," IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437-44, July 1997. [10] M.B. Cohen, C.J. Colbourn, and A.C.H. Ling, "Augmenting Simulated Annealing to Build Interaction Test Suites," Proc. 14th Int'l Symp. Software Reliability Eng., p. 394, 2003. [11] M.B. Cohen, M.B. Dwyer, and J. Shi, "Interaction Testing of Highly Configurable Systems in the Presence of Constraints," Proc. Int'l Symp. Software Testing and Analysis, pp. 129-139, 2007. [12] M.B. Cohen, P.B. Gibbons, W.B. Mugridge, and C.J. Colbourn, "Constructing Test Suites for Interaction Testing," Proc. 25th Int'l Conf. Software Eng., pp. 38-48, 2003. [13] J. Czerwonka, "Pairwise Testing in the Real World: Practical Extensions to Test-Case Scenarios," Proc. 24th Pacific Northwest Software Quality Conf., pp. 285-294, 2006. [14] S.R. Dalal, A. Jain, N. Karunanithi, J.M. Leaton, C.M. Lott, G.C. Patton, and B.M. Horowitz, "Model-Based Testing in Practice," Proc. Int'l Conf. Software Eng., pp. 285-294, 1999. [15] E. Dumlu, C. Yilmaz, M.B. Cohen, and A. Porter, "Feedback Driven Adaptive Combinatorial Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 243-253, 2011. [16] T. Eiter, G. Ianni, and T. Krennwallner, "Answer Set Programming: A Primer," Reasoning Web. Semantic Technologies for Information Systems, pp. 40-110, Springer, 2009. [17] E. Ellguth, M. Gebser, M. Gusowski, B. Kaufmann, R. Kaminski, S. Liske, T. Schaub, L. Schneidenbach, and B. Schnor, "A Simple Distributed Conflict-Driven Answer Set Solver," Proc. 10th Int'l Conf. Logic Programming and Nonmonotonic Reasoning, pp. 490-495, 2009. [18] S. Fouché, M.B. Cohen, and A. Porter, "Towards Incremental Adaptive Covering Arrays," The Sixth Joint Meeting on European Software Eng. Conf. and the ACM SIGSOFT Symp. Foundations of Software Eng.: Companion Papers, pp. 557-560, 2007. [19] S. Ghazi and M. Ahmed, "Pair-Wise Test Coverage Using Genetic Algorithms," Proc. Congress Evolutionary Computation, vol. 2, pp. 1420-1424, Dec. 2003. [20] A. Hartman, "Software and Hardware Testing Using Combinatorial Covering Suites," Graph Theory, Combinatorics and Algorithms, Operations Research/Computer Science Interfaces Series, M.C. Golumbic and I.B.-A. Hartman, eds., vol. 34, pp. 237-266, Springer, 2005. [21] N. Kobayashi, "Design and Evaluation of Automatic Test Generation Strategies for Functional Testing of Software," PhD thesis, Osaka Univ., 2002. [22] D. Kuhn, D.R. Wallace, and A.M. Gallo, "Software Fault Interactions and Implications for Software Testing," IEEE Trans. Soft. Eng., vol. 30, no. 6, pp. 418-421, June 2004. [23] Y. Lei, R. Kacker, D.R. Kuhn, V. Okun, and J. Lawrence, "Ipog-ipog-d: Efficient Test Generation for Multiway Combinatorial Testing," Software Testing Verification and Reliability, vol. 18, pp. 125-148, Sept. 2008. [24] V. Marek and M. Truszczyński, "Stable Models and an Alternative Logic Programming Paradigm," The Logic Programming Paradigm: A 25-Year Perspective, 1999. [25] G. Mats, O. Jeff, and M. Jonas, "Handling Constraints in the Input Space When Using Combination Strategies for Software Testing," Technical Report HS-IKI-TR-06-001, School of Humanities and Informatics, Univ. of Skvde, 2006. [26] C. Nie and H. Leung, "A Survey of Combinatorial Testing," ACM Computing Surveys, vol. 43, pp. 11:1-11:29, Feb. 2011. [27] I. Niemelä, "Logic Programs with Stable Model Semantics as a Constraint Programming Paradigm," Ann. Math. and Artificial Intelligence, vol. 25, nos. 3/4, pp. 241-273, 1999. [28] P.J. Schroeder, P. Bolaki, and V. Gopu, "Comparing the Fault Detection Effectiveness of N-Way and Random Test Suites," Proc. Int'l Symp. Empirical Software Eng., pp. 49-59, 2004. [29] T. Shiba, T. Tsuchiya, and T. Kikuno, "Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing," Proc. Ann. Int'l Computer Software and Applications Conf., pp. 72-77, 2004. [30] K.-C. Tai and Y. Lei, "A Test Generation Strategy for Pairwise Testing," IEEE Trans. Software Eng., vol. 28, no. 1, pp. 109-111, Jan. 2002. [31] Y.-W. Tung and W. Aldiwan, "Automating Test Case Generation for the New Generation Mission Software System," Proc. IEEE Aerospace Conf., vol. 1, pp. 431-437, 2000. [32] A.W. Williams, "Determination of Test Configurations for Pair-Wise Interaction Coverage," Proc. 13th IFIP TC6/WG6.1 Int'l Conf. Testing Communicating Systems: Tools and Techniques, pp. 59-74, 2000. [33] A.W. Williams and R.L. Probert, "Formulation of the Interaction Test Coverage Problem as an Integer Program," Proc. 14th IFIP Int'l Conf. Testing Communicating Systems, p. 283, 2002. [34] C. Yilmaz, M.B. Cohen, and A. Porter, "Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces," IEEE Trans. Software Eng., vol. 31, no. 1, pp. 20-34, Jan. 2006.