This Article 
 Bibliographic References 
 Add to: 
Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach
Sept.-Oct. 2008 (vol. 34 no. 5)
pp. 633-650
Myra B. Cohen, University of Nebraska-Lincoln, Lincoln
Matthew B. Dwyer, University of Nebraska-Lincoln, Lincoln
Jiangfan Shi, University of Nebraska-Lincoln, Lincoln
Researchers have explored the application of combinatorial interaction testing (CIT) methods to construct samples to drive systematic testing of software system configurations. Applying CIT to highly-configurable software systems is complicated by the fact that, in many such systems, there are constraints between specific configuration parameters that render certain combinations invalid. Many CIT algorithms lack a mechanism to avoid these. In recent work, automated constraint solving methods have been combined with search-based CIT construction methods to address the constraint problem with promising results. However, these techniques can incur a non-trivial overhead. In this paper, we build upon our previous work to develop a family of greedy CIT sample generation algorithms that exploit calculations made by modern boolean satisfiability (SAT) solvers to prune the search space of the CIT problem. We perform a comparative evaluation of the cost-effectiveness of these algorithms on four real-world highly-configurable software systems and on a population of synthetic examples that share the characteristics of those systems. In combination our techniques reduce the cost of CIT in the presence of constraints to 30\% of the cost of widely-used unconstrained CIT methods without sacrificing the quality of the solutions.

[1] D.L. Parnas, “On the Design and Development of Program Families,” IEEE Trans. Software Eng., vol. 2, no. 1, pp. 1-9, 1976.
[2] D. Dvorak, R. Rasmussen, G. Reeves, and A. Sacks, “Software Architecture Themes in JPL's Mission Data System,” Proc. IEEE Aerospace Conf., Mar. 2000.
[3] D. Kuhn, D.R. Wallace, and A.M. Gallo, “Software Fault Interactions and Implications for Software Testing,” IEEE Trans. Software Eng., vol. 30, no. 6, pp. 418-421, June 2004.
[4] C. Yilmaz, M.B. Cohen, and A. Porter, “Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces,” IEEE Trans. Software Eng., vol. 31, no. 1, pp. 20-34, Jan. 2006.
[5] Nokia Corporation, “Nokia Mobile Phone Line,” http://www.nokiausa.comphones, 2007.
[6] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437-444, July 1997.
[7] R. Brownlie, J. Prowse, and M.S. Phadke, “Robust Testing of AT&T PMX/StarMAIL Using OATS,” AT&T Technical J., vol. 71, no. 3, pp. 41-47, 1992.
[8] R.C. Bryce, C.J. Colbourn, and M.B. Cohen, “A Framework of Greedy Methods for Constructing Interaction Test Suites,” Proc. 27th Int'l Conf. Software Eng., pp. 146-155, May 2005.
[9] M.B. Cohen, C.J. Colbourn, P.B. Gibbons, and W.B. Mugridge, “Constructing Test Suites for Interaction Testing,” Proc. 25th Int'l Conf. Software Eng., pp. 38-48, May 2003.
[10] J. Czerwonka, “Pairwise Testing in Real World,” Proc. 24th Pacific Northwest Software Quality Conf., pp. 419-430, Oct. 2006.
[11] I.S. Dunietz, W.K. Ehrlich, B.D. Szablak, C.L. Mallows, and A. Iannino, “Applying Design of Experiments to Software Testing,” Proc. 19th Int'l Conf. Software Eng., pp. 205-215, 1997.
[12] C. Lott, A. Jain, and S. Dalal, “Modeling Requirements for Combinatorial Software Testing,” Proc. First Int'l Workshop Advances in Model-Based Testing, pp. 1-7, May 2005.
[13] C.J. Colbourn, M.B. Cohen, and R.C. Turban, “A Deterministic Density Algorithm for Pairwise Interaction Coverage,” Proc. IASTED Int'l Conf. Software Eng., pp. 345-352, Feb. 2004.
[14] A. Hartman and L. Raskin, “Problems and Algorithms for Covering Arrays,” Discrete Math., vol. 284, pp. 149-156, 2004.
[15] B. Hnich, S. Prestwich, E. Selensky, and B. Smith, “Constraint Models for the Covering Test Problem,” Constraints, vol. 11, pp.199-219, 2006.
[16] K. Nurmela, “Upper Bounds for Covering Arrays by Tabu Search,” Discrete Applied Math., vol. 138, nos. 1-2, pp. 143-152, 2004.
[17] K.C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing,” IEEE Trans. Software Eng., vol. 28, no. 1, pp. 109-111, Jan. 2002.
[18] Y. Tung and W.S. Aldiwan, “Automating Test Case Generation for the New Generation Mission Software System,” Proc. IEEE Aerospace Conf., pp. 431-437, 2000.
[19] A.W. Williams and R.L. Probert, “A Measure for Component Interaction Test Coverage,” Proc. ACS/IEEE Int'l Conf. Computer Systems and Applications, pp. 301-311, Oct. 2001.
[20] G. Sherwood, “,”, 2006.
[21] IBM alphaWorks, “IBM Intelligent Test Case Handler,” , 2005.
[22] R.C. Bryce and C.J. Colbourn, “Prioritized Interaction Testing for Pair-Wise Coverage with Seeding and Constraints,” J. Information and Software Technology, vol. 48, no. 10, pp. 960-970, 2006.
[23] M.B. Cohen, M.B. Dwyer, and J. Shi, “Interaction Testing of Highly-Configurable Systems in the Presence of Constraints,” Proc. Int'l Symp. Software Testing and Analysis, pp. 129-139, July 2007.
[24] M.B. Cohen, M.B. Dwyer, and J. Shi, “Exploiting Constraint Solving History to Construct Interaction Test Suites,” Proc. Testing: Academic and Industrial Conf.—Practice and Research Techniques, pp.121-130, Sept. 2007.
[25] M.B. Cohen, M.B. Dwyer, and J. Shi, “Coverage and Adequacy in Software Product Line Testing,” Proc. ISSTA Workshop Role of Software Architecture for Testing and Analysis, pp. 53-63, July 2006.
[26] R. Mandl, “Orthogonal Latin Squares: An Application of Experiment Design to Compiler Testing,” Comm. ACM, vol. 28, no. 10, pp. 1054-1058, Oct. 1985.
[27] J.D. McGregor, “Testing a Software Product Line,” technical report, Software Eng. Inst., Carnegie Mellon Univ., Dec. 2001.
[28] A. Hartman, “Software and Hardware Testing Using Combinatorial Covering Suites,” Proc. Graph Theory, Combinatorics and Algorithms: Interdisciplinary Applications, pp. 266-327, 2005.
[29] M. Davis and H. Putnam, “A Computing Procedure for Quantification Theory,” J. ACM, vol. 7, no. 3, pp. 201-215, 1960.
[30] J.P. Marques-Silva and K.A. Sakallah, “GRASP: A Search Algorithm for Propositional Satisfiability,” IEEE Trans. Computers, vol. 48, no. 5, pp. 506-521, May 1999.
[31] T. Walsh, “SAT v CSP,” Proc. Int'l Conf. Principles and Practice of Constraint Programming, vol. 1894, pp. 441-456, Sept. 2000.
[32] N. Eén and N. Sörrenson, MiniSAT-C v1.14.1, MiniSatMiniSat.html, 2007.
[33] G.J. Holzmann, “The Model Checker SPIN,” IEEE Trans. Software Eng., vol. 23, no. 5, pp. 279-295, May 1997.
[34] Free Software Foundation, “GNU 4.1.1 manpages,” /, 2005.
[35] Apache Software Foundation, “Apache HTTP sever,” , 2007.
[36] Mozilla Organization, “Bugzilla,”, 2007.
[37] G.J. Holzmann, “On-the-Fly, LTL Model Checking with SPIN: Man Pages,”, 2006.
[38] C. Tinelli, “A DPLL-Based Calculus for Ground Satisfiability Modulo Theories,” Proc. Eighth European Conf. Logics in Artificial Intelligence, G. Ianni and S. Flesca, eds., pp. 308-319, 2002.
[39] M. Harman, “The Current State and Future of Search Based Software Engineering,” Proc. Future of Software Eng., pp. 342-357, 2007.
[40] B. Korel, “Automated Software Test Data Generation,” IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.
[41] P. McMinn, “Search-Based Software Test Data Generation: A Survey,” J. Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[42] R. Pargas, M.J. Harrold, and R. Peck, “Test-Data Generation Using Genetic Algorithms,” J. Software Testing, Verification and Reliability, vol. 9, no. 3, pp. 263-282, 1999.
[43] Z. Li, M. Harman, and R.M. Hierons, “Search Algorithms for Regression Test Case Prioritization,” IEEE Trans. Software Eng., vol. 33, no. 4, pp. 225-237, Apr. 2007.
[44] M.B. Cohen, C.J. Colbourn, and A.C.H. Ling, “Augmenting Simulated Annealing to Build Interaction Test Suites,” Proc. 14th IEEE Int'l Symp. Software Reliability Eng., pp. 394-405, Nov. 2003.
[45] S. Yoo and M. Harman, “Pareto Efficient Multi-Objective Test Case Selection,” Proc. Int'l Symp. Software Testing and Analysis, pp.140-150, July 2007.

Index Terms:
Testing strategies, Testing tools
Myra B. Cohen, Matthew B. Dwyer, Jiangfan Shi, "Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach," IEEE Transactions on Software Engineering, vol. 34, no. 5, pp. 633-650, Sept.-Oct. 2008, doi:10.1109/TSE.2008.50
Usage of this product signifies your acceptance of the Terms of Use.