The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.05 - Sept.-Oct. (2008 vol.34)
pp: 633-650
Myra B. Cohen , University of Nebraska-Lincoln, Lincoln
Matthew B. Dwyer , University of Nebraska-Lincoln, Lincoln
Jiangfan Shi , University of Nebraska-Lincoln, Lincoln
ABSTRACT
Researchers have explored the application of combinatorial interaction testing (CIT) methods to construct samples to drive systematic testing of software system configurations. Applying CIT to highly-configurable software systems is complicated by the fact that, in many such systems, there are constraints between specific configuration parameters that render certain combinations invalid. Many CIT algorithms lack a mechanism to avoid these. In recent work, automated constraint solving methods have been combined with search-based CIT construction methods to address the constraint problem with promising results. However, these techniques can incur a non-trivial overhead. In this paper, we build upon our previous work to develop a family of greedy CIT sample generation algorithms that exploit calculations made by modern boolean satisfiability (SAT) solvers to prune the search space of the CIT problem. We perform a comparative evaluation of the cost-effectiveness of these algorithms on four real-world highly-configurable software systems and on a population of synthetic examples that share the characteristics of those systems. In combination our techniques reduce the cost of CIT in the presence of constraints to 30\% of the cost of widely-used unconstrained CIT methods without sacrificing the quality of the solutions.
INDEX TERMS
Testing strategies, Testing tools
CITATION
Myra B. Cohen, Matthew B. Dwyer, Jiangfan Shi, "Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach", IEEE Transactions on Software Engineering, vol.34, no. 5, pp. 633-650, Sept.-Oct. 2008, doi:10.1109/TSE.2008.50
REFERENCES
[1] D.L. Parnas, “On the Design and Development of Program Families,” IEEE Trans. Software Eng., vol. 2, no. 1, pp. 1-9, 1976.
[2] D. Dvorak, R. Rasmussen, G. Reeves, and A. Sacks, “Software Architecture Themes in JPL's Mission Data System,” Proc. IEEE Aerospace Conf., Mar. 2000.
[3] D. Kuhn, D.R. Wallace, and A.M. Gallo, “Software Fault Interactions and Implications for Software Testing,” IEEE Trans. Software Eng., vol. 30, no. 6, pp. 418-421, June 2004.
[4] C. Yilmaz, M.B. Cohen, and A. Porter, “Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces,” IEEE Trans. Software Eng., vol. 31, no. 1, pp. 20-34, Jan. 2006.
[5] Nokia Corporation, “Nokia Mobile Phone Line,” http://www.nokiausa.comphones, 2007.
[6] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437-444, July 1997.
[7] R. Brownlie, J. Prowse, and M.S. Phadke, “Robust Testing of AT&T PMX/StarMAIL Using OATS,” AT&T Technical J., vol. 71, no. 3, pp. 41-47, 1992.
[8] R.C. Bryce, C.J. Colbourn, and M.B. Cohen, “A Framework of Greedy Methods for Constructing Interaction Test Suites,” Proc. 27th Int'l Conf. Software Eng., pp. 146-155, May 2005.
[9] M.B. Cohen, C.J. Colbourn, P.B. Gibbons, and W.B. Mugridge, “Constructing Test Suites for Interaction Testing,” Proc. 25th Int'l Conf. Software Eng., pp. 38-48, May 2003.
[10] J. Czerwonka, “Pairwise Testing in Real World,” Proc. 24th Pacific Northwest Software Quality Conf., pp. 419-430, Oct. 2006.
[11] I.S. Dunietz, W.K. Ehrlich, B.D. Szablak, C.L. Mallows, and A. Iannino, “Applying Design of Experiments to Software Testing,” Proc. 19th Int'l Conf. Software Eng., pp. 205-215, 1997.
[12] C. Lott, A. Jain, and S. Dalal, “Modeling Requirements for Combinatorial Software Testing,” Proc. First Int'l Workshop Advances in Model-Based Testing, pp. 1-7, May 2005.
[13] C.J. Colbourn, M.B. Cohen, and R.C. Turban, “A Deterministic Density Algorithm for Pairwise Interaction Coverage,” Proc. IASTED Int'l Conf. Software Eng., pp. 345-352, Feb. 2004.
[14] A. Hartman and L. Raskin, “Problems and Algorithms for Covering Arrays,” Discrete Math., vol. 284, pp. 149-156, 2004.
[15] B. Hnich, S. Prestwich, E. Selensky, and B. Smith, “Constraint Models for the Covering Test Problem,” Constraints, vol. 11, pp.199-219, 2006.
[16] K. Nurmela, “Upper Bounds for Covering Arrays by Tabu Search,” Discrete Applied Math., vol. 138, nos. 1-2, pp. 143-152, 2004.
[17] K.C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing,” IEEE Trans. Software Eng., vol. 28, no. 1, pp. 109-111, Jan. 2002.
[18] Y. Tung and W.S. Aldiwan, “Automating Test Case Generation for the New Generation Mission Software System,” Proc. IEEE Aerospace Conf., pp. 431-437, 2000.
[19] A.W. Williams and R.L. Probert, “A Measure for Component Interaction Test Coverage,” Proc. ACS/IEEE Int'l Conf. Computer Systems and Applications, pp. 301-311, Oct. 2001.
[20] G. Sherwood, “Testcover.com,” http://testcover.com/pubconstex.php, 2006.
[21] IBM alphaWorks, “IBM Intelligent Test Case Handler,” http://www.alphaworks.ibm.com/techwhitch , 2005.
[22] R.C. Bryce and C.J. Colbourn, “Prioritized Interaction Testing for Pair-Wise Coverage with Seeding and Constraints,” J. Information and Software Technology, vol. 48, no. 10, pp. 960-970, 2006.
[23] M.B. Cohen, M.B. Dwyer, and J. Shi, “Interaction Testing of Highly-Configurable Systems in the Presence of Constraints,” Proc. Int'l Symp. Software Testing and Analysis, pp. 129-139, July 2007.
[24] M.B. Cohen, M.B. Dwyer, and J. Shi, “Exploiting Constraint Solving History to Construct Interaction Test Suites,” Proc. Testing: Academic and Industrial Conf.—Practice and Research Techniques, pp.121-130, Sept. 2007.
[25] M.B. Cohen, M.B. Dwyer, and J. Shi, “Coverage and Adequacy in Software Product Line Testing,” Proc. ISSTA Workshop Role of Software Architecture for Testing and Analysis, pp. 53-63, July 2006.
[26] R. Mandl, “Orthogonal Latin Squares: An Application of Experiment Design to Compiler Testing,” Comm. ACM, vol. 28, no. 10, pp. 1054-1058, Oct. 1985.
[27] J.D. McGregor, “Testing a Software Product Line,” technical report, Software Eng. Inst., Carnegie Mellon Univ., Dec. 2001.
[28] A. Hartman, “Software and Hardware Testing Using Combinatorial Covering Suites,” Proc. Graph Theory, Combinatorics and Algorithms: Interdisciplinary Applications, pp. 266-327, 2005.
[29] M. Davis and H. Putnam, “A Computing Procedure for Quantification Theory,” J. ACM, vol. 7, no. 3, pp. 201-215, 1960.
[30] J.P. Marques-Silva and K.A. Sakallah, “GRASP: A Search Algorithm for Propositional Satisfiability,” IEEE Trans. Computers, vol. 48, no. 5, pp. 506-521, May 1999.
[31] T. Walsh, “SAT v CSP,” Proc. Int'l Conf. Principles and Practice of Constraint Programming, vol. 1894, pp. 441-456, Sept. 2000.
[32] N. Eén and N. Sörrenson, MiniSAT-C v1.14.1, http://www.cs.chalmers.se/Cs/Research/FormalMethods/ MiniSatMiniSat.html, 2007.
[33] G.J. Holzmann, “The Model Checker SPIN,” IEEE Trans. Software Eng., vol. 23, no. 5, pp. 279-295, May 1997.
[34] Free Software Foundation, “GNU 4.1.1 manpages,” http://gcc.gnu.org/onlinedocs/gcc-4.1.1gcc /, 2005.
[35] Apache Software Foundation, “Apache HTTP sever,” http://httpd.apache.org/docs/2.2/modquickreference.html , 2007.
[36] Mozilla Organization, “Bugzilla,” http://www.bugzilla.org/docs/tiphtml/, 2007.
[37] G.J. Holzmann, “On-the-Fly, LTL Model Checking with SPIN: Man Pages,” http://spinroot.com/spin/Manindex.html, 2006.
[38] C. Tinelli, “A DPLL-Based Calculus for Ground Satisfiability Modulo Theories,” Proc. Eighth European Conf. Logics in Artificial Intelligence, G. Ianni and S. Flesca, eds., pp. 308-319, 2002.
[39] M. Harman, “The Current State and Future of Search Based Software Engineering,” Proc. Future of Software Eng., pp. 342-357, 2007.
[40] B. Korel, “Automated Software Test Data Generation,” IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.
[41] P. McMinn, “Search-Based Software Test Data Generation: A Survey,” J. Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[42] R. Pargas, M.J. Harrold, and R. Peck, “Test-Data Generation Using Genetic Algorithms,” J. Software Testing, Verification and Reliability, vol. 9, no. 3, pp. 263-282, 1999.
[43] Z. Li, M. Harman, and R.M. Hierons, “Search Algorithms for Regression Test Case Prioritization,” IEEE Trans. Software Eng., vol. 33, no. 4, pp. 225-237, Apr. 2007.
[44] M.B. Cohen, C.J. Colbourn, and A.C.H. Ling, “Augmenting Simulated Annealing to Build Interaction Test Suites,” Proc. 14th IEEE Int'l Symp. Software Reliability Eng., pp. 394-405, Nov. 2003.
[45] S. Yoo and M. Harman, “Pareto Efficient Multi-Objective Test Case Selection,” Proc. Int'l Symp. Software Testing and Analysis, pp.140-150, July 2007.
25 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool