DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TSE.2013.53
Cemal Yilmaz , Sabanci University, Istanbul
Emine Dumlu , Sabanci University, Istanbul
Myra B. Cohen , University of Nebraska-Lincoln, Lincoln
Adam A. Porter , University of Maryland, College Park
The configuration spaces of software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. The basic justification for CIT approaches is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. We conjecture, however, that in practice some of these behaviors are not actually tested because of unanticipated masking effects -- failures that perturb execution so as to prevent some behaviors from being exercised. In this work, we attempt to reduce the consequences of masking effects. First we define a novel interaction coverage criterion, which aims to ensure that each test case has a fair chance to test all valid t-way combinations of option settings. We then introduce a feedback driven adaptive combinatorial testing process (FDA-CIT) to materialize this criterion. We also compare FDA-CIT against error locating arrays, a state of the art approach for detecting and locating failures, and against perfect test scenarios, in which all masking effects are known a priori. Our results suggest that masking effects do exist in practice, and that our approach provides an efficient way to work around them.
Testing tools, Testing and Debugging, Testing strategies, Test design
C. Yilmaz, E. Dumlu, M. B. Cohen and A. A. Porter, "Reducing Masking Effects in Combinatorial Interaction Testing: A Feedback Driven Adaptive Approach," in IEEE Transactions on Software Engineering.