This Article 
 Bibliographic References 
 Add to: 
Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces
January 2006 (vol. 32 no. 1)
pp. 20-34
Many modern software systems are designed to be highly configurable so they can run on and be optimized for a wide variety of platforms and usage scenarios. Testing such systems is difficult because, in effect, you are testing a multitude of systems, not just one. Moreover, bugs can and do appear in some configurations, but not in others. Our research focuses on a subset of these bugs that are "option-related”—those that manifest with high probability only when specific configuration options take on specific settings. Our goal is not only to detect these bugs, but also to automatically characterize the configuration subspaces (i.e., the options and their settings) in which they manifest. To improve efficiency, our process tests only a sample of the configuration space, which we obtain from mathematical objects called covering arrays. This paper compares two different kinds of covering arrays for this purpose and assesses the effect of sampling strategy on fault characterization accuracy. Our results strongly suggest that sampling via covering arrays allows us to characterize option-related failures nearly as well as if we had tested exhaustively, but at a much lower cost. We also provide guidelines for using our approach in practice.

[1] L. Breiman, J. Freidman, R. Olshen, and C. Stone, Classification and Regression Trees. Wadsworth, 1984.
[2] R. Brownlie, J. Prowse, and M.S. Phadke, “Robust Testing of AT&T PMX/StarMAIL Using OATS,” AT&T Technical J., vol. 71, no. 3, pp. 41-47, 1992.
[3] K. Burr and W. Young, “Combinatorial Test Techniques: Table-Based Automation, Test Generation and Code Coverage,” Proc. Int'l Conf. Software Testing, Analysis, and Review, 1998.
[4] M. Chateauneuf and D. Kreher, “On the State of Strength-Three Covering Arrays,” J. Combinatorial Designs, vol. 10, no. 4, pp. 217-238, 2002.
[5] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437-444, 1997.
[6] M.B. Cohen, C.J. Colbourn, J. Collofello, P.B. Gibbons, and W.B. Mugridge, “Variable Strength Interaction Testing of Components,” Proc. Int'l Computer Software and Applications Conf. (COMPSAC), pp. 413-418, 2003.
[7] M.B. Cohen, C.J. Colbourn, P.B. Gibbons, and W.B. Mugridge, “Constructing Test Suites for Interaction Testing,” Proc. Int'l Conf. Software Eng. (ICSE), pp. 38-44 2003.
[8] S.R. Dalal, A. Jain, N. Karunanithi, J.M. Leaton, C.M. Lott, G.C. Patton, and B.M. Horowitz, “Model-Based Testing in Practice,” Proc. Int'l Conf. Software Eng. (ICSE), pp. 285-294, 1999.
[9] I.S. Dunietz, W.K. Ehrlich, B.D. Szablak, C.L. Mallows, and A. Iannino, “Applying Design of Experiments to Software Testing,” Proc. Int'l Conf. Software Eng. (ICSE), pp. 205-215, 1997.
[10] D. Kuhn and M. Reilly, “An Investigation of the Applicability of Design of Experiments to Software Testing,” Proc. 27th Ann. NASA Goddard/IEEE Software Eng. Workshop, pp. 91-95, 2002.
[11] B. Liblit, A. Aiken, Z. Zheng, and M. Jordan, “Bug Isolation via Remote Program Sampling,” Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 141-154, 2003.
[12] R. Mandl, “Orthogonal Latin Squares: An Application of Experiment Design to Compiler Testing,” Comm. ACM, vol. 28, no. 10, pp. 1054-1058, 1985.
[13] A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D.C. Schmidt, and B. Natarajan, “Skoll: Distributed Continuous Quality Assurance,” Proc. Int'l Conf. Software Eng. (ICSE), pp. 459-468, 2004.
[14] C.V. Rijsbergen, Information Retrieval. London, UK: Butterworths, 1979.
[15] D. Schmidt, D. Levine, and S. Mungee, “The Design and Performance of the TAO Real-Time Object Request Broker,” Computer Comm., special issue on building quality of service into distributed systems, vol. 21, no. 4, 1998.
[16] D.C. Schmidt and S.D. Huston, C++ Network Programming, Volume 1: Mastering Complexity with ACE and Patterns. Boston: Addison-Wesley, 2002.
[17] I.H. Witten and E. Frank,Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 1999.
[18] C. Yilmaz, A. Krishna, A. Memon, A. Porter, D. Schmidt, A. Gokhale, and B. Natarajan, “Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems,” Proc. Int'l Conf. Software Eng. (ICSE), pp. 293-302, 2005.
[19] C. Yilmaz, M.B. Cohen, and A. Porter, “Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces,” Proc. Int'l Symp. Software Testing and Analysis (ISSTA), pp. 45-54, 2004.
[20] A. Zeller and R. Hildebrandt, “Simplifying and Isolating Failure-Inducing Input,” IEEE Trans. Software Eng., vol. 28, no. 2, pp. 183-200, 2002.

Index Terms:
Software testing, distributed continuous quality assurance, fault characterization, covering arrays.
Cemal Yilmaz, Myra B. Cohen, Adam A. Porter, "Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces," IEEE Transactions on Software Engineering, vol. 32, no. 1, pp. 20-34, Jan. 2006, doi:10.1109/TSE.2006.8
Usage of this product signifies your acceptance of the Terms of Use.