This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Incremental Test Generation for Software Product Lines
May/June 2010 (vol. 36 no. 3)
pp. 309-322
ASCII Text
x 
 
Engin Uzuncaova, Sarfraz Khurshid, Don Batory, "Incremental Test Generation for Software Product Lines," IEEE Transactions on Software Engineering, vol. 36, no. 3, pp. 309-322, May/June, 2010.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2010.30,
author = {Engin Uzuncaova and Sarfraz Khurshid and Don Batory},
title = {Incremental Test Generation for Software Product Lines},
journal ={IEEE Transactions on Software Engineering},
volume = {36},
number = {3},
issn = {0098-5589},
year = {2010},
pages = {309-322},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2010.30},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - Incremental Test Generation for Software Product Lines
IS - 3
SN - 0098-5589
SP309
EP322
EPD - 309-322
A1 - Engin Uzuncaova,
A1 - Sarfraz Khurshid,
A1 - Don Batory,
PY - 2010
KW - Software/program verification
KW - testing and debugging
KW - software engineering.
VL - 36
JA - IEEE Transactions on Software Engineering
ER -
 
Engin Uzuncaova, Microsoft, Redmond
Sarfraz Khurshid, University of Texas at Austin, Austin
Don Batory, University of Texas at Austin, Austin
Recent advances in mechanical techniques for systematic testing have increased our ability to automatically find subtle bugs, and hence, to deploy more dependable software. This paper builds on one such systematic technique, scope-bounded testing, to develop a novel specification-based approach for efficiently generating tests for products in a software product line. Given properties of features as first-order logic formulas in Alloy, our approach uses SAT-based analysis to automatically generate test inputs for each product in a product line. To ensure soundness of generation, we introduce an automatic technique for mapping a formula that specifies a feature into a transformation that defines incremental refinement of test suites. Our experimental results using different data structure product lines show that an incremental approach can provide an order of magnitude speedup over conventional techniques. We also present a further optimization using dedicated integer constraint solvers for feature properties that introduce integer constraints, and show how to use a combination of solvers in tandem for solving Alloy formulas.

[1] W. Adjie-Winoto, E. Schwartz, H. Balakrishnan, and J. Lilley, "The Design and Implementation of an Intentional Naming System," Proc. 17th ACM Symp. Operating Systems Principles, Dec. 1999.
[2] T. Asikainen, T. Soininen, and T. Männistö, "A Koala Based Approach for Modelling and Deploying Configurable Software Product Families," Proc. Fifth Int'l Workshop Software Product-Family Eng., 2003.
[3] C.W. Barrett, D.L. Dill, and A. Stump, "Checking Satisfiability of First-Order Formulas by Incremental Translation to SAT," Proc. 14th Int'l Conf. Computer Aided Verification, July 2002.
[4] D. Batory, "Feature Models, Grammars, and Propositional Formulas," Proc. Ninth Int'l Software Product Line Conf., 2005.
[5] D. Batory, "From Implementation to Theory in Program Synthesis," Proc. 34th Ann. ACM Symp. Principles of Programming Languages, 2007.
[6] D. Batory, G. Chen, E. Robertson, and T. Wang, "Design Wizards and Visual Programming Environments for Genvoca Generators," IEEE Trans. Software Eng., vol. 26, no. 5, pp. 441-452, May 2000.
[7] D. Batory, B. Lofaso, and Y. Smaragdakis, "JTS: Tools for Implementing Domain-Specific Languages," Proc. Int'l Conf. Software Reuse, 1998.
[8] D. Batory and S. O'Malley, "The Design and Implementation of Hierarchical Software Systems with Reusable Components," Comm. ACM, vol. 1, no. 4, pp. 355-398, 1992.
[9] D. Batory, J.N. Sarvela, and A. Rauschmayer, "Scaling Step-Wise Refinement," Proc. 25th Int'l Conf. Software Eng., 2003.
[10] D. Batory, J.N. Sarvela, and A. Rauschmayer, "Scaling Step-Wise Refinement," IEEE Trans. Software Eng., vol. 30, no. 6, pp. 355-371, June 2004.
[11] A. Bertolino and S. Gnesi, "PLUTO: A Test Methodology for Product Families," Proc. Fifth Int'l Workshop Software Product-Family Eng., 2003.
[12] M. Calder, M. Kolberg, E.H. Magill, and S. Reiff-Marganiec, "Feature Interaction: A Critical Review and Considered Forecast," Computer Networks, vol. 41, no. 1, pp. 115-141, 2003.
[13] J. Chang and D.J. Richardson, "Structural Specification-Based Testing: Automated Support and Experimental Evaluation," Proc. Seventh ACM SIGSOFT Symp. Foundations of Software Eng., Sept. 1999.
[14] A.S. Christensen, A. Møller, and M.I. Schwartzbach, "Precise Analysis of String Expressions," Proc. 10th Int'l Static Analysis Symp., 2003.
[15] Y.L.T. Clémentine Nebut and J.-M. Jézéquel, "System Testing of Product Lines: From Requirements to Test Cases," Software Product Lines—Research Issues in Eng. and Management, pp. 447-478, Springer, 2006.
[16] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, "The AETG System: An Approach to Testing Based on Combinatorial Design," IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437-444, July 1997.
[17] M.B. Cohen, M.B. Dwyer, and J. Shi, "Exploiting Constraint Solving History to Construct Interaction Test Suites," Proc. Testing: Academic and Industrial Conf. Practice and Research Techniques— MUTATION, pp. 121-132, Sept. 2007.
[18] M.B. Cohen, M.B. Dwyer, and J. Shi, "Coverage and Adequacy in Software Product Line Testing," Proc. ISSTA 2006 Workshop Role of Software Architecture for Testing and Analysis, July 2006.
[19] T.H. Cormen, C.E. Leiserson, and R.L. Rivest, Introduction to Algorithms. The MIT Press, 1990.
[20] K. Czarnecki and K. Pietroszek, "Verifying Feature-Based Model Templates against Well-Formedness OCL Constraints," Proc. Fifth Int'l Conf. Generative Programming and Component Eng., 2006.
[21] L. de Moura and N. Bjørner, "Z3: An Efficient SMT Solver," Proc. 14th Int'l Conf. Tools and Algorithms for Construction and Analysis of Systems, Apr. 2008.
[22] C. Denger and R. Kolb, "Testing and Inspecting Reusable Product Line Components: First Empirical Results," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng., 2006.
[23] D. Jackson, "Alloy: A Lightweight Object Modeling Notation," ACM Trans. Software Eng. and Methodology, vol. 11, no. 2, pp. 256-290, Apr. 2002.
[24] S.A. Edwards, "The Challenges of Hardware Synthesis from C-Like Languages," Proc. Conf. Design Automation and Test in Europe, 2005.
[25] N. Een and N. Sorensson, "An Extensible SAT-Solver," Proc. Sixth Int'l Conf. Theory and Applications of Satisfiability Testing, May 2003.
[26] P. Godefroid, N. Klarlund, and K. Sen, "Dart: Directed Automated Random Testing," Proc. 2005 ACM SIGPLAN Conf. Programming Language Design and Implementation, 2005.
[27] J. Goodenough and S. Gerhart, "Toward a Theory of Test Data Selection," IEEE Trans. Software Eng., vol. 1, no. 2, pp. 156-173, June 1975.
[28] W. Grieskamp, Y. Gurevich, W. Schulte, and M. Veanes, "Generating Finite State Machines from Abstract State Machines," Proc. Int'l Symp. Software Testing and Analysis, July 2002.
[29] S.A. Hendrickson and A. van der Hoek, "Modeling Product Line Architectures through Change Sets and Relationships," Proc. 29th Int'l Conf. Software Eng., 2007.
[30] D. Jackson, Software Abstractions: Logic, Language and Analysis. The MIT Press, 2006.
[31] T. Kahsai, M. Roggenbach, and B.-H. Schlingloff, "Specification-Based Testing for Software Product Lines," Proc. Sixth IEEE Int'l Conf. Software Eng. and Formal Methods, 2008.
[32] S.A. Khalek and S. Roychowdhury, "Modeling Linked Data Structures Using SMT Solvers," Class Project Report for Khurshid's Graduate Course on Verification and Validation of Software (EE382C-3), 2008.
[33] S. Khurshid, "Generating Structurally Complex Tests from Declarative Constraints," PhD thesis, Dept. of Electrical Eng. and Computer Science, Massachusetts Inst. of Tech nology, Dec. 2003.
[34] S. Khurshid and D. Marinov, "Checking Java Implementation of a Naming Architecture Using Testera," Electronic Notes in Theoretical Computer Science, vol. 55, pp. 322-342, 2001.
[35] S. Khurshid, C. Pasareanu, and W. Visser, "Generalized Symbolic Execution for Model Checking and Testing," Proc. Ninth Int'l Symp. Theoretical Aspects of Computer Software, Apr. 2003.
[36] J.C. King, "Symbolic Execution and Program Testing," Comm. ACM, vol. 19, no. 7, pp. 385-394, 1976.
[37] A. Kuehlmann and J. Baumgartner, "Transformation-Based Verification Using Generalized Retiming," Proc. Int'l Conf. Computer-Aided Verification, 2001.
[38] G.T. Leavens, A.L. Baker, and C. Ruby, "Preliminary Design of JML: A Behavioral Interface Specification Language for Java," Technical Report TR 98-06i, Dept. of Computer Science, Iowa State Univ., June 1998.
[39] A.H. Lin, "Automated Analysis of Security APIs," master's thesis, Dept. of Electrical Eng. and Computer Science, Massachusetts Inst. of Tech nology, May 2005.
[40] B. Liskov and J. Guttag, Program Development in Java: Abstraction, Specification, and Object-Oriented Design. Addison-Wesley, 2000.
[41] J. Liu, D. Batory, and S. Nedunuri, "Modeling Interactions in Feature Oriented Software Designs," Proc. Eighth Int'l Conf. Feature Interactions in Telecomm. and Software Systems, 2005.
[42] D. Marinov and S. Khurshid, "TestEra: A Novel Framework for Automated Testing of Java Programs," Proc. 16th IEEE Int'l Conf. Automated Software Eng., Nov. 2001.
[43] H. Muccini and A. van der Hoek, "Towards Testing Product Line Architectures," Proc. Int'l Workshop Test and Analysis of Component-Based Systems, Apr. 2003.
[44] Z. Pap, M. Subramaniam, G. Kovács, and G.Á. Németh, "A Bounded Incremental Test Generation Algorithm for Finite State Machines," Proc. Seventh Int'l Workshop Formal Approaches to Testing of Software, 2007.
[45] B.C. Pierce, Basic Category Theory for Computer Scientists. The MIT Press, 1991.
[46] K. Pohl and A. Metzger, "Variability Management in Software Product Line Engineering," Proc. 28th Int'l Conf. Software Eng., 2006.
[47] M. Roggenbach, "CSP-CASL—A New Integration of Process Algebra and Algebraic Specification," Theoretical Computer Science, vol. 354, pp. 42-71, 2006.
[48] G. Rothermel and M.J. Harrold, "Analyzing Regression Test Selection Techniques," IEEE Trans. Software Eng., vol. 22, no. 8, pp. 529-551, Aug. 1996.
[49] P.G. Selinger, M.M. Astrahan, D.D. Chamberlin, R.A. Lorie, and T.G. Price, "Access Path Selection in a Relational Database Management System," Proc. ACM SIGMOD, 1979.
[50] K. Sen, D. Marinov, and G. Agha, "CUTE: A Concolic Unit Testing Engine for C," Proc. 13th ACM SIGSOFT Symp. Foundations of Software Eng., Sept. 2005.
[51] D. Shao, S. Khurshid, and D. Perry, "Whispec: White-Box Testing of Libraries Using Declarative Specifications," Proc. ACM SIGPLAN Symp. Library-Centric Software Design, Oct. 2007.
[52] K. Sullivan, J. Yang, D. Coppit, S. Khurshid, and D. Jackson, "Software Assurance by Bounded Exhaustive Testing," Proc. Int'l Symp. Software Testing and Analysis, 2004.
[53] M. Taghdiri, "Inferring Specifications to Detect Errors in Code," Proc. 19th IEEE Int'l Conf. Automated Software Eng., 2004.
[54] E. Torlak and D. Jackson, "Kodkod: A Relational Model Finder," Proc. 13th Int'l Conf. Tools and Algorithms for Construction and Analysis of Systems, Mar. 2007.
[55] S. Trujillo, D. Batory, and O. Diaz, "Feature Oriented Model Driven Development: A Case Study for Portlets," Proc. 29th Int'l Conf. Software Eng., May 2007.
[56] E. Uzuncaova, "Efficient Specification-Based Testing Using Incremental Techniques," PhD thesis, Univ. of Texas at Austin, 2008.
[57] E. Uzuncaova, D. Garcia, S. Khurshid, and D. Batory, "Testing Software Product Lines Using Incremental Test Generation," Proc. 19th Int'l Symp. Software Reliability Eng., Nov. 2008.
[58] E. Uzuncaova and S. Khurshid, "Constraint Prioritization for Efficient Analysis of Declarative Models," Proc. 15th Int'l Symp. Formal Methods, May 2008.
[59] M. Vaziri, "Finding Bugs Using a Constraint Solver," PhD thesis, Computer Science and Artificial Intelligence Lab, Masschusetts Inst. of Tech nology, 2003.
[60] F. Zaraket, A. Aziz, and S. Khurshid, "Sequential Circuits for Relational Analysis," Proc. 29th Int'l Conf. Software Eng., May 2007.

Index Terms:
Software/program verification, testing and debugging, software engineering.
Citation:
Engin Uzuncaova, Sarfraz Khurshid, Don Batory, "Incremental Test Generation for Software Product Lines," IEEE Transactions on Software Engineering, vol. 36, no. 3, pp. 309-322, May-June 2010, doi:10.1109/TSE.2010.30
Usage of this product signifies your acceptance of the Terms of Use.