The Community for Technology Leaders
RSS Icon
Issue No.10 - Oct. (2013 vol.39)
pp: 1376-1402
Shaukat Ali , Simula Research Lab, Norway
Muhammad Zohaib Iqbal , National University of Computer and Emerging Sciences, Islamabad and University of Luxembourg, Luxembourg
Andrea Arcuri , Simula Research Lab, Norway
Lionel C. Briand , University of Luxembourg, Luxembourg
Model-based testing (MBT) aims at automated, scalable, and systematic testing solutions for complex industrial software systems. To increase chances of adoption in industrial contexts, software systems can be modeled using well-established standards such as the Unified Modeling Language (UML) and the Object Constraint Language (OCL). Given that test data generation is one of the major challenges to automate MBT, we focus on test data generation from OCL constraints in this paper. This endeavor is all the more challenging given the numerous OCL constructs and operations that are designed to facilitate the definition of constraints. Though search-based software testing has been applied to test data generation for white-box testing (e.g., branch coverage), its application to the MBT of industrial software systems has been limited. In this paper, we propose a set of search heuristics targeted to OCL constraints to guide test data generation and automate MBT in industrial applications. We evaluate these heuristics for three search algorithms: Genetic Algorithm, (1+1) Evolutionary Algorithm, and Alternating Variable Method. We empirically evaluate our heuristics using complex artificial problems, followed by empirical analyses of the feasibility of our approach on one industrial system in the context of robustness testing. Our approach is also compared with the most widely referenced OCL solver (UMLtoCSP) in the literature and shows to be significantly more efficient.
model-based testing, OCL, search-based testing, test data generation, empirical evaluation, search-based software engineering,
Shaukat Ali, Muhammad Zohaib Iqbal, Andrea Arcuri, Lionel C. Briand, "Generating Test Data from OCL Constraints with Search Techniques", IEEE Transactions on Software Engineering, vol.39, no. 10, pp. 1376-1402, Oct. 2013, doi:10.1109/TSE.2013.17
[1] M. Utting and B. Legeard, Practical Model-Based Testing: A Tools Approach. Morgan-Kaufmann, 2007.
[2] OCL, Object Constraint Language Specification, Version 2.2,, 2011.
[3] MOF, Meta Object Facility (MOF),, 2006.
[4] T. Yue, L. Briand, B. Selic, and Q. Gan, "Experiences with Model-Based Product Line Engineering for Developing a Family of Integrated Control Systems: An Industrial Case Study," Technical Report 2012-06, Simula Research Laboratory, 2012.
[5] S. Ali, L. Briand, A. Arcuri, and S. Walawege, "An Industrial Application of Robustness Testing Using Aspect-Oriented Modeling, UML/MARTE, and Search Algorithms," Proc. 14th ACM/IEEE Int'l Conf. Model Driven Eng. Languages and Systems, 2011.
[6] A. Arcuri, M.Z. Iqbal, and L. Briand, "Black-Box System Testing of Real-Time Embedded Systems Using Random and Search-Based Testing," Proc. IFIP Int'l Conf. Testing Software and Systems, 2010.
[7] M.Z. Iqbal, A. Arcuri, and L. Briand, "Environment Modeling with UML/MARTE to Support Black-Box System Testing for Real-Time Embedded Systems: Methodology and Industrial Case Studies," Proc. 13th Int'l Conf. Model Driven Eng. Languages and Systems, pp. 286-300, 2010.
[8] CertifyIt, CertifyIt, http:/, 2011.
[9] QTRONIC, QTRONIC, http://www.conformiq.comqtronic.php, 2010.
[10] L.v. Aertryck and T. Jensen, "UML-Casting: Test Synthesis from UML Models Using Constraint Resolution," Proc. Approches Formelles dans l'Assistance au Développement de Logiciels, 2003.
[11] M. Benattou, J. Bruel, and N. Hameurlain, "Generating Test Data from OCL Specification," Proc. ECOOP Workshop Integration and Transformation of UML Models, 2002.
[12] L. Bao-Lin, L. Zhi-shu, L. Qing, and C.Y. Hong, "Test Case Automate Generation from UML Sequence Diagram and OCL Expression," Proc. Int'l Conf. Computational Intelligence and Security, 2007.
[13] S. Ali, M.Z. Iqbal, A. Arcuri, and L. Briand, "A Search-Based OCL Constraint Solver for Model-Based Test Data Generation," Proc. 11th Int'l Conf. Quality Software, pp. 41-50, 2011.
[14] J. Cabot, R. Claris, and D. Riera, "Verification of UML/OCL Class Diagrams Using Constraint Programming," Proc. IEEE Int'l Conf. Software Testing Verification and Validation Workshop, 2008.
[15] A.P. Mathur, Foundations of Software Testing. Pearson Education, 2008.
[16] A. Bertolino, "Software Testing Research: Achievements, Challenges, Dreams," Proc. Future of Software Eng., 2007.
[17] M. Harman and B.F. Jones, "Search-Based Software Engineering," Information and Software Technology, vol. 43, pp. 833-839, 2001.
[18] J. Clarke, J.J. Dolado, M. Harman, R. Hierons, B. Jones, M. Lumkin, B. Mitchell, S. Mancoridis, K. Rees, M. Roper, and M. Shepperd, "Reformulating Software Engineering as a Search Problem," IEE Software vol. 150, no. 3, pp. 161-175, May/June 2003.
[19] M. Harman, A. Mansouri, and Y. Zhang, "Search Based Software Engineering: Trends, Techniques and Applications," ACM Computing Surveys, vol. 45, article 11, 2012.
[20] R. Drechsler and N. Drechsler, Evolutionary Algorithms for Embedded System Design. Kluwer Academic Publishers, 2002.
[21] D.A. Coley, An Introduction to Genetic Algorithms for Scientists and Engineers. World Scientific Publishing Company, 1997.
[22] W. Afzal, R. Torkar, and R. Feldt, "A Systematic Review of Search-Based Testing for Non-Functional System Properties," Information and Software Technology, vol. 51, pp. 957-976, 2009.
[23] T. Mantere and J.T. Alander, "Evolutionary Software Engineering, a Review," Applied Soft Computing, vol. 5, pp. 315-331, 2005.
[24] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, p. 52, 2004.
[25] J.T.d. Souza, C.L. Maia, F.G.d. Freitas, and D.P. Coutinho, "The Human Competitiveness of Search Based Software Engineering," Proc. Second Int'l Symp. Search Based Software Eng., 2010.
[26] G. Fraser and A. Arcuri, "EvoSuite: Automatic Test Suite Generation for Object-Oriented Software," Proc. ACM Symp. Foundations of Software Eng., pp. 416-419, 2011.
[27] G. Fraser and A. Arcuri, "Sound Empirical Evidence in Software Testing," Proc. ACM/IEEE Int'l Conf. Software Eng., 2012.
[28] E.K. Burke and G. Kendall, Search Methodologies: Introductory Tutorials in Optimization and Decision Support Techniques. Springer, 2006.
[29] A. Arcuri, M.Z. Iqbal, and L. Briand, "Random Testing: Theoretical Results and Practical Implications," IEEE Trans. Software Eng., vol. 38, no. 2, pp. 258-277, Mar./Apr. 2012.
[30] S. Ali, L.C. Briand, H. Hemmati, and R.K. Panesar-Walawege, "A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation," IEEE Trans. Software Eng., vol. 36, no. 6, pp. 742-762, Nov./Dec. 2010.
[31] P. McMinn, "Search-Based Software Test Data Generation: A Survey: Research Articles," Software Testing Verification and Reliability, vol. 14, pp. 105-156, 2004.
[32] S. Droste, T. Jansen, and I. Wegener, "On the Analysis of the (1+1) Evolutionary Algorithm," Theoretical Computer Science, vol. 276, pp. 51-81, 2002.
[33] B. Korel, "Automated Software Test Data Generation," IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.
[34] D. Chiorean, M. Bortes, D. Corutiu, C. Botiza, and A. Cârcu, "OCLE," V2.0 ed, 2010.
[35] Model Development Tools, , 2012.
[36] Dresden OCL, DresdenOCL , 2012.
[37] UML-based Specification Environment (USE), http://sourceforge. net/apps/mediawiki/useocl index.php?title=The_UML-based_ Specification_Environment , 2012.
[38] M. Egea, "EyeOCL Software," 2010.
[39] B. Bordbar and K. Anastasakis, "UML2Alloy: A Tool for Lightweight Modelling of Discrete Event Systems," Proc. IADIS Int'l Conf. Applied Computing, 2005.
[40] D. Distefano, J.-P. Katoen, and A. Rensink, "Towards Model Checking OCL," Proc. ECOOP-Workshop Defining Precise Semantics for UML, 2000.
[41] M. Clavel and M.A.G.d. Dios, "Checking Unsatisfiability for OCL Constraints," Proc. Ninth OCL Workshop at the UML/MoDELS Conf., 2009.
[42] B.K. Aichernig and P.A.P. Salas, "Test Case Generation by OCL Mutation and Constraint Solving," Proc. Fifth Int'l Conf. Quality Software, 2005.
[43] D. Berardi, D. Calvanese, and G.D. Giacomo, "Reasoning on UML Class Diagrams," Artificial Intelligence, vol. 168, pp. 70-118, 2005.
[44] J. Winkelmann, G. Taentzer, K. Ehrig, and J.M. ster, "Translation of Restricted OCL Constraints into Graph Constraints for Generating Meta Model Instances by Graph Grammars," Electronic Notes Theoretical Computer Science, vol. 211, pp. 159-170, 2008.
[45] M. Kyas, H. Fecher, F.S.d. Boer, J. Jacob, J. Hooman, M.v.d. Zwaag, T. Arons, and H. Kugler, "Formalizing UML Models and OCL Constraints in PVERSUS," Electronic Notes Theoretical Computer Science, vol. 115, pp. 39-47, 2005.
[46] M.P. Krieger, A. Knapp, and B. Wolff, "Automatic and Efficient Simulation of Operation Contracts," Proc. Ninth Int'l Conf. Generative Programming and Component Eng., 2010.
[47] S. Weißleder and B.-H. Schlingloff, "Deriving Input Partitions from UML Models for Automatic Test Generation," Models in Software Eng., pp. 151-163, Springer-Verlag, 2008.
[48] M. Gogolla, F. Bttner, and M. Richters, "USE: A UML-Based Specification Environment for Validating UML and OCL," Science Computer Programming, vol. 69, pp. 27-34, 2007.
[49] A.D. Brucker, M.P. Krieger, D. Longuet, and B. Wolff, "A Specification-Based Test Case Generation Method for UML/OCL," Proc. Int'l Conf. Models in Software Eng., 2011.
[50] W. Ahrendt, T. Baar, B. Beckert, M. Giese, E. Habermalz, R. Hähnle, W. Menzel, and P.H. Schmitt, "The KeY Approach: Integrating Object Oriented Design and Formal Verification," Proc. European Workshop Logics in Artificial Intelligence, 2000.
[51] D. Jackson, I. Schechter, and H. Shlyahter, "Alcoa: The Alloy Constraint Analyzer," Proc. 22nd Int'l Conf. Software Eng., 2000.
[52] M. Krieger and A. Knapp, "Executing Underspecified OCL Operation Contracts with a SAT Solver," Proc. Eighth Int'l Workshop OCL Concepts and Tools, 2008.
[53] C. Doungsa-ard, K. Dahal, A. Hossain, and T. Suwannasart, "GA-Based Automatic Test Data Generation for UML State Diagrams with Parallel Paths," Advanced Design and Manufacture to Gain a Competitive Edge, pp. 147-156, Springer 2008.
[54] R. Lefticaru and F. Ipate, "Functional Search-Based Testing from State Machines," Proc. Int'l Conf. Software Testing, Verification, and Validation, 2008.
[55] M. Alshraideh and L. Bottaci, "Search-Based Software Test Data Generation for String Data Using Program-Specific Search Operators: Research Articles," Software Testing Verification and Reliability, vol. 16, pp. 175-203, 2006.
[56] P. McMinn, M. Shahbaz, and M. Stevenson, "Search-Based Test Input Generation for String Data Types Using the Results of Web Queries," Proc. Fifth IEEE Int'l Conf. Software Testing, Verification and Validation, 2012.
[57] G. Fraser and A. Arcuri, "The Seed Is Strong: Seeding Strategies in Search-Based Software Testing," Proc. Fifth IEEE Int'l Conf. Software Testing, Verification and Validation, 2012.
[58] G. Fraser and A. Arcuri, "It Is Not the Length That Matters, It Is How You Control It," Proc. Fourth IEEE Int'l Conf. Software Testing, Verification and Validation, 2011.
[59] G. Fraser and A. Arcuri, "Whole Test Suite Generation," IEEE Trans. Software Eng., vol. 39, no. 2, pp. 276-291, Feb. 2013.
[60] S. Poulding, J.A. Clark, and H. Waeselynck, "A Principled Evaluation of the Effect of Directed Mutation on Search-Based Statistical Testing," Proc. Fourth IEEE Int'l Conf. Software Testing, Verification and Validation Workshops, 2011.
[61] S. Yoo, M. Harman, and S. Ur, "Highly Scalable Multi Objective Test Suite Minimisation Using Graphics Cards," Proc. Third Int'l Conf. Search Based Software Eng., 2011.
[62] M. Harman, S.A. Mansouri, and Y. Zhang, "Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications," Technical Report TR-09-03, King's College, 2009.
[63] A. Arcuri, "It Really Does Matter How You Normalize the Branch Distance in Search-Based Software Testing," Proc. Third Int'l Conf. Software Testing, Verification and Reliability, 2011.
[64] R.V. Binder, Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley Longman Publishing Co., Inc., 1999.
[65] H. Li and Gordon, "Bytecode Testability Transformation," Proc. Symp. Search Based Software Eng. Co-Located with ESEC/FSE, 2011.
[66] S. Ali, H. Hemmati, N.E. Holt, E. Arisholm, and L.C. Briand, "Model Transformations as a Strategy to Automate Model-Based Testing—A Tool and Industrial Case Studies," technical report, Simula Research Laboratory, 2010.
[67] M. Phil, "Input Domain Reduction through Irrelevant Variable Removal and Its Effect on Local, Global, and Hybrid Search-Based Structural Test Data Generation," IEEE Trans. Software Eng., vol. 38, no. 2, pp. 453-477, Mar./Apr. 2012.
[68] Kermeta, Kermeta—Breathe Life into Your Metamodels, http:/, 2011.
[69] F. Lindlar and A. Windisch, "A Search-Based Approach to Functional Hardware-in-the-Loop Testing," Proc. Second Int'l Symp. Search Based Software Eng., 2010.
[70] S. Ali, L.C. Briand, and H. Hemmati, "Modeling Robustness Behavior Using Aspect-Oriented Modeling to Support Robustness Testing of Industrial Systems," Software and Systems Modeling, vol. 11, pp. 633-670, 2012.
[71] Cisco C90, index.html, 2012.
[72] MARTE, Modeling and Analysis of Real-Time and Embedded Systems (MARTE), http:/, 2010.
[73] A. Arcuri and L. Briand, "A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering," Proc. Int'l Conf. Software Eng., 2011.
[74] A. Arcuri and G. Fraser, "On Parameter Tuning in Search Based Software Engineering," Proc. Int'l Symp. Search Based Software Eng., 2011.
[75] D.J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures. Chapman and Hall/CRC, 2007.
62 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool