This Article 
 Bibliographic References 
 Add to: 
Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments
March/April 2011 (vol. 37 no. 2)
pp. 161-187
Samar Mouchawrab, Carleton University, Ottawa, Canada
Lionel C. Briand, Simula Research Laboratory, Lysaker, Norway
Yvan Labiche, Carleton University, Ottawa, Canada
Massimiliano Di Penta, University of Sannio, Benevento, Italy
A large number of research works have addressed the importance of models in software engineering. However, the adoption of model-based techniques in software organizations is limited since these models are perceived to be expensive and not necessarily cost-effective. Focusing on model-based testing, this paper reports on a series of controlled experiments. It investigates the impact of state machine testing on fault detection in class clusters and its cost when compared with structural testing. Based on previous work showing this is a good compromise in terms of cost and effectiveness, this paper focuses on a specific state-based technique: the round-trip paths coverage criterion. Round-trip paths testing is compared to structural testing, and it is investigated whether they are complementary. Results show that even when a state machine models the behavior of the cluster under test as accurately as possible, no significant difference between the fault detection effectiveness of the two test strategies is observed, while the two test strategies are significantly more effective when combined by augmenting state machine testing with structural testing. A qualitative analysis also investigates the reasons why test techniques do not detect certain faults and how the cost of state machine testing can be brought down.

[1] Eclipse, http:/, 2009.
[2] Eclipse Metrics Plugin,, 2009.
[3] Eclipse Test and Performance Tools Platform Project (TPTP), http://www.eclipse.orgtptp, 2009.
[4] Software-Artifact Infrastructure Repository (SIR), http://sir.unl. edu/portalindex.html, 2009.
[5] J.H. Andrews, L.C. Briand, and Y. Labiche, “Is Mutation an Appropriate Tool for Testing Experiments?” Proc. 27th Int'l Conf. Software Eng., pp. 402-411, 2005.
[6] J.H. Andrews, L.C. Briand, Y. Labiche, and A.S. Namin, “Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria,” IEEE Trans. Software Eng., vol. 32, no. 8, pp. 608-624, Aug. 2006.
[7] E. Arisholm and D.I. Sjøberg, “Evaluating the Effect of a Delegated versus Centralized Control Style on the Maintainability of Object-Oriented Software,” IEEE Trans. Software Eng., vol. 30, no. 8, pp. 521-534, Aug. 2004.
[8] D. Baldwin and F. Sayward, “Heuristics for Determining Equivalence of Program Mutations,” Dept. of Computer Science 276, Yale Univ., 1979.
[9] V.R. Basili, F. Shull, and F. Lanubile, “Building Knowledge through Families of Experiments,” IEEE Trans. Software Eng., vol. 25, no. 4, pp. 456-473, July/Aug. 1999.
[10] B. Beizer, Software Testing Techniques, second ed. Van Nostrand Reinhold Co., 1990.
[11] J.M. Bieman and J.L. Schultz, “An Empirical Evaluation (and Specification) of the All-du-Paths Testing Criterion,” ACM Software Eng. J., vol. 7, no. 1, pp. 43-51, 1992.
[12] R.V. Binder, Testing Object-Oriented Systems—Models, Patterns, and Tools. Addison-Wesley, 1999.
[13] K. Bogdanov and M. Holcombe, “Statechart Testing Method for Aircraft Control Systems,” Software Testing, Verification and Reliability, vol. 11, no. 1, pp. 39-54, 2001.
[14] L.C. Briand, “A Critical Analysis of Empirical Research in Software Testing,” Proc. Int'l Symp. Empirical Software Eng. and Measurement, Sept. 2007.
[15] L.C. Briand, M. Di Penta, and Y. Labiche, “Assessing and Improving State-Based Class Testing: A Series of Experiments,” IEEE Trans. Software Eng., vol. 30, no. 11, pp. 770-783, Nov. 2004.
[16] L.C. Briand and Y. Labiche, “A UML-Based Approach to System Testing,” Software and Systems Modeling, vol. 1, no. 1, pp. 10-42, 2002.
[17] L.C. Briand, Y. Labiche, and J. Cui, “Automated Support for Deriving Test Requirements from UML Statecharts,” J. Software and Systems Modeling, special issue, vol. 4, no. 4, pp. 399-423, 2005.
[18] L.C. Briand, Y. Labiche, and Q. Lin, “Improving State-Based Coverage Criteria Using Data Flow Information,” Proc. IEEE Int'l Conf. Software Reliability Eng., pp. 95-104, 2005.
[19] L.C. Briand, Y. Labiche, and H. Sun, “Investigating the Use of Analysis Contracts to Improve the Testability of Object-Oriented Code,” Software—Practice and Experience, vol. 33, no. 7, pp. 637-672, 2003.
[20] L.C. Briand, Y. Labiche, and Y. Wang, “Using Simulation to Empirically Investigate State Coverage Criteria Based on Statecharts,” Proc. ACM Int'l Conf. Software Eng., pp. 86-95, May 2004.
[21] B. Bruegge and A.H. Dutoit, Object-Oriented Software Engineering: Using UML, Patterns and Java. Prentice Hall, 2004.
[22] P. Chevalley and P. Thevenod-Fosse, “Automated Generation of Statistical Test Cases from UML State Diagrams,” Proc. 25th Ann. Int'l Computer Software and Applications Conf., pp. 205-214, 2001.
[23] P. Chevalley and P. Thevenod-Fosse, “An Empirical Evaluation of Statistical Testing Designed from UML State Diagrams: The Flight Guidance System Case Study,” Proc. 12th Int'l Symp. Software Reliability Eng., pp. 254-263, 2001.
[24] T.S. Chow, “Testing Software Design Modeled by Finite-State Machines,” IEEE Trans. Software Eng., vol. 4, no. 3, pp. 178-187, May 1978.
[25] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, second ed. Lawrence Erlbaum Assoc. Inc., 1988.
[26] J. Daly, A. Brooks, J. Miller, M. Roper, and M. Wood, “Verification of Results in Software Maintenance through External Replication,” Proc. Int'l Conf. Software Maintenance, pp. 50-57, 1994.
[27] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
[28] J.L. Devore and N. Farnum, Applied Statistics for Engineers and Scientists, 1999.
[29] T. Dybå, V.B. Kampenes, and D.I.K. Sjøberg, “A Systematic Review of Statistical Power in Software Engineering Experiments,” Information and Software Technology, vol. 48, no. 8, pp. 745-755, 2006.
[30] G.A. Ferguson and Y. Takane, Statistical Analysis in Psychology and Education, sixth ed. McGraw-Hill Ryerson Limited, 2005.
[31] P.G. Frankl and S.N. Weiss, “An Experimental Comparison of the Effectiveness of the All-Uses and All-Edges Adequacy Criteria,” Proc. Fourth Symp. Software Testing, Analysis and Verification, pp. 154-164, Oct. 1991.
[32] P.G. Frankl, S.N. Weiss, and C. Hu, “All-Uses versus Mutation Testing: An Experimental Comparison of Effectiveness,” Systems and Software, vol. 38, no. 3, pp. 235-253, 1997.
[33] H. Gomaa, Designing Concurrent, Distributed, and Real-Time Applications with UML. Addison-Wesley Professional, 2000.
[34] R.J. Harris, A Primer of Multivariate Statistics. Lawrence Erlbaum Assoc., 2001.
[35] S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian J. Statistics, vol. 6, pp. 65-70, 1979.
[36] N. Holt, B. Anda, K. Asskildt, L.C. Briand, J. Endresen, and S. Frøystein, “Experiences with Precise State Modeling in an Industrial Safety Critical System,” Proc. Critical Systems Development Using Modeling Languages, Workshop in Conjunction with UML '06, pp. 68-77, 2006.
[37] R. Holt, W.D. Boehm-Davis, and A.C. Shultz, “Mental Representations of Programs for Student and Professional Programmers,” Empirical Studies of Programmers: Second Workshop, pp. 33-46, Ablex Publishing Corp., 1987.
[38] H.S. Hong, Y.G. Kim, S.D. Cha, D.H. Bae, and H. Ural, “A Test Sequence Selection Method for Statecharts,” Software Testing, Verification and Reliability, vol. 10, no. 4, pp. 203-227, 2000.
[39] M. Höst, B. Regnell, and C. Wohlin, “Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment,” Empirical Software Eng., vol. 5, no. 3, pp. 201-214, 2000.
[40] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments of the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria,” Proc. 16th Int'l Conf. Software Eng., 1994.
[41] A. Kleppe, J. Warmer, and W. Bast, MDA Explained: The Model Driven Architecture: Practice and Promise. Addison-Wesley Professional, 2003.
[42] C. Larman, Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development. Prentice Hall, 2004.
[43] D. Lee and M. Yannakakis, “Principles and Methods of Testing Finite State Machines—A Survey,” Proc. IEEE, vol. 84, no. 8, pp. 1090-1123, Aug. 1996.
[44] P.S. Levy and S. Lemeshow, Sampling of Populations: Methods and Applications, third ed. Wiley, 1999.
[45] Y.-S. Ma, Y.-R. Kwon, and J. Offutt, “Inter-Class Mutation Operators for Java,” Proc. 13th Int'l Symp. Software Reliability Eng., Nov. 2002.
[46] Y.-S. Ma and J. Offutt, “Description of Method-Level Mutation Operators for Java,” mutopsMethod.pdf , 2010.
[47] Y.-S. Ma, J. Offutt, and Y.R. Kwon, “MuJava: An Automated Class Mutation System,” Software Testing, Verification and Reliability, vol. 15, no. 2, pp. 97-133, 2005.
[48] B. Marick, Craft of Software Testing: Subsystems Testing Including Object-Based and Object-Oriented Testing. Prentice-Hall, 1985.
[49] P. McMinn and M. Holcombe, “Evolutionary Testing Using an Extended Chaining Approach,” Evolutionary Computation, vol. 14, no. 1, pp. 41-64, 2006.
[50] B. Meyer, “Applying Design by Contract,” Computer, vol. 25, no. 10, pp. 40-51, Oct. 1992.
[51] S. Mouchawrab, L.C. Briand, Y. Labiche, and M. Di Penta, “Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments,” Carleton Univ. TR SCE-08-09, 2009.
[52] K.R. Murphy and B. Myors, Statistical Power Analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests. Lawrence Erlbaum, 1998.
[53] C. Nebut, F. Fleurey, Y. Le Traon, and J.-M. Jezequel, “Automatic Test Generation: A Use Case Driven Approach,” IEEE Trans. Software Eng., vol. 32, no. 3, pp. 140-155, Mar. 2006.
[54] A.J. Offutt and A. Abdurazik, “Generating Tests from UML Specifications,” Proc. Second Int'l Conf. Unified Modeling Language, pp. 416-429, 1999.
[55] A.J. Offutt and W.M. Craft, “Using Compiler Optimization Techniques to Detect Equivalent Mutants,” J. Software Testing, Verification and Reliability, vol. 4, no. 3, pp. 131-154, 1994.
[56] J. Offutt, S. Liu, A. Abdurazik, and P. Ammann, “Generating Test Data from State-Based Specifications,” J. Software Testing, Verification and Reliability, vol. 13, no. 1, pp. 25-53, 2003.
[57] J. Offutt and J. Pan, “Automatically Detecting Equivalent Mutants and Infeasible Paths,” Software Testing, Verification and Reliability, vol. 7, no. 3, pp. 165-192, 1997.
[58] T.J. Ostrand and M.J. Balcer, “The Category-Partition Method for Specifying and Generating Fuctional Tests,” Comm. ACM, vol. 31, no. 6, pp. 676-686, 1988.
[59] P. Piwowarski, M. Ohba, and J. Caruso, “Coverage Measurement Experience during Function Test,” Proc. 15th Int'l Conf. Software Eng., 1998.
[60] A. Pretschner, W. Prenninger, S. Wagner, C. Kühnel, M. Baumgartner, B. Sostawa, R. Zölch, and T. Stauner, “One Evaluation of Model-Based Testing and Its Automation,” Proc. Int'l Conf. Software Eng., pp. 392-401, 2005.
[61] F. Shull, J. Singer, and D.I.K. Sjøberg, Guide to Advanced Empirical Software Engineering. Springer, 2008.
[62] S. Siegel, Non-Parametric Statistics for the Behavioral Sciences. McGraw-Hill, 1956.
[63] S. Sinha and M.J. Harrold, “Analysis and Testing of Programs with Exception Handling Constructs,” IEEE Trans. Software Eng., vol. 26, no. 9, pp. 849-871, Sept. 2000.
[64] L.E. Toothaker, Introductory Statistics for the Behavioral Sciences, second ed. McGraw-Hill College, 1986.
[65] M. Utting and B. Legeard, Practical Model-Based Testing: A Tools Approach. Morgan-Kaufmann, 2006.
[66] J. Warmer and A. Kleppe, The Object Constraint Language: Getting Your Models Ready for MDA. Addison-Wesley Professional, 2004.
[67] E. Weyuker, “The Cost of Data Flow Testing: An Empirical Study,” IEEE Trans. Software Eng., vol. 16, no. 2, pp. 121-128, Feb. 1990.
[68] E. Weyuker, T. Goradia, and A. Singh, “Automatically Generating Test Data from a Boolean Specification,” IEEE Trans. Software Eng., vol. 20, no. 5, pp. 353-363, May 1994.
[69] C. Wohlin, P. Runeson, M. Host, M.C. Ohlsson, B. Regnell, and A. Wesslen, Experimentation in Software Engineering: An Introduction. Kluwer, 2000.

Index Terms:
State-based software testing, structural testing, controlled experiments, state machines.
Samar Mouchawrab, Lionel C. Briand, Yvan Labiche, Massimiliano Di Penta, "Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments," IEEE Transactions on Software Engineering, vol. 37, no. 2, pp. 161-187, March-April 2011, doi:10.1109/TSE.2010.32
Usage of this product signifies your acceptance of the Terms of Use.