This Article 
 Bibliographic References 
 Add to: 
Assessing and Improving State-Based Class Testing: A Series of Experiments
November 2004 (vol. 30 no. 11)
pp. 770-793
Lionel C. Briand, IEEE Computer Society
This paper describes an empirical investigation of the cost effectiveness of well-known state-based testing techniques for classes or clusters of classes that exhibit a state-dependent behavior. This is practically relevant as many object-oriented methodologies recommend modeling such components with statecharts which can then be used as a basis for testing. Our results, based on a series of three experiments, show that in most cases state-based techniques are not likely to be sufficient by themselves to catch most of the faults present in the code. Though useful, they need to be complemented with black-box, functional testing. We focus here on a particular technique, Category Partition, as this is the most commonly used and referenced black-box, functional testing technique. Two different oracle strategies have been applied for checking the success of test cases. One is a very precise oracle checking the concrete state of objects whereas the other one is based on the notion of state invariant (abstract states). Results show that there is a significant difference between them, both in terms of fault detection and cost. This is therefore an important choice to make that should be driven by the characteristics of the component to be tested, such as its criticality, complexity, and test budget.

[1] dnsjava Home Page, http:/, Apr. 2004.
[2] A.T. Acree, T.A. Budd, R.A. Demillo, R.J. Lipton, and F.G. Sayward, “Mutation Analysis,” Technical Report GIT-ICS-79/08, School of Information and Computer Science, Georgia Inst. of Tech nology, Sept. 1979.
[3] R.T. Alexander and A.J. Offutt, “Criteria for Testing Polymorphic Relationships,” Proc. 11th IEEE Int'l Symp. Software Reliability Eng. (ISSRE '00), pp. 15-23, 2000.
[4] G. Antoniol, L.C. Briand, M. Di Penta, and Y. Labiche, “A Case Study Using the Round-Trip Strategy for State-Based Class Testing,” Proc. 13th IEEE Int'l Symp. Software Reliability Eng. (ISSRE '02), pp. 269-279, Nov. 2002.
[5] T. Ball, D. Hoffman, F. Ruskey, R. Webber, and L. White, “State Generation and Automated Class Testing,” Software Testing, Verification and Reliability, vol. 10, no. 3, pp. 149-170, 2000.
[6] B. Beizer, Software Testing Techniques, second ed. Van Nostrand Reinhold, 1990.
[7] R.V. Binder, Testing Object-Oriented Systems— Models, Patterns, and Tools, Object Technology. Addison-Wesley, 1999.
[8] K. Bogdanov and M. Holcombe, “Statechart Testing Method for Aircraft Control Systems,” Software Testing, Verification and Reliability, vol. 11, no. 1, pp. 39-54, 2001.
[9] G. Booch, J. Rumbaugh, and I. Jacobson, The Unified Modeling Language User Guide. Addison Wesley, 1999.
[10] L. Briand, Y. Labiche, and Y. Wang, “Using Simulation to Empirically Investigate Test Coverage Criteria,” Proc. IEEE/ACM Int'l Conf. Software Eng., pp. 86-95, May 2004.
[11] L.C. Briand, M. Di Penta, and Y. Labiche, “Assessing and Improving State-Based Class Testing: A Series of Experiments,” Technical Report SCE-04-02, Carleton Univ., http://www.sce. carleton.casquall, 2004.
[12] L.C. Briand, W. Dzidek, and Y. Labiche, “Using Aspect-Oriented Programming to Instrument OCL Contracts in Java,” Technical Report SCE-04-03, Carleton Univ., http://www.sce.carleton.casquall, 2004.
[13] B. Bruegge and A.H. Dutoit, Object-Oriented Software Engineering— Conquering Complex and Changing Systems. Prentice Hall, 2000.
[14] T.A. Budd and D. Angluin, “Two Notions of Correctness and their Relation to Testing,” Acta Informatica, vol. 18, no. 1, pp. 31-45, 1982.
[15] T.S. Chow, “Testing Software Design Modeled by Finite-State Machines,” IEEE Trans. Software Eng., vol. 4, no. 3, pp. 178-187, 1978.
[16] M.E. Delamaro, J.C. Maldonado, and A.P. Mathur, “Interface Mutation: An Approach for Integration Testing,” IEEE Trans. Software Eng., vol. 27, no. 3, pp. 228-247, 2001.
[17] R.A. DeMillo, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, vol. 11, pp. 34-41, 1978.
[18] J.L. Devore, Probability and Statistics for Engineering and the Sciences, fifth ed. Duxbury Press, 1999.
[19] N.E. Fenton and S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach, second ed. PWS Publishing, 1998.
[20] P.G. Frankl and S.N. Weiss, “An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing,” IEEE Trans. Software Eng., vol. 19, no. 8, pp. 774-787, 1993.
[21] P.G. Frankl, S.N. Weiss, and C. Hu, “All-Uses versus Mutation Testing: An Experimental Comparison of Effectiveness,” J. Systems and Software, vol. 38, no. 3, pp. 235-253, 1997.
[22] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Patterns— Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995.
[23] H. Gomaa, Designing Concurrent, Distributed, and Real-Time Applications with UML, Object Technology. Addison Wesley, 2000.
[24] T.L. Graves, M.J. Harrold, J.-M. Kim, A. Porter, and G. Rothermel, “An Empirical Study of Regression Test Selection Techniques,” ACM Trans. Software Eng. and Methodology, vol. 10, no. 2, pp. 184-208, 2001.
[25] M.J. Harrold, J.D. McGregor, and K.J. Fitzpatrick, “Incremental Testing of Object-Oriented Class Structures,” Proc. 14th IEEE Int'l Conf. Software Eng. (ICSE), pp. 68-80, May 1992.
[26] H.S. Hong, Y.G. Kim, S.D. Cha, D.H. Bae, and H. Ural, “A Test Sequence Selection Method for Statecharts,” Software Testing, Verification and Reliability, vol. 10, no. 4, pp. 203-227, 2000.
[27] N. Juristo and A.M. Moreno, Basics of Software Engineering Experimentation. Kluwer, 2001.
[28] S. Kim, J.A. Clark, and J.A. McDermid, “The Rigorous Generation of Java Mutation Using HAZOP,” Proc. 12th Int'l Conf. Software and Systems Eng. and Their Applications (ICSSEA '99), pp. 9-10, Dec. 1999.
[29] S. Kim, J.A. Clark, and J.A. McDermid, “Class Mutation: Mutation Testing for Object-Oriented Programs,” Proc. Net. Object Days, Oct. 2000.
[30] S. Kim, J.A. Clark, and J.A. McDermid, “Investigating the Effectiveness of Object-Oriented Testing Strategies with the Mutation Method,” Proc. Mutation 2000 Conf., Oct. 2000.
[31] K.N. King and A.J. Offutt, “A Fortran Language System for Mutation-Based Software Testing,” Software— Practice and Experience, vol. 21, no. 7, pp. 686-718, 1991.
[32] D. Kung, N. Suchak, J. Gao, P. Hsia, Y. Toyoshima, and C. Chen, “On Object State Testing,” Proc. IEEE Int'l Computer Software and Application Conf. (COMPSAC), pp. 222-227, 1994.
[33] D. Lee and M. Yannakakis, “Principles and Methods of Testing Finite State Machines— A Survey,” Proc. IEEE, vol. 84, no. 8, pp. 1090-1123, 1996.
[34] B. Meyer, “Design by Contracts,” Computer, vol. 25, no. 10, pp. 40-52, 1992.
[35] A.J. Offutt, “Investigations of the Software Testing Coupling Effect,” ACM Trans. Software Eng. and Methodology, vol. 1, no. 1, pp. 3-18, 1992.
[36] A.J. Offutt and A. Abdurazik, “Generating Tests from UML Specifications,” Proc. Second Int'l Conf. the Unified Modeling Language (UML '99), pp. 416-429, Oct. 1999.
[37] A.J. Offutt, Y. Xiong, and S. Liu, “Criteria for Generating Specification-Based Tests,” Proc. Fifth Int'l Conf. Eng. of Complex Computer Systems (ICECCS), pp. 119-129, Oct. 1999.
[38] T.J. Ostrand and M.J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Test,” Comm. ACM, vol. 31, no. 6, pp. 676-686, 1988.
[39] W.M. Trochim, “The Research Methods Knowledge Base,” , 2004.
[40] J. Warmer and A. Kleppe, The Object Constraint Language. Addison-Wesley, errata at errata.html , 1999.
[41] E. Weyuker, “The Cost of Data Flow Testing: An Empirical Study,” IEEE Trans. Software Eng., vol. 16, no. 2, pp. 121-128, 1990.
[42] E. Weyuker, “Automatically Generating Test Data from a Boolean Specification,” IEEE Trans. Software Eng., vol. 20, no. 5, pp. 353-363, 1994.
[43] E.J. Weyuker, “On Testing Nontestable Programs,” The Computer J., vol. 25, no. 4, pp. 465-470, 1982.
[44] C. Wohlin, P. Runeson, M. Host, M.C. Ohlsson, B. Regnell, and A. Wesslen, Experimentation in Software Engineering— An Introduction. Kluwer, 2000.

Index Terms:
State-based testing, testing experimentation, UML statecharts, category partition.
Lionel C. Briand, Massimiliano Di Penta, Yvan Labiche, "Assessing and Improving State-Based Class Testing: A Series of Experiments," IEEE Transactions on Software Engineering, vol. 30, no. 11, pp. 770-793, Nov. 2004, doi:10.1109/TSE.2004.79
Usage of this product signifies your acceptance of the Terms of Use.