This Article 
 Bibliographic References 
 Add to: 
A Framework for Specification-Based Testing
November 1996 (vol. 22 no. 11)
pp. 777-793

Abstract—Test templates and a test template framework are introduced as useful concepts in specification-based testing. The framework can be defined using any model-based specification notation and used to derive tests from model-based specifications—in this paper, it is demonstrated using the Z notation. The framework formally defines test data sets and their relation to the operations in a specification and to other test data sets, providing structure to the testing process. Flexibility is preserved, so that many testing strategies can be used. Important application areas of the framework are discussed, including refinement of test data, regression testing, and test oracles.

[1] O.-J. Dahl, E.W. Dijkstra, and C.A.R. Hoare, Structured Programming, APIC Studies in Data Processing, no. 8. Academic Press, 1972.
[2] Andrew S. Tanenbaum, "In Defense of Program Testing or Correctness Proofs Considered Harmful," SIGPLAN Notices, vol. 11, p. 68, May 1976.
[3] D. Carrington and P. Stocks, "A Tale of Two Paradigms: Formal Methods and Software Testing," Proc. Eighth Z User Meeting, Springer-Verlag, pp. 51-68, 1994.
[4] J.B. Goodenough and S.L. Gerhart, "Toward a Theory of Test Data Selection," IEEE Trans. Software Eng., vol. 1, no. 2, pp. 156-173, June 1975.
[5] D.J. Richardson, S.L. Aha, and T.O. O'Malley, “Specification-Based Test Oracles for Reactive Systems,” Proc. Int'l Conf. Software Eng. (ICSE), pp. 105–118, May 1992.
[6] B. Hetzel, The Complete Guide to Software Testing, QED Information Sciences, Wellesley, Mass., second edition, 1988.
[7] C. B. Jones,Systematic Software Development Using VDM. Englewood Cliffs, NJ: Prentice-Hall, 1990, 2nd ed.
[8] Specification Case Studies, Series in Computer Science. I. Hayes, ed., Prentice Hall Int'l, second edition, 1993.
[9] J.M. Spivey, The Z Notation: A Reference Manual, Prentice-Hall, Englewood Cliffs, N.J., 1992.
[10] P.A.V. Hall, "Towards Testing with Respect to Formal Specifications," Second IEE/BCS Conf. Software Eng. 88. pp. 159-163, July 1988.
[11] P.A.V. Hall, "Relationship Between Specifications and Testing," Information and Software Technology, vol. 33, no. 1, pp. 47-52, 1991.
[12] J. Dick and A. Faivre, "Automating the Generating and Sequencing of Test Cases from Model-Based Specification," FME '93: Industrial Strength Formal Methods, Fifth Int'l Symp. Formal Methods Europe, J.C.P. Woodcock and P.G. Larsen, eds., Odense, Denmark, vol. 670of Lecture Notes in Computer Science, pp. 268-284. Springer-Verlag, Apr. 1992.
[13] H. Hörcher, “Improving Software Tests Using Z Specifications,” in Bowen and Hinchey [4], pp. 152–166,
[14] H.-M. Hörcher and J. Peleska, "Using Formal Specifications to Support Software Testing," Software Quality J., vol. 4, no. 4, pp. 309-327, Dec. 1995.
[15] S. Stepney, "Testing as Abstraction," Bowen and Hinchey [41], pp. 137-151, Proc. Ninth Int'l Conf. Z Users.
[16] P.A. Stocks and D.A. Carrington, "Test Templates: A Specification-Based Testing Framework," Proc. IEEE Int'l Conf. Software Eng. (ICSE-15), pp. 405-414, May 1993.
[17] P. Stocks, "Applying Formal Methods to Software Testing," PhD thesis, the Univ. of Queensland, Australia, 1993.
[18] T.J. Ostrand and M.J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Tests,” Comm. ACM, vol. 31, no. 6, June 1988.
[19] M. Balcer, W. Hasling, and T. Ostrand, “Automatic Generation of Test Scripts from Formal Test Specifications,” Proc. Third Symp. Software Testing, Analysis, and Verification, pp. 210–218, Dec. 1989.
[20] J. Pachl, "A Notation for Specifying Test Selection Criteria," Protocol Specification, Testing, and Verification, X, L. Logrippo, R.L. Probert, and H. Ural, eds., pp. 71-84. North-Holland, 1990.
[21] T.J. Ostrand, R. Sigal, and E.J. Weyuker, "Design for a Tool to Manage Specification-Based Testing," Workshop on Software Testing,Banff, Canada, pp. 41-50, IEEE CS Press, July 1986.
[22] R.A. Kemmerer, "Testing Formal Specifications to Detect Design Errors," IEEE Trans. Software Eng., vol. 11, no. 1, pp. 32-43, Jan. 1985.
[23] R. Kneuper, "Symbolic Execution as a Tool for Validation of Specifications," PhD dissertation, Technical Report UMCS-89-7-1, Dept. of Computer Science, Univ. of Manchester, 1989.
[24] E.J. Weyuker and T.J. Ostrand, "Theories of Program Testing and the Application of Revealing Subdomains," IEEE Trans. Software Eng., vol. 6, no. 3, pp. 236-246, May 1980.
[25] D.J. Richardson and L.A. Clarke, "Partition Analysis: A Method Combining Testing and Verification," IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1,477-1,490, Dec. 1985.
[26] J.W. Duran and S.C. Ntafos, "An Evaluation of Random Testing," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.
[27] P. Stocks and D. Carrington, “Test Template Framework: A Specification-Based Testing Case Study,” Proc. Int'l Symp. Software Testing and Analysis (ISSTA '93), pp. 11–18, June 1993.
[28] I. MacColl, D. Carrington, and P. Stocks, "An Experiment in Specification-Based Testing," Proc. 19th Australasian Computer Science Conf., ACSC'96, K. Ramamohanarao, ed., vol. 18, pp. 159-168, Feb. 1996.
[29] The RAISE Language Group, The RAISE Specification Language. The BCS Practitioners Series, Prentice Hall, 1992.
[30] E.J. Weyuker and B. Jeng,“Analyzing partition testing strategies,” IEEE Trans. Software Engineering, vol. 17, pp. 703-711, 1991.
[31] A. Parrish and S.H. Zweben, "Analysis and Refinement of Software Test Data Adequacy Properties," IEEE Trans. Software Eng., vol. 17, no. 6, pp. 565-581, June 1991.
[32] E.J. Weyuker, T. Goradia, and A. Singh, "Automatically Generating Test Data from Boolean Specification," IEEE Trans. Software Eng., vol. 20, no. 5, pp. 353-363, May 1994.
[33] P. Stocks and D.A. Carrington, "Deriving Software Test Cases from Formal Specifications," Sixth Australian Software Eng. Conf., pp. 327-340, July 1991.
[34] R. G. Hamlet,“Theoretical comparison of testing methods,”inProc. Third Symp. Testing, Anal., Verification, Key West, FL, Dec. 1989, pp. 28–37.
[35] D. Hamlet and J. Voas, "Faults on its sleeve: Amplifying software reliability testing," 1993 Int'l Symp. Software Testing and Analysis (ISSTA),Cambridge, Mass., pp. 89-98, June28-30, 1993, ACM SIGSOFT Software Engineering Notes, vol. 18 no. 3, July 1993.
[36] I.J. Hayes, "Specification directed module testing," IEEE Trans. Software Engineering, vol. 12, pp. 124-133, Jan. 1986.
[37] W. Howden, “A Functional Approach to Program Testing and Analysis,” IEEE Trans. Software Eng., vol. 12, pp. 997-1005, 1986.
[38] W.E. Howden, Functional Program Testing Analysis. McGraw-Hill, 1987.
[39] D. Carrington,D. Duke,I. Hayes,, and J. Welsh,“Deriving modular designs from formal specifications,” Software Engineering Notes, Proc. First ACM SIGSOFT Symp. Foundations of Software Engineering, vol. 18, no. 5, pp. 89-98,Los Angeles, Calif., Dec.7-10, 1993.
[40] J. Gannon, R. Hamlet, and P. McMullin, “Data Abstraction Implementation, Specification, and Testing,” ACM Trans. Programming Languages and Systems, vol. 3, pp. 211-223, 1981.
[41] "ZUM '95: The Z Formal Specification Notation," J.P. Bowen and M.G. Hinchey, eds., vol. 967, Lecture Notes in Computer Science,Limerick, Ireland, Proc. Ninth Int'l Conf. of Z Users, Springer-Verlag, June 1995.

Index Terms:
Specification-based testing, testing strategies, test data, test oracles, Z notation.
Phil Stocks, David Carrington, "A Framework for Specification-Based Testing," IEEE Transactions on Software Engineering, vol. 22, no. 11, pp. 777-793, Nov. 1996, doi:10.1109/32.553698
Usage of this product signifies your acceptance of the Terms of Use.