This Article 
 Bibliographic References 
 Add to: 
Using Test Oracles Generated from Program Documentation
March 1998 (vol. 24 no. 3)
pp. 161-173

Abstract—This paper illustrates how software can be described precisely using LD-relations, how these descriptions can be presented in a readable manner using tabular notations, and one way such descriptions can be used to test programs. We describe an algorithm that can be used to generate a test oracle from program documentation, and present the results of using a tool based on it to help test part of a commercial network management application. The results demonstrate that these methods can be effective at detecting errors and greatly increase the speed and accuracy of test evaluation when compared with manual evaluation. Such oracles can be used for unit testing, in situ testing, constructing self-checking software, and ensuring consistency between code and documentation.

[1] R.F. Abraham, "Evaluating Generalized Tabular Expressions in Software Documentation," MEng thesis, Dept. of Electrical and Computer Engineering, McMaster Univ., Hamilton, Ontario, Canada, also printed as CRL346, Telecommunications Research Inst. of Ontario, Feb. 1997.
[2] ADL Language Reference Manual for ANSI C Programmers, Release 1.1. Sun Microsystems Inc., Document Reference MITI/0002/D/R1.1, Dec. 1996.
[3] S. Antoy and R. Hamlet, "Objects that Check Themselves Against Formal Specifications," TR 91-1, Dept. of Computer Science, Portland State Univ. School of Engineering and Applied Sciences, Oregon, 1991.
[4] B.J. Bauer, "Documenting Complicated Programs," MEng thesis, Dept. of Electrical and Computer Engineering, McMaster Univ., Ontario, also printed as CRL316, Telecommunications Research Inst. of Ontario, Dec. 1995.
[5] G. Bernot, M.C. Gaudel, and B. Marre, Software Testing Based on Formal Specifications: A Theory and a Tool Software Eng. J., vol. 6, no. 6, pp. 387-405, 1991.
[6] D. Chapman, "A Program Testing Assistant," Comm. ACM, vol. 25, no. 9, pp. 625-634, Sept. 1982.
[7] S. Faulk et al., "The Core Method For Real-Time Requirements," IEEE Software, Sept. 1992, pp. 22-33.
[8] J. Gannon, R. Hamlet, and P. McMullin, “Data Abstraction Implementation, Specification, and Testing,” ACM Trans. Programming Languages and Systems, vol. 3, pp. 211-223, 1981.
[9] R. Hamlet, "Testing Programs with the Aid of a Compiler," IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279-290, July 1977.
[10] E.C.R. Hehner, "Predicative Programming Part 1," Comm. ACM, vol. 27, no. 2, pp. 134-143, Feb. 1984.
[11] K. Heninger, "Specifying Software Requirements for Complex Systems: New Techniques and their Applications," IEEE Trans. Software Eng, vol. 6, no. 1, pp. 2-13, Jan. 1980.
[12] D.N. Hoover and Z. Chen, "Tablewise, A Decision Table Tool," Proc. Ninth Conf. Computer Assurance, COMPASS '95, June 1995.
[13] W.E. Howden, Functional Program Testing Analysis. McGraw-Hill, 1987.
[14] H. Hörcher, “Improving Software Tests Using Z Specifications,” in Bowen and Hinchey [4], pp. 152–166,
[15] M.S. Jaffe, N.G. Leveson, M.P.E. Heimdahl, and B. Melhart, "Software Requirements Analysis for Real-Time Process-Control Systems," IEEE Trans. Software Engineering, vol. 17, no. 3, pp. 241-258, Mar. 1991.
[16] R. Janicki, "Towards a Formal Semantics of Parnas Tables," Proc. 17th Int'l Conf. Software Eng., ICSE, pp. 231-240, Apr. 1995.
[17] R. Janicki, D.L. Parnas, and J. Zucker, "Tabular Representations in Relational Documents," Brink et al. [6], ch. 12, pp. 184-196.
[18] D. C. Luckham, F. W. von Henke, B. Krieg-Brückner, and O. Owe,Anna—A Language for Annotating Ada Programs. Lecture Notes in Computer Science No. 260. New York: Springer-Verlag, 1987.
[19] E. Mikk, “Compilation of Z Specifications into C for Automatic Test Result Evaluation,” in Bowen and Hinchey [4], pp. 167–180, 1995.
[20] D.J. Panzl, "Automatic Software Test Drivers," Computer, pp. 44-50, Apr. 1978.
[21] D.L. Parnas, "A Generalized Control Structure and Its Formal Definition," Comm. ACM, vol. 26, no. 8, pp. 572-581, Aug. 1983.
[22] D.L. Parnas and Y. Wang, "The Trace Assertion Method of Module Interface Specification," TR89-261, Queen's Univ., Telecommunications Research Inst. of Ontario, TRIO, Oct. 1989.
[23] D.L. Parnas and J. Madey, “Functional Documentation for Computer Systems,” Science of Computer Programming, vol. 25, no. 1, pp. 41–61, Oct. 1995.
[24] D.L. Parnas, J. Madey, and M. Iglewski, "Precise Documentation of Well-Structured Programs," IEEE Trans. Software Eng., vol. 20, no. 12, pp. 948-976, Dec. 1994.
[25] D.L. Parnas, "Tabular Representation of Relations," CRL260, Telecommunications Research Inst. of Ontario, Nov. 1992.
[26] D.L. Parnas, “Predicate Logic for Software Engineering,” IEEE Trans Software Eng., vol. 19, no. 9 pp. 856–862 Sept. 1993.
[27] D.L. Parnas, "Mathematical Description and Specification of Software," Proc. IFIP Congress, vol. I, pp. 354-359, NorthHolland, Aug. 1994.
[28] D.L. Parnas, "A Logic for Describing, not Verifying, Software," Erkenntnis, vol. 43, no. 3, pp. 321-338, Kluwer, Nov. 1995.
[29] D.K. Peters, "Shortest Path Algorithm: Formal Program Documentation," CRL280, Telecommunications Research Inst. of Ontario, Feb. 1996, to appear.
[30] D.K. Peters, "Generating a Test Oracle from Program Documentation," MEng thesis, Dept. of Electrical and Computer Eng., McMaster Univ., Hamilton, Ontario, also printed as CRL302, Telecommunications Research Inst. of Ontario, Apr. 1995.
[31] D.J. Richardson, S.L. Aha, and T.O. O'Malley, “Specification-Based Test Oracles for Reactive Systems,” Proc. Int'l Conf. Software Eng. (ICSE), pp. 105–118, May 1992.
[32] D.S. Rosenblum, “A Practical Approach to Programming With Assertions,” IEEE Trans. Software Eng., vol. 21, no. 1, pp. 19–31, Jan. 1995.
[33] S. Sankar and R. Hayes, "Specifying and Testing Software Components Using ADL," Technical Report SMLI TR-94-23, Sun Microsystems Lab, Inc., Apr. 1994.
[34] Software Engineering Research Group, "Table Tool System Developer's Guide," CRL339, Telecommunications Research Inst. Of Ontario, Jan. 1997.
[35] P. Stocks and D. Carrington, “Test Template Framework: A Specification-Based Testing Case Study,” Proc. Int'l Symp. Software Testing and Analysis (ISSTA '93), pp. 11–18, June 1993.
[36] Y. Wang, "Specifying and Simulating the Externally Observable Behavior of Modules," PhD thesis, Dept. of Computing and Information Science, Queen's Univ., Kingston, Ontario, Canada, 1994.
[37] E.J. Weyuker, "On Testing Nontestable Programs," The Computer J., vol. 25, no. 4, pp. 465-470, 1982.

Index Terms:
Program testing, test oracle, formal specification, finite state machine.
Dennis K. Peters, David Lorge Parnas, "Using Test Oracles Generated from Program Documentation," IEEE Transactions on Software Engineering, vol. 24, no. 3, pp. 161-173, March 1998, doi:10.1109/32.667877
Usage of this product signifies your acceptance of the Terms of Use.