This Article 
 Bibliographic References 
 Add to: 
Cryptographic Verification of Test Coverage Claims
February 2000 (vol. 26 no. 2)
pp. 178-192

Abstract—The market for software components is growing, driven on the “demand side” by the need for rapid deployment of highly functional products and, on the “supply side,” by distributed object standards. As components and component vendors proliferate, there is naturally a growing concern about quality and the effectiveness of testing processes. White-box testing, particularly the use of coverage criteria, is a widely used method for measuring the “thoroughness” of testing efforts. High levels of test coverage are used as indicators of good quality control procedures. Software vendors who can demonstrate high levels of test coverage have a credible claim to high quality. However, verifying such claims involves knowledge of the source code, test cases, build procedures, etc. In applications where reliability and quality are critical, it would be desirable to verify test coverage claims without forcing vendors to give up valuable technical secrets. In this paper, we explore cryptographic techniques that can be used to verify such claims. Our techniques have certain limitations, which we discuss in this paper. However, vendors who have done the hard work of developing high levels of test coverage can use these techniques (for a modest additional cost) to provide credible evidence of high coverage, while simultaneously reducing disclosure of intellectual property.

[1] Source code for bin2chall. .
[2] The Mondex Magazine, Mondex Int'l Limited, July 1997. (See also:http:/
[3] “Guideline for Automatic Data Processing and Risk Analysis FIPS 65,” Nat'l Bureau of Standards, Gaithersburg, Md., Aug. 1979.
[4] Delta Software Testing (accredited by Danish Accreditation Authority-DANAK).
[5] H. Agrawal, “Dominators, Super Blocks and Program Coverage,” Proc., ACM Principles of Programming Language '94, 1986.
[6] R. Atkinson, “Security Architecture for the Internet Protocol. Request for Comments (Proposed Standard) rfc 1825,” Aug. 1995.
[7] T. Ball and J. Larus, “Efficient Path Profiling,” Micro '96, Proc., Dec. 1996.
[8] C. Cifuentes, “Partial Automation of an Integrated Reverse Engineering Environment for Binary Code,” Proc. Third Working Conf. Reverse Eng., 1996.
[9] C. Cifuentes and J. Gough, “Decompilation of Binary Programs,” Software Practice and Experience, July 1995.
[10] P. Devanbu, “Genoa—A Language and Front-End Independent Source Code Analyzer Generator,” Proc. 14th Int'l Conf. Software Eng., 1992.
[11] P. Devanbu, “The GEN++ page,” http://seclab.cs.ucdavis.edudevanbugenp, 1998.
[12] P. Devanbu, P.W. Fong, and S. Stubblebine, “Techniques for Trusted Software Engineering,” Proc. 20th Int'l Conf. Software Eng., 1998.
[13] J.W. Duran and S.C. Ntafos, “An Evaluation of Random Testing,” IEEE Trans. Software Eng., vol. 10, no. 4, July 1984.
[14] “Trust in Cyberspace,” F.B. Schneider, ed., Nat'l Academy Press (Computer Science and Telecomm. Board, Nat'l Research Council), 1998.
[15] P. Frankl and O. Iakounenko, “Further Empirical Studies of Test Effectiveness,” Proc. ACM SIGSOFT Sixth Symp. Foundations of Software Eng., SIGSOFT Software Eng. Notes, vol. 23, no. 6, pp. 153-162, Nov. 1998.
[16] P.G. Frankl and S.N Weiss,“An experimental comparison of the effectiveness of branch testing and data flow testing,” IEEE Transactions on Software Engineering, vol. 19, no. 8, Oct. 1993, pp. 774-787.
[17] P. Frankl and E.J. Weyuker, “An Applicable Family of Data Flow Testing Criteria,” IEEE Trans. Software Eng., vol. 14, no. 10, pp. 1,483–1,498, Oct. 1988.
[18] P.G. Frankl and E.J. Weyuker,“A formal analysis of the fault-detecting ability of testing methods,” IEEE Transactions on Software Engineering, vol. 19, no. 3, Mar. 1993, pp. 202-213,
[19] A. Freier, P. Karlton, and P. Kocher, “The SSL Protocol,” version 3.0 (internet draft), Mar. 1996.
[20] J. Gannon, personal conversation, Apr. 1997.
[21] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments on the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria,” Proc. Int'l Conf. Software Eng., pp. 191–200, May 1994.
[22] Plum Hall Inc. http:/
[23] M.J. Kearns and U.V. Vazirani, An Introduction to Computational Learning Theory. MIT Press, 1994.
[24] National Software Testing Labs, http:/
[25] Software Testing Labs, http:/
[26] J.R. Larus and E. Schnarr, "EEL: Machine Independent Executable Editing," Proc. Sigplan Programming Languages, Design, and Implementation, ACM Press, New York, 1995, pp. 291-300.
[27] J. Lidiard, “Can COTS Software Be Trusted?” Interface: Newsletter FAA Software Eng. Process Group, Nov. 1996.
[28] Y. Malaiya, N. Li, J. Bieman, R. Karcich, and B. Skibbe, “Software Test Coverage and Reliability,” technical report, Colorado State Univ., 1996.
[29] D. McIlroy, personal e-mail communication, 1996.
[30] A.J. Menezes, P.C. van Oorschot, and S.A. Vanstone, Handbook of Applied Cryptography, CRC Press, Boca Raton, Fla., 1996, pp. 543-590.
[31] P.G. Neumann, Computer Related Risks, Addison-Wesley, Reading, Mass., 1995.
[32] Committee on Application of Digital Instrumentation, Control Systems to Nuclear Power Plant Operations, and Safety. Digital Instrumentation and Control Systems in Nuclear Power Plants—Safety and Reliability Issues–Final Report, Nat'l Academy Press (Board on Energy and Environmental Systems, Nat'l Research Council), 1997.
[33] N. Ramsey and M. Fernandez, “Specifying Representations of Machine Instructions,” ACM Trans. Programming Languages and Systems, 1997.
[34] J.A. Scott, G.G. Preckshot, and J.M. Gallagher, “Using Commercial Off-the-Shelf (COTS) Software in High-Consequence Safety Systems,” Technical Report UCRL-JC-122246, Lawrence Livermore Nat'l Laboratory, 1995.
[35] A. Srivastava and A. Eustace, “Atom: A Tool for Building Customized Program Analysis Tools,” Technical Report 1994/2, DEC Western Research Labs, 1994.
[36] Applied Testing and Technology Inc. http:/
[37] J.M. Voas, “Certifying Off-the-Shelf Software Components,” Computer, vol. 31, no. 6, 1998.
[38] E.J. Weyuker, “On Testing Non-Testable Programs,” The Computer J., vol. 25, no. 4, pp. 465–470, 1982.
[39] B. Yee and D. Tygar, “Secure Coprocessors in Electronic Commerce Applications,” Proc. First USENIX Workshop Electronic Commerce, July 1995.

Index Terms:
Testing, verification, cryptography, components, safety-critical systems, trust.
Premkumar Thomas Devanbu, Stuart G. Stubblebine, "Cryptographic Verification of Test Coverage Claims," IEEE Transactions on Software Engineering, vol. 26, no. 2, pp. 178-192, Feb. 2000, doi:10.1109/32.841116
Usage of this product signifies your acceptance of the Terms of Use.