This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
An Analysis of Test Data Selection Criteria Using the RELAY Model of Fault Detection
June 1993 (vol. 19 no. 6)
pp. 533-553

RELAY is a model of faults and failures that defines failure conditions, which describe test data for which execution will guarantee that a fault originates erroneous behavior that also transfers through computations and information flow until a failure is revealed. This model of fault detection provides a framework within which other testing criteria's capabilities can be evaluated. Three test data selection criteria that detect faults in six fault classes are analyzed. This analysis shows that none of these criteria is capable of guaranteeing detection for these fault classes and points out two major weaknesses of these criteria. The first weakness is that the criteria do not consider the potential unsatisfiability of their rules. Each criterion includes rules that are sufficient to cause potential failures for some fault classes, yet when such rules are unsatisfiable, many faults may remain undetected. Their second weakness is failure to integrate their proposed rules.

[1] T. A. Budd, "Mutation analysis: Ideas, examples, problems, and prospects," inComputer Program Testing, B. Chandrasekaran and S. Radicchi, Eds. Amsterdam: North-Holland, 1981, pp. 129-148.
[2] T. A. Budd, "The portable mutation testing suite," Tech. Rep. TR 83-8, Univ. Arizona, Mar. 1983.
[3] T. A. Budd and D. Angluin, "Two notions of correctness and their relation to testing," Tech. Rep. TR 80-19, Univ. Arizona, 1980.
[4] L. A. Clarke, J. Hassell, and D. J. Richardson, "A close look at domain testing,"IEEE Trans. Software Eng., vol. SE-8, no. 4, July 1982.
[5] L. A. Clarke, A. Podgurski, D. J. Richardson, and S. J. Zeil, "A comparison of data flow path selection criteria," inProc. 8th Inf. Conf. Sofware Eng.(London UK), Aug. 1985. Los Alamitos, CA: IEEE Computer Soc., 1985, pp. 244-251.
[6] L. A. Clarke, "A formal evaluation of data flow path selection criteria,"IEEE Trans. Software Eng., vol. SE-15, no. 11, Nov. 1989.
[7] R. DeMillo, R. Lipton, and F. Sayward, "Hints on test data selection: Help for the practicing programmer,"Computer, vol. 4, no. 11, Apr. 1978.
[8] R. DeMillo, R. Lipton, and F. Sayward, "Program mutation: A new approach to program testing," Tech. Rep. v. 2, Infotech International, 1979.
[9] R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J. Offutt, "An extended overview of the Mothra software testing environment," inProc. Second Workshop Software Testing, Verification and Analysis, Banff, Alberta, July 1988, pp. 142-151.
[10] R. A. DeMillo and A. J. Offutt, "Constraint-based automatic test data generation,"IEEE Trans. Software Eng., vol. 17, no. 9, pp. 900-910, Sept. 1991.
[11] D. E. Denning and P. J. Denning, "Certification of programs for secure information flow,"Commun. ACM, vol. 20, no. 7, pp. 504- 513, July 1977.
[12] J. Ferrante, K. Ottenstein, and J. Warren, "The program dependence graph and its use in optimization,"ACM Trans. Program. Lang. Syst., vol. 9, no. 3, pp. 319-349, July 1987.
[13] K. A. Foster, "Error sensitive test case analysis (ESTCA),"IEEE Trans. Software Eng., vol. SE-6, no. 3, pp. 258-264, May 1980.
[14] K. A. Foster, "Comment on the application of error-sensitive testing strategies to debugging,"ACM Software Eng. Notes, vol. 8, no. 5, pp. 40-42, Oct. 1983.
[15] K. A. Foster, "Sensitive test data for logical expressions,"ACM Software Eng. Notes, vol. 9, no. 3, July 1984.
[16] K. A. Foster, "Revision of an error sensitive test rule,"ACM Software Eng. Notes, vol. 10, no. 1, Jan. 1985.
[17] R. L. Glass, "Persistent software errors,"IEEE Trans. Software Eng., vol. SE-7, no. 2, pp. 162-168, Mar. 1981.
[18] J. B. Goodenough and S. L. Gerhart, "Toward a theory of test data selection,"IEEE Trans. Software Eng., vol. SE-1, no. 2, pp. 156-173, June 1975.
[19] R. G. Hamlet, "Testing programs with the aid of a compiler,"IEEE Trans. Software Eng., vol. SE-3, no. 4, pp. 279-290, July 1977.
[20] S. Horwitz, J. Prins, and T. Reps, "On the adequacy of program dependence graphs for representing programs," inProc. Fifteenth ACM Symp. Principles of Programming Languages, ACM, New York, 1988, pp. 146-157.
[21] W. E. Howden, "Reliability of the path analysis testing strategy,"IEEE Trans. Software Eng., vol. SE-2, no. 3, Sept. 1976.
[22] W. E. Howden, "Introduction to the theory of testing," inTutorial: Software Testing and Validation Techniques, E. Miller and W. E. Howden, Eds. New York: IEEE, 1978, pp. 16-19.
[23] W. E. Howden, "Weak mutation testing and completeness of test sets,"IEEE Trans. Software Eng., vol. SE-8, no. 2, pp. 371-379, July 1982.
[24] W. E. Howden, "The theory and practice of functional testing,"IEEE Software, vol. 2, no. 5, pp. 6-17, Sept. 1985.
[25] W. E. Howden, "A functional approach to program testing and analysis,"IEEE Trans. Software Eng., vol. SE-12, no. 10, pp. 997-1005, Oct. 1986.
[26] W. Howden,Functional Program Testing and Analysis. New York: Mc-Graw-Hill, 1987.
[27] W. E. Howden and P. Eichhorst, "Proving properties of programs from program traces," inTutorial: Software Testing and Validation Techniques, E. Miller and W. E. Howden, Eds. New York, IEEE, 1978, pp. 16-19.
[28] J. W. Laski and B. Korel, "A data flow oriented program testing strategy,"IEEE Trans. Software Eng., vol. SE-9, no. 3, pp. 347-354, May 1983.
[29] D. C. Luckham and F. W. von Henke, "An overview of ANNA, a specification language for Ada,"IEEE Trans. Software Eng., vol. 2, no. 2, pp. 9-22, Mar. 1985.
[30] L. J. Morell, "A theory of error-based testing," Ph.D. dissertation, Dep. Comput. Sci., Univ. Maryland, Tech. Rep. TR-1395, Aug. 1984.
[31] L. J. Morell, "Theoretical insights into fault-based testing," inProc. 2nd Workshop on Software Testing, Verification, and Analysis(Banff, AB, Can.), July 1988. Los Alamitos, CA: IEEE Computer Soc., 1988, pp. 45-62.
[32] L. J. Morell, "A theory of fault-based testing,"IEEE Trans. Software Eng., vol. 16, no. 8, pp. 844-857, Aug. 1990.
[33] G. J. Myers,The Art of Software Testing. New York: Wiley, 1979.
[34] S. C. Ntafos, "On required element testing,"IEEE Trans. Software Eng., vol. SE-10, no. 6, pp. 795-803, Nov. 1984.
[35] A. Podgurski and L. A. Clarke, "A formal model of program dependences and its implications for software testing, debugging, and maintenance,"IEEE Trans. Software Eng., vol. 16, no. 9, pp. 965-979, Sept. 1990.
[36] A. Podgurski, "The significance of program dependences for software testing, debugging, and maintenance," Doctoral dissertation, Dep. Comput. Inform. Sci., Univ. Massachusetts, Amherst, 1989.
[37] S. Rapps and E. J. Weyuker, "Selecting software test data using data flow information,"IEEE Trans. Software Eng., vol. SE-11, no. 4, pp. 367-375, Apr. 1985.
[38] D. J. Richardson, S. L. Aha, and T. O. O'Malley, "Specification-based test oracles for reactive systems," inProc. Fourteenth Int. Conf. Software Eng., Melbourne, Australia, May 1992.
[39] D. J. Richardson, O. O'Malley, and C. Tittle, "Approaches to specification-based testing," inProc. ACM SIGSOFT '89 Third Symp. Software Testing, Analysis, and Verification (TAV3), ACM SIGSOFT, Key West, FL, Dec. 1989, pp. 86-96.
[40] D. J. Richardson and M. C. Thompson, "A formal framework for test data selection criteria," Tech. Rep. 86-56, Comput. and Inform. Sci., Univ. Massachusetts, Amherst, Nov. 1986.
[41] D. J. Richardson and M. C. Thompson, "An analysis of test data selection criteria using the RELAY model of error detection," Tech. Rep. 86-65, Comput. and Information Science, University of Massachusetts, Amherst, Dec. 1986.
[42] D. J. Richardson and M. C. Thompson, "RELAY: A new model of error detection," Tech. Rep. 86-64, Comput. and Inform. Sci., Univ. Massachusetts, Amherst, Dec. 1986.
[43] D. J. Richardson and M. C. Thompson, "The RELAY model of error detection and its application," inProc. Second Workshop Software Testing, Verification, and Analysis. IEEE Computer Society, Los Angeles, CA, 1988.
[44] M. C. Thompson, "An investigation of fault-based testing using the RELAY model," Ph.D. dissertation, Dep. Comput. and Information Science, Univ. Massachusetts, Amherst, May 1991.
[45] M. C. Thompson, D. J. Richardson, and L. A. Clarke, "Information flow transfer in the RELAY model," inProc. ACM SIGSOFT Int. Symp. Software Testing and Analysis (ISSTA '93), Cambridge, MA, June 28-30, 1993, ACM Press.
[46] E. J. Weyuker, "An error-based testing strategy," Tech. Rep. 027, Computer Science, Institute of Mathematical Sciences, New York Univ., Jan. 1981.
[47] E. J. Weyuker, "On testing nontestable programs,"Comput. J, vol. 25, no. 4, 1982.
[48] L. J. White and E. I. Cohen, "A domain strategy for computer program testing.IEEE Trans. Software Eng., vol. SE-6, no. 3, pp. 247-257, May 1980.
[49] S. J. Zeil, "Testing for perturbations of program statements,"IEEE Trans. Software Engineering, vol. SE-9, no. 3, pp. 335-346, May 1983.
[50] S. J. Zeil, "Perturbation testing for computation errors," inProc. Seventh Int. Conf. Software Eng., Mar. 1984.
[51] S. J. Zeil, "Perturbation techniques for detecting domain errors,"IEEE Trans. Software Eng., vol. 15, no. 6, pp. 737-746, June 1989.

Index Terms:
test data selection criteria; fault detection; RELAY; failure conditions; erroneous behavior; information flow; program debugging; program testing; software reliability
Citation:
D.J. Richardson, M.C. Thompson, "An Analysis of Test Data Selection Criteria Using the RELAY Model of Fault Detection," IEEE Transactions on Software Engineering, vol. 19, no. 6, pp. 533-553, June 1993, doi:10.1109/32.232020
Usage of this product signifies your acceptance of the Terms of Use.