This Article 
 Bibliographic References 
 Add to: 
Semantics Guided Regression Test Cost Reduction
August 1997 (vol. 23 no. 8)
pp. 498-516

Abstract—Software maintainers are faced with the task of regression testing: retesting a modified program on an often large number of test cases. The cost of regression testing can be reduced if the size of the program that must be retested is reduced and if old test cases and old test results can be reused. Two complimentary algorithms for reducing the cost of regression testing are presented. The first produces a program called differences that captures the semantic change between certified, a previously tested program, and modified, a changed version of certified. It is more efficient to test differences, because it omits unchanged computations. The program differences is computed using a combination of program slices.

The second algorithm identifies test cases for which certified and modified will produce the same output and existing test cases that will test components new in modified. Not rerunning test cases that produce the same output avoids unproductive testing; testing new components with existing test cases avoids the costly construction of new test cases. The second algorithm is based on the notion of common execution patterns, which is the interprocedural extension of the notion introduced by Bates and Horwitz. Program components with common execution patterns have the same execution pattern during some call to their procedure. They are computed using a new type of interprocedural slice called a calling context slice. Whereas an interprocedural slice includes the program components necessary to capture all possible executions of a statement, a calling context slice includes only those program components necessary to capture the execution of a statement in a particular calling context (i.e., a particular call to the procedure).

Together with differences, it is possible to test modified by running the smaller program differences on a smaller number of test cases. This is more efficient than running modified on a large number of test cases. A prototype implementation has been built to examine and illustrate these algorithms.

[1] S. Bates and S. Horwitz, “Incremental Program Testing Using Program Dependence Graphs,” Proc. 20th ACM Symp. Principles of Programming Languages, Jan. 1993.
[2] P. Benedusi, A. Cimitile, and U. De Carlini, “Post-Maintenance Testing Based on Path Change Analysis,” Proc. Conf. Software Maintenance—1988, pp. 352-361, Oct. 1988.
[3] D. Binkley, "Multi-Procedure Program Integration," PhD dissertation and Technical Report TR-1038, Computer Sciences Dept., Univ. of Wisconsin, Madison, Aug. 1991.
[4] D. Binkley, "Using Semantic Differencing to Reduce the Cost of Regression Testing," Proc. Conf. Software Maintenance—1992, pp. 41-50, Nov. 1992.
[5] D. Binkley, "Precise Executable Interprocedural Slices," ACM Letters on Programming Languages and Systems, vol. 2, pp. 1-4, 1993.
[6] D. Binkley, "Interprocedural Test Case Selection," Technical Report TR95001, Dept. of Computer Science, Loyola College, Baltimore, Md., Jan. 1995.
[7] D. Binkley, S. Horwitz, and T. Reps, ”Program Integration for Language with Procedure Calls,” ACM Trans. Software Eng. and Methodology, vol. 4, no. 1, pp. 3–35, Jan. 1995
[8] D. Binkley, "Reducing the cost of Regression Testing by Semantics Guided Test Case Selection," Proc. Conf. Software Maintenance—1995, pp. 251-260, Oct. 1995.
[9] J.D. Choi and J. Ferrante, “Static Slicing in the Presence ofgotoStatements,” ACM Trans. Programming Languages and Systems, vol. 16, no. 4, pp. 1,097–1,113, July 1994.
[10] L.A. Clarke, Formal Evaluation of Data Flow Path Selection Criteria IEEE Trans. Software Eng., vol. 15, no. 11, pp. 1318-1332, Nov. 1989.
[11] R. Cytron, J. Ferrante, B.K. Rosen, M.N. Wegman, and K. Zadeck, "An Efficient Method of Computing Static Single Assignment Form," Proc. Conf. Record 16th ACM Symp. Principles of Programming Languages,Austin, pp. 25-35, Texas, Jan. 1989, New York: ACM, 1989
[12] V. Donzeau-Gouge, G. Huet, G. Kahn, and B. Lang, "Programming Environments Based on Structured Editors: The MENTOR Experience," pp. 128-140, Interactive Programming Environments, D. Barstow, E. Sandewall, and H. Shrobe, eds. New York: McGraw-Hill, 1984.
[13] K.F. Fischer, F. Raji, and A. Chruscicki, "A Methodology for Re-testing modified Software," Proc. IEEE Nat'l Telecommunications Conf., pp. B6.3.1-6, Nov. 1981.
[14] K. Gallagher and J. Lyle, “Using Program Slicing in Software Maintenance,” IEEE Trans. Software Eng., Aug. 1991, pp. 751-761.
[15] R. Gupta, M.J. Harrold, and M.L. Soffa, “An Approach to Regression Testing Using Slicing,” Proc. Conf. Software Maintenance—1992, pp. 299-308, Nov. 1992.
[16] M.J. Harrold and M.L. Soffa, “An Incremental Approach to Unit Testing During Maintenance,” Proc. Conf. Software Maintenance—1988, pp. 362-367, Oct. 1988.
[17] S. Horwitz, “Identifying the Semantic and Textual Differences Between Two Versions of a Program,“ Proc. SIGPLAN '90 Conf. Programming Language Design and Implementation, pp. 234-244, 1990.
[18] S. Horwitz, J. Prins, and T. Reps, ”Integrating Noninterfering Versions of Programs,” ACM Trans. Programming Languages and Systems, vol. 11, no. 3, pp. 345–387, July 1989.
[19] S. Horwitz, T. Reps, and D. Binkley, “Interprocedural Slicing Using Dependence Graphs,” ACM Trans. Programming Languages and Systems. vol. 12, no. 1, pp. 26-60, Jan. 1990.
[20] S. Horwitz and T. Reps, The Use of Program Dependence Graphs in Software Engineering Proc. 14th Int'l Conf. Software Eng., pp. 392-411, May 1992.
[21] W.E. Howden, "Reliability of the Path Analysis Testing Strategy," IEEE Trans. Software Eng. vol. 2, no. 3, pp. 208-215, Sept. 1976.
[22] M. Kamkar, P. Fritzson, and N. Shahmehri, "Interprocedural Dynamic Slicing Applied to Interprocedural Data Flow Testing," Proc. IEEE Conf. Software Maintenance,Montreal, pp. 368-395, Quebec, Canada, Sept. 1993, IEEE Computer Soc., Washington, D.C., 1993.
[23] J. Laski and W. Szermer, “Identification of Program Modifications and Its Applications in Software Maintenance,” Proc. Conf. Software Maintenance—1992, pp. 282-290, Nov. 1992.
[24] H.K.N. Leung and L.J. White, “A Cost Model to Compare Regression Test Strategies,” Proc. Conf. Software Maintenance—1991, pp. 201-208, Oct. 1991.
[25] J.R. Lyle, D.R. Wallace, J.R. Graham, K.B. Gallagher, J.E. Poole, and D.W. Binkley, "A CASE Tool to Evaluate Functional Diversity in High Integrity Software," U.S. Dept. of Commerce, Technology Administration, Nat'l Inst. of Standards and Technology, Computer Systems Laboratory, Gaithersburg, Md., 1995.
[26] B. Marick, "A Survey of Test Effectiveness and Cost Studies," Report No. UIUCD.C.S-R-90-1652, Dept. of Computer Science, Univ. of Illinois at Urbana-Champaign, Dec. 1990.
[27] B. Marick, "Three Ways to Improve Your Testing," Unpublished Report, Testing Foundations,Champaign Ill., 1993.
[28] D. Notkin, R.J. Ellison, B.J. Staudt, G.E. Kaiser, E. Kant, A.N. Habermann, V. Ambriola, and C. Montangero, "Special Issue on the GANDALF Project," J. Systems and Software, vol. 5, no. 2, May 1985.
[29] T.J. Ostrand and E.J. Weyuker, "Using Data Flow Analysis for Regression Testing," Proc. Sixth Ann. Pacific Northwest Software Quality Conf.,Portland, Ore. Sept.19-20, 1988, New York: ACM, 1988.
[30] K.J. Ottenstein and L.M. Ottenstein, ”The Program Dependence Graph in a Software Development Environment,” ACM SIGPLAN Notices, vol. 19, no. 5, pp. 177–184, May 1984.
[31] A. Podgurski and L.A. Clarke, "A Formal Model of Program Dependences and Its Implications for Software Testing, Debugging, and Maintenance," IEEE Trans. Software Eng., vol. 16, no. 9, pp. 965-979, Sept. 1990.
[32] S. Rapps and E.J. Weyuker, “Selecting Software Test Data Using Data Flow Information,” IEEE Trans. Software Eng., vol. 11, no. 4, pp. 367–375, Apr. 1985.
[33] T. Reps and T. Teitelbaum, The Synthesizer Generator: A System for Constructing Language-Based Editors. New York: Springer-Verlag, 1988.
[34] G. Rothermel and M.J. Harrold, "A Safe, Efficient Algorithm for Regression Test Selection," Proc. Conf. Software Maintenance—1993, pp. 358-367, Sept. 1993.
[35] G. Rothermel and M.J. Harrold, "Selecting Tests and Identifying Test Coverage Requirements for Modified Software," Proc. 1994 Int'l Symp. Software Testing and Analysis, pp. 169-184, Aug. 1994.
[36] G. Rothermel and M.J. Harrold, “Analyzing Regression Test Selection Techniques,” IEEE Trans. Software Eng., vol. 22, no. 8, pp. 529-551, Aug. 1996.
[37] J. Stoy, Denotational Semantics, the Scott-Strachey Approach to Programming Language Theory, MIT Press, 1977.
[38] M. Weiser, "Program Slicing," pp. 439-449, Proc. Fifth Int'l Conf. Software Eng.,San Diego, Calif., Mar. 1981.
[39] E.J. Weyuker, "The Complexity of Data Flow Criteria for Test Data Selection," Information Processing Letters, vol. 19, pp. 103-109, 1984.
[40] L. White, "Software Testing and Verification," Advances in Computers, vol. 26, pp. 335-390, Academic Press, 1987.
[41] W.E. Wong, J.R. Horgan, S. London, and A.P. Mathur, "Effect of Test Set Minimization on Fault Detection Effectiveness," Proc. 17th Int'l Conf. Software Eng., pp. 41-50, IEEE, 1995.
[42] W. Yang, S. Horwitz, and T. Reps, "A Program Integration Algorithm that Accommodates Semantics-Preserving Transformations," Proc. Fourth ACM SIGSOFT Symp. Software Development Environments,Irvine, Calif., ACM SIGSOFT Software Eng. Notes, Dec. 1990.
[43] W. Yang, "Identifying Syntactic Differences Between Two Programs," Software—Practice and Experience, vol. 21, no. 7, pp. 739-755, July 1991.

Index Terms:
CTR Categories and Subject Descriptors: D.2.2 [Software Engineering]: tools and techniques—programmer workbench; D.2.5 [Software Enginering]: testing and debugging; D.3.3 [Programming Languages]: language constructs—control structures, procedures, functions, and subroutines; E.1 [Data Structures] graphs; other keywords: slicing, regression testing.
David Binkley, "Semantics Guided Regression Test Cost Reduction," IEEE Transactions on Software Engineering, vol. 23, no. 8, pp. 498-516, Aug. 1997, doi:10.1109/32.624306
Usage of this product signifies your acceptance of the Terms of Use.