This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Empirical Studies of a Safe Regression Test Selection Technique
June 1998 (vol. 24 no. 6)
pp. 401-419

Abstract—Regression testing is an expensive testing procedure utilized to validate modified software. Regression test selection techniques attempt to reduce the cost of regression testing by selecting a subset of a program's existing test suite. Safe regression test selection techniques select subsets that, under certain well-defined conditions, exclude no tests (from the original test suite) that if executed would reveal faults in the modified software. Many regression test selection techniques, including several safe techniques, have been proposed, but few have been subjected to empirical validation. This paper reports empirical studies on a particular safe regression test selection technique, in which the technique is compared to the alternative regression testing strategy of running all tests. The results indicate that safe regression test selection can be cost-effective, but that its costs and benefits vary widely based on a number of factors. In particular, test suite design can significantly affect the effectiveness of test selection, and coverage-based test suites may provide test selection results superior to those provided by test suites that are not coverage-based.

[1] H. Agrawal, J. Horgan, E. Krauser, and S. London, “Incremental Regression Testing,” Proc. Conf. Software Maintenance—1993, pp. 348-357, Sept. 1993.
[2] M. Balcer, W. Hasling, and T. Ostrand, “Automatic Generation of Test Scripts from Formal Test Specifications,” Proc. Third Symp. Software Testing, Analysis, and Verification, pp. 210–218, Dec. 1989.
[3] T. Ball, “On the Limit of Control Flow Analysis for Regression Test Selection,” Proc. 1998 Int'l Symp. Software Testing and Analysis, Mar. 1998.
[4] S. Bates and S. Horwitz, “Incremental Program Testing Using Program Dependence Graphs,” Proc. 20th ACM Symp. Principles of Programming Languages, Jan. 1993.
[5] P. Benedusi, A. Cimitile, and U. De Carlini, “Post-Maintenance Testing Based on Path Change Analysis,” Proc. Conf. Software Maintenance—1988, pp. 352-361, Oct. 1988.
[6] D. Binkley, “Semantics Guided Regression Test Cost Reduction,” IEEE Trans. Software Eng., vol. 23, no. 9, Aug. 1997.
[7] D. Binkley, "Reducing the cost of Regression Testing by Semantics Guided Test Case Selection," Proc. Conf. Software Maintenance—1995, pp. 251-260, Oct. 1995.
[8] Y.F. Chen, D.S. Rosenblum, and K.P. Vo, “TestTube: A System for Selective Regression Testing,” Proc. 16th Int'l Conf. Software Eng., pp. 211-222, May 1994.
[9] K.F. Fischer, "A Test Case Selection Method for the Validation of Software Maintenance Modification," Proc. COMPSAC'77, pp. 421-426, Nov. 1977.
[10] K.F. Fischer, F. Raji, and A. Chruscicki, "A Methodology for Retesting Modified Software," Proc. Nat'l Telecommunications Conf. B-6-3, pp. 1-6, Nov. 1981.
[11] T.L. Graves, M.J. Harrold, J-M. Kim, A. Porter, and G. Rothermel, "An Empirical Study of Regression Test Selection Techniques," Proc. 20th Int'l Conf. Software Eng., Apr. 1998.
[12] R. Gupta, M.J. Harrold, and M.L. Soffa, “An Approach to Regression Testing Using Slicing,” Proc. Conf. Software Maintenance—1992, pp. 299-308, Nov. 1992.
[13] M.J. Harrold, D.S. Rosenblum, G. Rothermel, and E.J. Weyuker, "Empirical Studies of a Prediction Model for Regression Test Selection," Technical Report OSU-CISRC-2/98-TR55, Ohio State Univ., Feb. 1998.
[14] M.J. Harrold and G. Rothermel, "Aristotle: A System for Research on and Development of Program-Analysis-Based Tools," Technical Report OSU-CISRC-3/97-TR17, Ohio State Univ., Mar. 1997.
[15] M.J. Harrold and M.L. Soffa, “An Incremental Approach to Unit Testing During Maintenance,” Proc. Conf. Software Maintenance—1988, pp. 362-367, Oct. 1988.
[16] M.J. Harrold and M.L. Soffa, "An Incremental Data Flow Testing Tool," Proc. Sixth Int'l Conf. Testing Computer Software, May 1989.
[17] J. Hartmann and D.J. Robson, "RETEST—Development of a Selective Revalidation Prototype Environment for Use in Software Maintenance," Proc. 23rd Hawaii Int'l Conf. System Sciences, pp. 92-101, Jan. 1990.
[18] J. Hartmann and D.J. Robson, “Techniques for Selective Revalidation,” IEEE Software, vol. 16, no. 1, pp. 31-38, Jan. 1990.
[19] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments on the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria,” Proc. Int'l Conf. Software Eng., pp. 191–200, May 1994.
[20] R. Johnson, Elementary Statistics, sixth edition. Belmont, Calif.: Duxbury Press, 1992.
[21] J. Laski and W. Szermer, “Identification of Program Modifications and Its Applications in Software Maintenance,” Proc. Conf. Software Maintenance—1992, pp. 282-290, Nov. 1992.
[22] J.A.N. Lee and X. He, “A Methodology for Test Selection,” The J. Systems and Software, vol. 13, no. 1, pp. 177-185, Sept. 1990.
[23] H.K.N. Leung and L.J. White, “Insights Into Regression Testing,” Proc. Conf. Software Maintenance—1989, pp. 60-69, Oct. 1989.
[24] H.K.N. Leung and L.J. White, "Insights into Testing and Regression Testing Global Variables," J. Software Maintenance, vol. 2, pp. 209-222, Dec. 1990.
[25] H.K.N. Leung and L.J. White, “A Study of Integration Testing and Software Regression at the Integration Level,” Proc. Conf. Software Maintenance—1990, pp. 290-300, Nov. 1990.
[26] H.K.N. Leung and L.J. White, “A Cost Model to Compare Regression Test Strategies,” Proc. Conf. Software Maintenance—1991, pp. 201-208, Oct. 1991.
[27] T.J. Ostrand and M.J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Tests,” Comm. ACM, vol. 31, no. 6, June 1988.
[28] T.J. Ostrand and E.J. Weyuker, "Using Dataflow Analysis for Regression Testing," Proc. Sixth Ann. Pacific Northwest Software Quality Conf., pp. 233-247, Sept. 1988.
[29] D.S. Rosenblum and G. Rothermel, "An Empirical Comparison of Regression Test Selection Techniques," Proc. Int'l Workshop for Empirical Studies of Software Maintenance, pp. 89-94, Oct. 1997.
[30] D.S. Rosenblum and E.J. Weyuker, “Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies,” IEEE Trans. Software Eng., vol. 23, no. 3, pp. 146-156, Mar. 1997.
[31] G. Rothermal, "Efficient, Effective Reqression Testing Using Safe Test Selection Techniques," Technical Report 96-101, Clemson Univ., Jan. 1996.
[32] G. Rothermel and M.J. Harrold, "A Safe, Efficient Algorithm for Regression Test Selection," Proc. Conf. Software Maintenance—1993, pp. 358-367, Sept. 1993.
[33] G. Rothermel and M.J. Harrold, “Analyzing Regression Test Selection Techniques,” IEEE Trans. Software Eng., vol. 22, no. 8, pp. 529-551, Aug. 1996.
[34] G. Rothermel and M.J. Harrold, “A Safe, Efficient Regression Test Selection Technique,” ACM Trans. Software Eng. and Methodology, vol. 6, no. 2, pp. 173-210, Apr. 1997.
[35] B. Eherlund and B. Korel, "Modification Oriented Software Testing," Conf. Proc.: Quality Week, pp. 1-17, 1991.
[36] B. Sherlund and B. Korel, "Logical Modification Oriented Software Testing," Proc. 12th Int'l Conf. Testing Computer Software, June 1995.
[37] A.B. Taha, S.M. Thebaut, and S.S. Liu, “An Approach to Software Fault Localization and Revalidation Based on Incremental Data Flow Analysis,” Proc. 13th Ann. Int'l Computer Software and Applications Conf., pp. 527-534, Sept. 1989.
[38] F. Vokolos and P. Frankl, “Pythia: A Regression Test Selection Tool Based on Textual Differencing,” Proc. Third Int'l Conf. Reliability, Quality, and Safety of Software Intensive Systems (ENCRESS '97), May 1997.
[39] L.J. White and H.K.N. Leung, "A Firewall Concept for Both Control-Flow and Data-Flow in Regression Integration Testing," Proc. Conf. Software Maintenance—1992, pp. 262-270, Nov. 1992.
[40] L.J. White, V. Narayanswamy, T. Friedman, M. Kirschenbaum, P. Piwowarski, and M. Oha, "Test Manager: A Regression Testing Tool," Proc. Conf. Software Maintenance—1993, pp. 338-347, Sept. 1993.
[41] S.S. Yau and Z. Kishimoto, "A Method for Revalidating Modified Programs in the Maintenance Phase," COMPSAC'87: Proc. 11th Ann. Int'l Computer Software and Applications Conf., pp. 272-277, Oct. 1987.

Index Terms:
Software maintenance, regression testing, selective retest, regression test selection, empirical studies.
Citation:
Gregg Rothermel, Mary Jean Harrold, "Empirical Studies of a Safe Regression Test Selection Technique," IEEE Transactions on Software Engineering, vol. 24, no. 6, pp. 401-419, June 1998, doi:10.1109/32.689399
Usage of this product signifies your acceptance of the Terms of Use.