This Article 
 Bibliographic References 
 Add to: 
Empirical Studies of a Prediction Model for Regression Test Selection
March 2001 (vol. 27 no. 3)
pp. 248-263

AbstractRegression testing is an important activity that can account for a large proportion of the cost of software maintenance. One approach to reducing the cost of regression testing is to employ a selective regression testing technique that 1) chooses a subset of a test suite that was used to test the software before the modifications, then 2) uses this subset to test the modified software. Selective regression testing techniques reduce the cost of regression testing if the cost of selecting the subset from the test suite together with the cost of running the selected subset of test cases is less than the cost of rerunning the entire test suite. Rosenblum and Weyuker recently proposed coverage-based predictors for use in predicting the effectiveness of regression test selection strategies. Using the regression testing cost model of Leung and White, Rosenblum and Weyuker demonstrated the applicability of these predictors by performing a case study involving 31 versions of the KornShell. To further investigate the applicability of the Rosenblum-Weyuker (RW) predictor, additional empirical studies have been performed. The RW predictor was applied to a number of subjects, using two different selective regression testing tools, DejaVu and TestTube. These studies support two conclusions. First, they show that there is some variability in the success with which the predictors work and second, they suggest that these results can be improved by incorporating information about the distribution of modifications. It is shown how the RW prediction model can be improved to provide such an accounting.

[1] H. Agrawal, J. Horgan, E. Krauser, and S. London, “Incremental Regression Testing,” Proc. Conf. Software Maintenance—1993, pp. 348-357, Sept. 1993.
[2] M. Balcer, W. Hasling, and T. Ostrand, “Automatic Generation of Test Scripts from Formal Test Specifications,” Proc. Third Symp. Software Testing, Analysis, and Verification, pp. 210–218, Dec. 1989.
[3] T. Ball, “On the Limit of Control Flow Analysis for Regression Test Selection,” Proc. 1998 Int'l Symp. Software Testing and Analysis, Mar. 1998.
[4] S. Bates and S. Horwitz, “Incremental Program Testing Using Program Dependence Graphs,” Proc. 20th ACM Symp. Principles of Programming Languages, Jan. 1993.
[5] B. Beizer, Software Testing Techniques, second ed. Boston, Mass.: Int'l Thomson Computer Press, 1990.
[6] P. Benedusi, A. Cimitile, and U. De Carlini, “Post-Maintenance Testing Based on Path Change Analysis,” Proc. Conf. Software Maintenance—1988, pp. 352-361, Oct. 1988.
[7] D. Binkley, “Semantics Guided Regression Test Cost Reduction,” IEEE Trans. Software Eng., vol. 23, no. 9, Aug. 1997.
[8] Y.F. Chen, D.S. Rosenblum, and K.P. Vo, “TestTube: A System for Selective Regression Testing,” Proc. 16th Int'l Conf. Software Eng., pp. 211-222, May 1994.
[9] K.F. Fischer, F. Raji, and A. Chruscicki, “A Methodology for Retesting Modified Software,” Proc. National Telecomm. Conf. B-6-3, pp. 1-6, Nov. 1981.
[10] R. Gupta, M.J. Harrold, and M.L. Soffa, “An Approach to Regression Testing Using Slicing,” Proc. Conf. Software Maintenance—1992, pp. 299-308, Nov. 1992.
[11] W. Harrison and C. Cook, “Insights on Improving the Maintenance Process through Software Measurement,” Proc. Conf. Software Maintenance—1990, pp. 37-45, Nov. 1990.
[12] M.J. Harrold and M.L. Soffa, “An Incremental Approach to Unit Testing During Maintenance,” Proc. Conf. Software Maintenance—1988, pp. 362-367, Oct. 1988.
[13] J. Hartmann and D.J. Robson, “Techniques for Selective Revalidation,” IEEE Software, vol. 16, no. 1, pp. 31-38, Jan. 1990.
[14] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments on the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria,” Proc. Int'l Conf. Software Eng., pp. 191–200, May 1994.
[15] J. Laski and W. Szermer, “Identification of Program Modifications and Its Applications in Software Maintenance,” Proc. Conf. Software Maintenance—1992, pp. 282-290, Nov. 1992.
[16] J.A.N. Lee and X. He, “A Methodology for Test Selection,” The J. Systems and Software, vol. 13, no. 1, pp. 177-185, Sept. 1990.
[17] H.K.N. Leung and L.J. White, “Insights Into Regression Testing,” Proc. Conf. Software Maintenance—1989, pp. 60-69, Oct. 1989.
[18] H.K.N. Leung and L.J. White, “A Study of Integration Testing and Software Regression at the Integration Level,” Proc. Conf. Software Maintenance—1990, pp. 290-300, Nov. 1990.
[19] H.K.N. Leung and L.J. White, “A Cost Model to Compare Regression Test Strategies,” Proc. Conf. Software Maintenance—1991, pp. 201-208, Oct. 1991.
[20] T.J. Ostrand and M.J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Tests,” Comm. ACM, vol. 31, no. 6, June 1988.
[21] T.J. Ostrand and E.J. Weyuker, “Using Dataflow Analysis for Regression Testing,” Proc. Sixth Ann. Pacific Northwest Software Quality Conf., pp. 233-247, Sept. 1988.
[22] D. Rosenblum and G. Rothermel, “An Empirical Comparison of Regression Test Selection Techniques,” Proc. Int'l Workshop Empirical Studies of Software Maintenance, pp. 89-94, Oct. 1997.
[23] D.S. Rosenblum and E.J. Weyuker, “Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies,” IEEE Trans. Software Eng., vol. 23, no. 3, pp. 146-156, Mar. 1997.
[24] G. Rothermel and M.J. Harrold, “Empirical Studies of a Safe Regression Test Selection Technique,” IEEE Trans. Software Eng., vol. 24, no. 6, pp. 401-419, June 1998.
[25] G. Rothermel and M.J. Harrold, “Analyzing Regression Test Selection Techniques,” IEEE Trans. Software Eng., vol. 22, no. 8, pp. 529-551, Aug. 1996.
[26] G. Rothermel and M.J. Harrold, “A Safe, Efficient Regression Test Selection Technique,” ACM Trans. Software Eng. and Methodology, vol. 6, no. 2, pp. 173-210, Apr. 1997.
[27] B. Sherlund and B. Korel, “Logical Modification Oriented Software Testing,” Proc. 12th Int'l Conf. Testing Computer Software, June 1995.
[28] A.B. Taha, S.M. Thebaut, and S.S. Liu, “An Approach to Software Fault Localization and Revalidation Based on Incremental Data Flow Analysis,” Proc. 13th Ann. Int'l Computer Software and Applications Conf., pp. 527-534, Sept. 1989.
[29] F. Vokolos and P. Frankl, “Pythia: A Regression Test Selection Tool Based on Textual Differencing,” Proc. Third Int'l Conf. Reliability, Quality, and Safety of Software Intensive Systems (ENCRESS '97), May 1997.
[30] S.S. Yau and Z. Kishimoto, “A Method for Revalidating Modified Programs in the Maintenance Phase,” Proc. 11th Ann. Int'l Computer Software and Applications Conf. (COMPSAC '87), pp. 272-277, Oct. 1987.

Index Terms:
Software maintenance, regression testing, selective retest, regression test selection.
Mary Jean Harrold, David Rosenblum, Gregg Rothermel, Elaine Weyuker, "Empirical Studies of a Prediction Model for Regression Test Selection," IEEE Transactions on Software Engineering, vol. 27, no. 3, pp. 248-263, March 2001, doi:10.1109/32.910860
Usage of this product signifies your acceptance of the Terms of Use.