This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Prioritizing Test Cases For Regression Testing
October 2001 (vol. 27 no. 10)
pp. 929-948
ASCII Text
x 
 
Gregg Rothermel, Roland H. Untch, Chengyun Chu, Mary Jean Harrold, "Prioritizing Test Cases For Regression Testing," IEEE Transactions on Software Engineering, vol. 27, no. 10, pp. 929-948, October, 2001.
 
 
BibTex
x
 
@article{ 10.1109/32.962562,
author = {Gregg Rothermel and Roland H. Untch and Chengyun Chu and Mary Jean Harrold},
title = {Prioritizing Test Cases For Regression Testing},
journal ={IEEE Transactions on Software Engineering},
volume = {27},
number = {10},
issn = {0098-5589},
year = {2001},
pages = {929-948},
doi = {http://doi.ieeecomputersociety.org/10.1109/32.962562},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - Prioritizing Test Cases For Regression Testing
IS - 10
SN - 0098-5589
SP929
EP948
EPD - 929-948
A1 - Gregg Rothermel,
A1 - Roland H. Untch,
A1 - Chengyun Chu,
A1 - Mary Jean Harrold,
PY - 2001
KW - Test case prioritization
KW - regression testing
KW - software testing
KW - empirical studies.
VL - 27
JA - IEEE Transactions on Software Engineering
ER -
 

Abstract—Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. Various goals are possible; one involves rate of fault detection—a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible. One application of prioritization techniques involves regression testing—the retesting of software following modifications; in this context, prioritization techniques can take advantage of information gathered about the previous execution of test cases to obtain test case orderings. In this paper, we describe several techniques for using test execution information to prioritize test cases for regression testing, including: 1) techniques that order test cases based on their total coverage of code components, 2) techniques that order test cases based on their coverage of code components not previously covered, and 3) techniques that order test cases based on their estimated ability to reveal faults in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites for various programs and measured the rates of fault detection achieved by the prioritized test suites, comparing those rates to the rates achieved by untreated, randomly ordered, and optimally ordered suites. Analysis of the data shows that each of the prioritization techniques studied improved the rate of fault detection of test suites, and this improvement occurred even with the least expensive of those techniques. The data also shows, however, that considerable room remains for improvement. The studies highlight several cost-benefit trade-offs among the techniques studied, as well as several opportunities for future work.

[1] A. Avritzer and E.J. Weyuker, “The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software,” IEEE Trans. Software Eng., vol. 21, no. 9, pp. 705–716, Sept. 1995.
[2] M. Balcer, W. Hasling, and T. Ostrand, “Automatic Generation of Test Scripts from Formal Test Specifications,” Proc. Third Symp. Software Testing, Analysis, and Verification, pp. 210–218, Dec. 1989.
[3] T. Ball, “On the Limit of Control Flow Analysis for Regression Test Selection,” Proc. 1998 Int'l Symp. Software Testing and Analysis, Mar. 1998.
[4] B. Beizer, Software Testing Techniques, second ed. Boston, Mass.: Int'l Thomson Computer Press, 1990.
[5] D. Binkley, “Semantics Guided Regression Test Cost Reduction,” IEEE Trans. Software Eng., vol. 23, no. 9, Aug. 1997.
[6] T.Y. Chen and M.F. Lau, “Dividing Strategies for the Optimization of a Test Suite,” Information Processing Letters, vol. 60, no. 3, pp. 135–141, Mar. 1996.
[7] Y.F. Chen, D.S. Rosenblum, and K.P. Vo, “TestTube: A System for Selective Regression Testing,” Proc. 16th Int'l Conf. Software Eng., pp. 211-222, May 1994.
[8] M.E. Delamaro and J.C. Maldonado, “Proteum—A Tool for the Assessment of Test Adequacy for C Programs,” Proc. Conf. Performability in Computing Systems (PCS '96), pp. 79–95, July 1996.
[9] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, vol. 11, no. 4, pp. 34–41, Apr. 1978.
[10] R.J. Freund and R.C. Littell, SAS for Linear Models: A Guide to the ANOVA and GLM Procedures. Cary, N.C.: SAS Institute Inc., 1981.
[11] M.R. Garey and D.S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness.New York: W.H. Freeman, 1979.
[12] T. Goradia, “Dynamic Impact Analysis: A Cost-Effective Technique to Enforce Error-Propagation,” Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 171–181, June 1993.
[13] R.G. Hamlet, “Testing Programs with the Aid of a Compiler,” IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279–290, July 1977.
[14] R.G. Hamlet, “Probable Correctness Theory,” Information Processing Letters, vol. 25, pp. 17–25, Apr. 1987.
[15] M.J. Harrold, R. Gupta, and M.L. Soffa, "A Methodology for Controlling the Size of a Test Suite," ACM Trans. Software Eng. and Methodology, vol. 2, no. 3, pp. 270-285, July 1993.
[16] M.J. Harrold and G. Rothermel, “Aristotle: A System for Research on and Development of Program Analysis Based Tools,” Technical Report OSU-CISRC-3/97-TR17, The Ohio State Univ., Mar. 1997.
[17] J.T. Helwig, SAS Introductory Guide. SAS Institute, Inc., 1978.
[18] W.E. Howden, “Weak Mutation Testing and Completeness of Test Sets,” IEEE Trans. Software Eng., vol. 8, no. 7, pp. 371–379, July 1982.
[19] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments on the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria,” Proc. Int'l Conf. Software Eng., pp. 191–200, May 1994.
[20] H.K.N. Leung and L.J. White, “Insights Into Regression Testing,” Proc. Conf. Software Maintenance—1989, pp. 60-69, Oct. 1989.
[21] H.K.N. Leung and L.J. White, “A Study of Integration Testing and Software Regression at the Integration Level,” Proc. Conf. Software Maintenance—1990, pp. 290-300, Nov. 1990.
[22] J.C. Munson and T.M. Khoshgoftaar, "The Detection of Fault-Prone Programs," IEEE Trans. Software Eng., vol. 18, May 1992.
[23] A.J. Offutt, A. Lee, G. Rothermel, R. Untch, and C. Zapf, “An Experimental Determination of Sufficient Mutation Operators,” ACM Trans. Software Eng. Methods, vol. 5, no. 2, pp. 99–118, Apr. 1996.
[24] K. Onoma, W-T. Tsai, M. Poonawala, and H. Suganuma, “Regression Testing in an Industrial Environment,” Comm. ACM, vol. 41, no. 5, pp. 81–86, May 1988.
[25] T.J. Ostrand and M.J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Tests,” Comm. ACM, vol. 31, no. 6, June 1988.
[26] L. Ott, An Introduction to Statistical Methods and Data Analysis, third ed. Boston, Mass.: PWS-Kent Publishing Company, 1988.
[27] S. Rapps and E.J. Weyuker, “Selecting Software Test Data Using Data Flow Information,” IEEE Trans. Software Eng., vol. 11, no. 4, pp. 367–375, Apr. 1985.
[28] G. Rothermel and M.J. Harrold, “Analyzing Regression Test Selection Techniques,” IEEE Trans. Software Eng., vol. 22, no. 8, pp. 529-551, Aug. 1996.
[29] G. Rothermel and M.J. Harrold, “A Safe, Efficient Regression Test Selection Technique,” ACM Trans. Software Eng. and Methodology, vol. 6, no. 2, pp. 173-210, Apr. 1997.
[30] G. Rothermel, M.J. Harrold, J. Ostrin, and C. Hong, Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites Proc. Int'l Conf. Software Maintenance, pp. 34-43, Nov. 1998.
[31] G. Rothermel, R.H. Untch, C. Chu, and M.J. Harrold, “Test Case Prioritization: An Empirical Study,” Proc. Int'l Conf. Software Maintenence, pp. 179–188, Aug. 1999.
[32] M.C. Thompson, D.J. Richardson, and L.A. Clarke, “An Information Flow Model of Fault Detection,” Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 182–192, June 1993.
[33] J. Voas, “PIE: A Dynamic Failure-Based Technique,” IEEE Trans. Software Eng., vol. 18, no. 8, pp. 717–727, Aug. 1992.
[34] F. Vokolos and P. Frankl, “Pythia: A Regression Test Selection Tool Based on Textual Differencing,” Proc. Third Int'l Conf. Reliability, Quality, and Safety of Software Intensive Systems (ENCRESS '97), May 1997.
[35] F.I. Vokolos and P.G. Frankl, “Empirical Evaluation of the Textual Differencing Regression Testing Technique,” Proc. Int'l Conf. Software Maintenence, pp. 44–53, Nov. 1998.
[36] W.E. Wong, J.R. Horgan, S. London, and H. Agrawal, “A Study of Effective Regression Testing in Practice,” Proc. Eighth Int'l Symp. Software Reliability Eng., pp. 230–238 Nov. 1997.
[37] W.E. Wong et al., "Effect of Test Set Minimization on Fault Detection Effectiveness," Software—Practice and Experience, Apr. 1998, pp. 347-369.
[38] W.E. Wong, J.R. Horgan, A.P. Mathur, and A. Pasquini, “Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application,” Proc. 21st Ann. Int'l Computer Software&Applications Conf., pp. 522–528, Aug. 1997.

Index Terms:
Test case prioritization, regression testing, software testing, empirical studies.
Citation:
Gregg Rothermel, Roland H. Untch, Chengyun Chu, Mary Jean Harrold, "Prioritizing Test Cases For Regression Testing," IEEE Transactions on Software Engineering, vol. 27, no. 10, pp. 929-948, Oct. 2001, doi:10.1109/32.962562
Usage of this product signifies your acceptance of the Terms of Use.