This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques
September 2006 (vol. 32 no. 9)
pp. 733-752
Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite's rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite's rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. More importantly, a comparison of our results with those collected using hand-seeded faults reveals several implications for researchers performing empirical studies of test case prioritization techniques in particular and testing techniques in general.

[1] J.H. Andrews, L.C. Briand, and Y. Labiche, “Is Mutation an Appropriate Tool for Testing Experiments,” Proc. Int'l Conf. Software Eng., pp. 402-411, May 2005.
[2] http:/ant.apache.org, 2004.
[3] http://jakarta.apache.orgbcel, 2004.
[4] J.M. Bieman, S. Ghosh, and R.T. Alexander, “A Technique for Mutation of Java Objects,” Proc. Int'l Conf. Automated Software Eng., pp. 337-340, Nov. 2001.
[5] T.A. Budd, “Mutation Analysis of Program Test Data,” PhD dissertation, Yale Univ., 1980.
[6] R. Coe, “What Is an Effect Size?” CEM Centre, Durham Univ., Mar. 2000.
[7] R.A. Demillo, R.J. Lipton, and F.G. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, pp. 34-41, 1978.
[8] H. Do, S. Elbaum, and G. Rothermel, “Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact,” Empirical Software Eng.: An Int'l J., vol. 10, no. 4, pp. 405-435, 2005.
[9] H. Do, G. Rothermel, and A. Kinneer, “Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis,” Empirical Software Eng.: An Int'l J., vol. 11, no.1, pp. 33-70, Mar. 2006.
[10] S. Elbaum, D. Gable, and G. Rothermel, “Understanding and Measuring the Sources of Variation in the Prioritization of Regression Test Suites,” Proc. Int'l Software Metrics Symp., pp.169-179, Apr. 2001.
[11] S. Elbaum, P. Kallakuri, A. Malishevsky, G. Rothermel, and S. Kanduri, “Understanding the Effects of Changes on the Cost-Effectiveness of Regression Testing Techniques,” J. Software Testing, Verification, and Reliability, vol. 12, no. 2, pp. 65-83, 2003.
[12] S. Elbaum, A. Malishevsky, and G. Rothermel, “Prioritizing Test Cases for Regression Testing,” Proc. Int'l Symp. Software Testing and Analysis, pp. 102-112, Aug. 2000.
[13] S. Elbaum, A. Malishevsky, and G. Rothermel, “Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization,” Proc. Int'l Conf. Software Eng., pp. 329-338, May 2001.
[14] S. Elbaum, A.G. Malishevsky, and G. Rothermel, “Test Case Prioritization: A Family of Empirical Studies,” IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002.
[15] P.G. Frankl, S.N. Weiss, and C. Hu, “All-Uses versus Mutation Testing: An Experimental Comparison of Effectiveness,” J. Systems and Software, vol. 38, no. 3, pp. 235-253, 1997.
[16] R.G. Hamlet, “Testing Programs with the Aid of a Compiler,” IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279-290, 1977.
[17] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, “Experiments on the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria,” Proc. Int'l Conf. Software Eng., pp. 191-200, May 1994.
[18] http://jakarta.apache.orgjmeter, 2004.
[19] http://xml.apache.orgsecurity, 2004.
[20] http:/www.junit.org, 2004.
[21] J. Kim and A. Porter, “A History-Based Test Prioritization Technique for Regression Testing in Resource Constrained Environments,” Proc. Int'l Conf. Software Eng., pp. 119-129, May 2002.
[22] J.-M. Kim, A. Porter, and G. Rothermel, “An Empirical Study of Regression Test Application Frequency,” Proc. Int'l Conf. Software Eng., pp. 126-135, June 2000.
[23] S. Kim, J.A. Clark, and J.A. McDermid, “Class Mutation: Mutation Testing for Object-Oriented Programs,” Proc. Net.ObjectDays Conf. Object-Oriented Software Systems, Oct. 2000.
[24] S. Kim, J.A. Clark, and J.A. McDermid, “Investigating the Effectiveness of Object-Oriented Testing Strategies with the Mutation Method,” J. Software Testing, Verification, and Reliability, vol. 11, no. 4, pp. 207-225, 2001.
[25] A. Kinneer, “Assessing the Cost-Benefits of Using Type Inference Algorithms to Improve the Representation of Exceptional Control Flow in Java,” MS thesis, Univ. of Nebraska-Lincoln, Aug. 2005.
[26] D. Leon and A. Podgurski, “A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases,” Proc. Int'l Symp. Software Reliability Eng., pp. 442-453, Nov. 2003.
[27] C. Lindsey, J. Tolliver, and T. Lindblad, JavaTech: An Introduction to Scientific and Technical Computing with Java. Cambridge Univ. Press, 2005.
[28] Y. Ma, Y. Kwon, and J. Offutt, “Inter-Class Mutation Operators for Java,” Proc. Int'l Symp. Software Reliability Engineering, pp. 352-363, Nov. 2002.
[29] C. Murphy and B. Myors, Statistical Power Analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests. Lawrence Erlbaum Associates, 1998.
[30] A.J. Offutt, J. Pan, K. Tewary, and T. Zhang, “An Experimental Evaluation of Data Flow and Mutation Testing,” Software-Practice and Experience, vol. 26, no. 2, pp. 165-176, Feb. 1996.
[31] K. Onoma, W-T. Tsai, M. Poonawala, and H. Suganuma, “Regression Testing in an Industrial Environment,” Comm. ACM, vol. 41, no. 5, pp. 81-86, May 1988.
[32] T.J. Ostrand and M.J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Tests,” Comm. ACM, vol. 31, no. 6, pp. 676-686, June 1988.
[33] G. Rothermel, S. Elbaum, A. Malishevsky, P. Kallakuri, and B. Davia, “The Impact of Test Suite Granularity on the Cost-Effectiveness of Regression Testing,” Proc. Int'l Conf. Software Eng., pp. 230-240, May 2002.
[34] G. Rothermel, S. Elbaum, A.G. Malishevsky, P. Kallakuri, and X. Qiu, “On Test Suite Composition and Cost-Effective Regression Testing,” ACM Trans. Software Eng. and Methodology, vol. 13, no. 3, pp. 227-331, July 2004.
[35] G. Rothermel and M.J. Harrold, “Analyzing Regression Test Selection Techniques,” IEEE Trans. Software Eng., vol. 22, no. 8, pp.529-551, Aug. 1996.
[36] G. Rothermel, R. Untch, C. Chu, and M.J. Harrold, “Prioritizing Test Cases for Regression Testing,” IEEE Trans. Software Eng., vol. 27, no. 10, pp. 929-948, Oct. 2001.
[37] http://www.insightful.com/productssplus, 2004.
[38] A. Srivastava and J. Thiagarajan, “Effectively Prioritizing Tests in Development Environment,” Proc. Int'l Symp. Software Testing and Analysis, pp. 97-106, July 2002.
[39] C. Wohlin, P. Runeson, M. Host, M. Ohlsoon, B. Regnell, and A. Wesslen, Experimentation in Software Engineering: An Introduction. Kluwer Academic, 2000.
[40] W.E. Wong, J.R. Horgan, S. London, and H. Agrawal, “A Study of Effective Regression Testing in Practice,” Proc. Int'l Symp. Software Reliability Eng., pp. 230-238, Nov. 1997.
[41] http://xml.apache.orgsecurity, 2004.

Index Terms:
Regression testing, test case prioritization, program mutation, empirical studies.
Citation:
Hyunsook Do, Gregg Rothermel, "On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques," IEEE Transactions on Software Engineering, vol. 32, no. 9, pp. 733-752, Sept. 2006, doi:10.1109/TSE.2006.92
Usage of this product signifies your acceptance of the Terms of Use.