The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.05 - September/October (2010 vol.36)
pp: 593-617
Hyunsook Do , North Dakota State University, Fargo
Siavash Mirarab , IBM, Vancouver
Ladan Tahvildari , University of Waterloo, Waterloo
Gregg Rothermel , University of Nebraska—Lincoln, Lincoln
ABSTRACT
Regression testing is an expensive process used to validate modified software. Test case prioritization techniques improve the cost-effectiveness of regression testing by ordering test cases such that those that are more important are run earlier in the testing process. Many prioritization techniques have been proposed and evidence shows that they can be beneficial. It has been suggested, however, that the time constraints that can be imposed on regression testing by various software development processes can strongly affect the behavior of prioritization techniques. If this is correct, a better understanding of the effects of time constraints could lead to improved prioritization techniques and improved maintenance and testing processes. We therefore conducted a series of experiments to assess the effects of time constraints on the costs and benefits of prioritization techniques. Our first experiment manipulates time constraint levels and shows that time constraints do play a significant role in determining both the cost-effectiveness of prioritization and the relative cost-benefit trade-offs among techniques. Our second experiment replicates the first experiment, controlling for several threats to validity including numbers of faults present, and shows that the results generalize to this wider context. Our third experiment manipulates the number of faults present in programs to examine the effects of faultiness levels on prioritization and shows that faultiness level affects the relative cost-effectiveness of prioritization techniques. Taken together, these results have several implications for test engineers wishing to cost-effectively regression test their software systems. These include suggestions about when and when not to prioritize, what techniques to employ, and how differences in testing processes may relate to prioritization cost--effectiveness.
INDEX TERMS
Regression testing, test case prioritization, cost-benefits, Bayesian networks, empirical studies.
CITATION
Hyunsook Do, Siavash Mirarab, Ladan Tahvildari, Gregg Rothermel, "The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments", IEEE Transactions on Software Engineering, vol.36, no. 5, pp. 593-617, September/October 2010, doi:10.1109/TSE.2010.58
REFERENCES
[1] S. Alspaugh, K.R. Walcott, M. Belanich, G.M. Kapfhammer, and M.L. Soffa, "Efficient Time-Aware Prioritization with Knapsack Solvers," Proc. ACM Int'l Workshop Empirical Assessment of Software Eng. Languages and Technologies, pp. 17-31, Nov. 2007.
[2] J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. Int'l Conf. Software Eng., pp. 402-411, May 2005.
[3] R. Bell, T. Ostrand, and E. Weyuker, "Looking for Bugs in All the Right Places," Proc. Int'l Symp. Software Testing and Analysis, pp. 61-72, July 2006.
[4] B. Boehm, "Value-Based Software Engineering," ACM SIGSOFT Software Eng. Notes, vol. 28, no. 2, p. 4, 2003.
[5] B. Boehm, C. Abts, A.W. Brown, S. Chulani, E. Horowitz, B.K. Clark, R. Madachy, D. Reifer, and B. Steece, Software Cost Estimation with COCOMO II. Prentice-Hall, 2000.
[6] S.R. Chidamber and C.F. Kemerer, "Towards a Metrics Suite for Object Oriented Design," ACM SIGPLAN Notes, vol. 26, no. 11, pp. 197-211, Nov. 1991.
[7] C. Collberg, G. Myles, and M. Stepp, "An Empirical Study of Java Bytecode Programs," Technical Report TR04-11, Dept. of Computer Science, Univ. of Arizona, 2004.
[8] Culpepper and Assoc., Inc., "Culpepper Compensation and Benefit Surveys," http:/www.culpepper.com, 2010.
[9] H. Do, S. Elbaum, and G. Rothermel, "Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact," Empirical Software Eng.: An Int'l J., vol. 10, no. 4, pp. 405-435, 2005.
[10] H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, "An Empirical Study of the Effect of Time Constraints on the Cost-Benefits of Regression Testing," Proc. ACM SIGSOFT, pp. 71-82, Nov. 2008.
[11] H. Do and G. Rothermel, "An Empirical Study of Regression Testing Techniques Incorporating Context and Lifetime Factors and Improved Cost-Benefit Models," Proc. ACM SIGSOFT, pp. 141-151, Nov. 2006.
[12] H. Do and G. Rothermel, "On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques," IEEE Trans. Software Eng., vol. 32, no. 9, pp. 733-752, Sept. 2006.
[13] H. Do and G. Rothermel, "Using Sensitivity Analysis to Create Simplified Economic Models for Regression Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 51-61, July 2008.
[14] H. Do, G. Rothermel, and A. Kinneer, "Empirical Studies of Test Case Prioritization in a JUnit Testing Environment," Proc. Int'l Symp. Software Reliability Eng., pp. 113-124, Nov. 2004.
[15] H. Do, G. Rothermel, and A. Kinneer, "Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis," Empirical Software Eng.: An Int'l J., vol. 11, no. 1, pp. 33-70, 2006.
[16] S. Elbaum, A. Malishevsky, and G. Rothermel, "Prioritizing Test Cases for Regression Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 102-112, Aug. 2000.
[17] S. Elbaum, A. Malishevsky, and G. Rothermel, "Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization," Proc. Int'l Conf. Software Eng., pp. 329-338, May 2001.
[18] S. Elbaum, A.G. Malishevsky, and G. Rothermel, "Test Case Prioritization: A Family of Empirical Studies," IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002.
[19] B. Freimut, L.C. Briand, and F. Vollei, "Determining Inspection Cost-Effectiveness by Combining Project Data and Expert Opinion," IEEE Trans. Software Eng., vol. 31, no. 12, pp. 1074-1092, Dec. 2005.
[20] M.J. Harrold, D. Rosenblum, G. Rothermel, and E. Weyuker, "Empirical Studies of a Prediction Model for Regression Test Selection," IEEE Trans. Software Eng., vol. 27, no. 3, pp. 248-263, Mar. 2001.
[21] S. Hou, L. Zhang, T. Xie, and J. Sun, "Quota-Constrained Test Case Prioritization for Regression Testing of Service-Centric Systems," Proc. Int'l Conf. Software Maintenance, pp. 257-266, Sept. 2008.
[22] D. Jeffrey and N. Gupta, "Test Case Prioritization Using Relevant Slices," Proc. Ann. Int'l Computer Software and Applications Conf., pp. 411-420, Sept. 2006.
[23] R. Johnson, Elementary Statistics, sixth ed. Duxbury Press, 1992.
[24] C. Jones, Applied Software Measurement: Assuring Productivity and Quality. McGraw-Hill, 1997.
[25] J. Jones and M.J. Harrold, "Test Suite Reduction and Prioritization for Modified Condition/Decision Coverage," IEEE Trans. Software Eng., vol. 29, no. 3, pp. 193-209, Mar. 2003.
[26] J. Kim and A. Porter, "A History-Based Test Prioritization Technique for Regression Testing in Resource Constrained Environments," Proc. Int'l Conf. Software Eng., pp. 119-129, May 2002.
[27] A. Kinneer, M. Dwyer, and G. Rothermel, "Sofya: A Flexible Framework for Development of Dynamic Program Analysis for Java Software," Technical Report TR-UNL-CSE-2006-0006, Univ. of Nebraska–Lincoln, Apr. 2006.
[28] B. Korel, G. Koutsogiannakis, and L. Tahat, "Application of System Models in Regression Test Suite Prioritization," Proc. Int'l Conf. Software Maintenance, pp. 247-256, Sept. 2008.
[29] B. Korel, L. Tahat, and M. Harman, "Test Prioritization Using System Models," Proc. Int'l Conf. Software Maintenance, pp. 559-568, Sept. 2005.
[30] S. Kusumoto, K. Matsumoto, T. Kikuno, and K. Torii, "A New Metric for Cost-Effectiveness of Software Reviews," IEICE Trans. Information Systems, vol. E75-D no. 5, pp. 674-680, 1992.
[31] D. Leon and A. Podgurski, "A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases," Proc. Int'l Symp. Software Reliability Eng., pp. 442-453, Nov. 2003.
[32] H.K.N. Leung and L.J. White, "A Cost Model to Compare Regression Test Strategies," Proc. Int'l Conf. Software Maintenance, pp. 201-208, Oct. 1991.
[33] Z. Li, M. Harman, and R.M. Hierons, "Search Algorithms for Regression Test Case Prioritization," IEEE Trans. Software Eng., vol. 33, no. 4, pp. 225-237, Apr. 2007.
[34] A. Malishevsky, G. Rothermel, and S. Elbaum, "Modeling the Cost-Benefits Tradeoffs for Regression Testing Techniques," Proc. Int'l Conf. Software Maintenance, pp. 204-213, Oct. 2002.
[35] S. Mirarab, "A Bayesian Framework for Software Regression Testing," master's thesis, Dept. of Electrical and Computer Eng., Univ. of Waterloo, Aug. 2008.
[36] S. Mirarab and L. Tahvildari, "A Prioritization Approach for Software Test Cases on Bayesian Networks," Proc. Int'l Conf. Fundamental Approaches to Software Eng. , pp. 276-290, Mar. 2007.
[37] S. Mirarab and L. Tahvildari, "An Empirical Study on Bayesian Network-Based Approach for Test Case Prioritization," Proc. Int'l Conf. Software Testing, Verification, and Validation, pp. 278-287, Apr. 2008.
[38] M.M. Muller and F. Padberg, "About the Return on Investment of Test-Driven Development," Proc. Int'l Workshop Economics-Driven Software Eng. Research, pp. 2631-2636, May 2003.
[39] K. Onoma, W.-T. Tsai, M. Poonawala, and H. Suganuma, "Regression Testing in an Industrial Environment," Comm. ACM, vol. 41, no. 5, pp. 81-86, 1988.
[40] T.J. Ostrand and M.J. Balcer, "The Category-Partition Method for Specifying and Generating Functional Tests," Comm. ACM, vol. 31, no. 6, pp. 676-688, 1988.
[41] H. Park, J. Ryu, and J. Baik, "Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing," Proc. Int'l Conf. Secure System Integration and Reliability Improvement, pp. 39-46, July 2008.
[42] D.E. Perry and C.S. Stieg, "Software Faults in Evolving a Large, Real-Time System: A Case Study," Proc. European Software Eng. Conf., pp. 48-67, Sept. 1993.
[43] J.J. Phillips, Return on Investment in Training and Performance Improvement Programs. Gulf Publishing Company, 1997.
[44] X. Qu, M.B. Cohen, and K.M. Woolf, "Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization," Proc. Int'l Conf. Software Maintenance, pp. 255-264, Oct. 2007.
[45] F.L. Ramsey and D.W. Schafer, The Statistical Sleuth. Duxbury Press, 1997.
[46] G. Rothermel and M.J. Harrold, "Analyzing Regression Test Selection Techniques," IEEE Trans. Software Eng., vol. 22, no. 8, pp. 529-551, Aug. 1996.
[47] G. Rothermel and M.J. Harrold, "A Safe, Efficient Regression Test Selection Technique," ACM Trans. Software Eng. and Methodology, vol. 6, no. 2, pp. 173-210, 1997.
[48] G. Rothermel, R. Untch, C. Chu, and M.J. Harrold, "Test Case Prioritization," IEEE Trans. Software Eng., vol. 27, no. 10, pp. 929-948, Oct. 2001.
[49] G. Rothermel, R.H. Untch, C. Chu, and M.J. Harrold, "Test Case Prioritization: An Empirical Study," Proc. Int'l Conf. Software Maintenance, pp. 179-188, Aug. 1999.
[50] A. Saltelli, S. Tarantola, F. Campolongo, and M. Ratto, Sensitivity Analysis in Practice. John Wiley, 2004.
[51] S. Sampath, R. Bryce, G. Viswanath, V. Kandimalla, and A. Koru, "Prioritizing User-Session-Based Test Cases for Web Applications Testing," Proc. Int'l Conf. Software Testing, Verification, and Validation, pp. 141-150, Apr. 2008.
[52] M. Sherriff, M. Lake, and L. Williams, "Prioritization of Regression Tests Using Singular Value Decomposition with Empirical Change Records," Proc. Int'l Symp. Software Reliability Eng., pp. 81-90, Nov. 2007.
[53] F. Shull, V. Basili, B. Boehm, A.W. Brown, P. Costa, M. Lindvall, D. Port, I. Rus, R. Tesoriero, and M. Zelkowitz, "What We Have Learned about Fighting Defects," Proc. Int'l Software Metrics Symp., pp. 249-258, June 2002.
[54] R. Solingen, "Measuring the ROI of Software Process Improvement," IEEE Software, vol. 21, no. 3, pp. 32-38, May/June 2004.
[55] A. Srivastava and J. Thiagarajan, "Effectively Prioritizing Tests in Development Environmentm," ACM SIGSOFT Software Eng. Notes, vol. 27, no. 4, pp. 97-106, 2002.
[56] T. Ostrand, E. Weyuker, and R. Bell, "Predicting the Location and Number of Faults in Large Software Systems," IEEE Trans. Software Eng., vol. 31, no. 4, pp. 340-355, Apr. 2005.
[57] S. Wagner, "A Model and Sensitivity Analysis of the Quality Economic of Defect-Detection Techniques," Proc. Int'l Symp. Software Testing and Analysis, pp. 73-84, July 2006.
[58] S. Wagner, "An Approach to Global Sensitivity Analysis: FAST on COCOMO," Proc. Int'l Symp. Empirical Software Eng. and Measurement, pp. 440-442, Sept. 2007.
[59] A. Walcott, M.L. Soffa, G.M. Kapfhammer, and R.S. Roos, "Time-Aware Test Suite Prioritization," Proc. Int'l Symp. Software Testing and Analysis, pp. 1-12, July 2006.
[60] W.E. Wong, J.R. Horgan, S. London, and H. Agrawal, "A Study of Effective Regression Testing in Practice," Proc. Int'l Symp. Software Reliability Eng., pp. 264-274, Nov. 1997.
[61] S. Yoo, M. Harman, P. Tonella, and A. Susi, "Clustering Test Cases to Achieve Effective and Scalable Prioritisation Incorporating Expert Knowledge," Proc. Int'l Symp. Software Testing and Analysis, pp. 201-212, July 2009.
[62] L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei, "Time-Aware Test-Case Prioritization Using Integer Linear Programming," Proc. Int'l Symp. Software Testing and Analysis, pp. 213-224, July 2009.
20 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool