This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Size-Constrained Regression Test Case Selection Using Multicriteria Optimization
July-Aug. 2012 (vol. 38 no. 4)
pp. 936-956
ASCII Text
x 
 
Siavash Mirarab, Soroush Akhlaghi, Ladan Tahvildari, "Size-Constrained Regression Test Case Selection Using Multicriteria Optimization," IEEE Transactions on Software Engineering, vol. 38, no. 4, pp. 936-956, July-Aug., 2012.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2011.56,
author = {Siavash Mirarab and Soroush Akhlaghi and Ladan Tahvildari},
title = {Size-Constrained Regression Test Case Selection Using Multicriteria Optimization},
journal ={IEEE Transactions on Software Engineering},
volume = {38},
number = {4},
issn = {0098-5589},
year = {2012},
pages = {936-956},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2011.56},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - Size-Constrained Regression Test Case Selection Using Multicriteria Optimization
IS - 4
SN - 0098-5589
SP936
EP956
EPD - 936-956
A1 - Siavash Mirarab,
A1 - Soroush Akhlaghi,
A1 - Ladan Tahvildari,
PY - 2012
KW - Software regression testing
KW - test case selection
KW - integer programming
KW - Pareto optimality
VL - 38
JA - IEEE Transactions on Software Engineering
ER -
 
Siavash Mirarab, University of Texas at Austin, Austin
Soroush Akhlaghi, Shahed University, Tehran
Ladan Tahvildari, University of Waterloo, Waterloo
To ensure that a modified software system has not regressed, one approach is to rerun existing test cases. However, this is a potentially costly task. To mitigate the costs, the testing effort can be optimized by executing only a selected subset of the test cases that are believed to have a better chance of revealing faults. This paper proposes a novel approach for selecting and ordering a predetermined number of test cases from an existing test suite. Our approach forms an Integer Linear Programming problem using two different coverage-based criteria, and uses constraint relaxation to find many close-to-optimal solution points. These points are then combined to obtain a final solution using a voting mechanism. The selected subset of test cases is then prioritized using a greedy algorithm that maximizes minimum coverage in an iterative manner. The proposed approach has been empirically evaluated and the results show significant improvements over existing approaches for some cases and comparable results for the rest. Moreover, our approach provides more consistency compared to existing approaches.

[1] H.K.N. Leung and L.J. White, "Insights into Regression Testing," Proc. Int'l Conf. Software Maintenance, pp. 60-69, 1989.
[2] K. Fischer, "A Test Case Selection Method for the Validation of Software Maintenance Modifications," Proc. Int'l Computer Software and Applications Conf., pp. 421-426, 1977.
[3] K. Fischer, F. Raji, and A. Chruscicki, "A Methodology for Retesting Modified Software," Proc. Nat'l Telecomm. Conf., pp. B6.3.1-B6.3.5, 1981.
[4] J. Hartmann and D.J. Robson, "Techniques for Selective Revalidation," IEEE Software, vol. 7, no. 1, pp. 31-36, Jan. 1990.
[5] Y.-F. Chen, D.S. Rosenblum, and K.-P. Vo, "Testtube: A System for Selective Regression Testing," Proc. ACM Int'l Conf. Software Eng., pp. 211-220, 1994.
[6] G. Rothermel and M.J. Harrold, "A Safe, Efficient Regression Test Selection Technique," ACM Trans. Software Eng. and Methodology, vol. 6, no. 2, pp. 173-210, 1997.
[7] M. Harrold and M. Soffa, "An Incremental Approach to Unit Testing during Maintenance," Proc. Int'l Conf. Software Maintenance, pp. 362-367, 1988.
[8] L. White and H. Leung, "A Firewall Concept for Both Control-Flow and Data-Flow in Regression Integration Testing," Proc. Int'l Conf. Software Maintenance, pp. 262-271, 1992.
[9] D. Binkley, "Semantics Guided Regression Test Cost Reduction," IEEE Trans. Software Eng., vol. 23, no. 8, pp. 498-516, Aug. 1997.
[10] L. Briand, Y. Labiche, and G. Soccar, "Automating Impact Analysis and Regression Test Selection Based on UML Designs," Proc. Int'l Conf. Software Maintenance, pp. 252-261, 2002.
[11] S. Yoo and M. Harman, "Pareto Efficient Multi-Objective Test Case Selection," Proc. Int'l Symp. Software Testing and Analysis, pp. 140-150, 2007.
[12] W.E. Wong, J.R. Horgan, S. London, and H.A. Bellcore, "A Study of Effective Regression Testing in Practice," Proc. Int'l Symp. Software Reliability Eng., pp. 264-274, 1997.
[13] S. Elbaum, A.G. Malishevsky, and G. Rothermel, "Test Case Prioritization: A Family of Empirical Studies," IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002.
[14] G. Rothermel, R.H. Untch, C. Chu, and M.J. Harrold, "Prioritizing Test Cases for Regression Testing," IEEE Trans. Software Eng., vol. 27, no. 10, pp. 929-948, Oct. 2001.
[15] A. Srivastava and J. Thiagarajan, "Effectively Prioritizing Tests in Development Environment," Proc. Int'l Symp. Software Testing and Analysis, pp. 97-106, 2002.
[16] J.-M. Kim and A. Porter, "A History-Based Test Prioritization Technique for Regression Testing in Resource Constrained Environments," Proc. ACM Int'l Conf. Software Eng., pp. 119-129, 2002.
[17] D. Saff and M.D. Ernst, "Reducing Wasted Development Time via Continuous Testing," Proc. Int'l Symp. Software Reliability Eng., pp. 281-292, 2003.
[18] D. Leon and A. Podgurski, "A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases," Proc. Int'l Symp. Software Reliability Eng., pp. 442-453, 2003.
[19] W. Dickinson, D. Leon, and A. Podgurski, "Finding Failures by Cluster Analysis of Execution Profiles," Proc. ACM Int'l Conf. Software Eng., pp. 339-348, 2001.
[20] X. Zhang, C. Nie, B. Xu, and B. Qu, "Test Case Prioritization Based on Varying Testing Requirement Priorities and Test Case Costs," Proc. Int'l Conf. Quality Software, pp. 15-24, 2007.
[21] S. Alspaugh, K.R. Walcott, M. Belanich, G.M. Kapfhammer, and M.L. Soffa, "Efficient Time-Aware Prioritization with Knapsack Solvers," Proc. ACM Int'l Workshop Empirical Assessment of Software Eng. Languages and Technologies, 2007.
[22] K.R. Walcott, M.L. Soffa, G.M. Kapfhammer, and R.S. Roos, "Timeaware Test Suite Prioritization," Proc. Int'l Symp. Software Testing and Analysis, pp. 1-12, 2006.
[23] A.M. Smith and G.M. Kapfhammer, "An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization," Proc. 24th Symp. Applied Computing, 2009.
[24] Z. Li, M. Harman, and R.M. Hierons, "Search Algorithms for Regression Test Case Prioritization," IEEE Trans. Software Eng., vol. 33, no. 4, pp. 225-237, Apr. 2007.
[25] S. Mirarab and L. Tahvildari, "A Prioritization Approach for Software Test Cases on Bayesian Networks," Proc. Int'l Conf. Fundamental Approaches to Software Eng., pp. 276-290, 2007.
[26] D. Jeffrey and N. Gupta, "Test Case Prioritization Using Relevant Slices," Proc. Int'l Computer Software and Applications Conf., pp. 411-420, 2006.
[27] H. Do, G. Rothermel, and A. Kinneer, "Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis," Empirical Software Eng.: An Int'l J., vol. 11, no. 1, pp. 33-70, 2006.
[28] H. Do and G. Rothermel, "An Empirical Study of Regression Testing Techniques Incorporating Context and Lifetime Factors and Improved Cost-Benefit Models," Proc. Int'l Symp. Foundations of Software Eng., pp. 141-151, 2006.
[29] H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, "An Empirical Study of the Effect of Time Constraints on the Cost-Benefits of Regression Testing," Proc. ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 71-82, 2008.
[30] T.L. Graves, M.J. Harrold, J.-M. Kim, A. Porter, and G. Rothermel, "An Empirical Study of Regression Test Selection Techniques," ACM Trans. Software Eng. and Methodology, vol. 10, no. 2, pp. 184-208, 2001.
[31] H. Do, S. Elbaum, and G. Rothermel, "Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact," Empirical Software Eng.: An Int'l J., vol. 10, no. 4, pp. 405-435, 2005.
[32] X. Qu, M.B. Cohen, and G. Rothermel, "Configuration-Aware Regression Testing: An Empirical Study of Sampling and Prioritization," Proc. Int'l Symp. Software Testing and Analysis, pp. 75-86, 2008.
[33] S. Elbaum, G. Rothermel, S. Kanduri, and A.G. Malishevsky, "Selecting a Cost-Effective Test Case Prioritization Technique," Software Quality Control, vol. 12, no. 3, pp. 185-210, 2004.
[34] L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei, "Time-Aware Test-Case Prioritization Using Integer Linear Programming," Proc. Int'l Symp. Software Testing and Analysis, pp. 213-224, 2009.
[35] K. Okumoto and A.L. Goel, "Optimum Release Time for Software Systems Based on Reliability and Cost Criteria." J. Systems and Software, vol. 1, no. 4, pp. 315-318, 1980.
[36] P. Tannenbaum, Excursions in Modern Mathematics. Prentice Hall, 2006.
[37] H. Do and G. Rothermel, "On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques," IEEE Trans. Software Eng., vol. 32, no. 9, pp. 733-752, Sept. 2006.
[38] R.S. Arnold, Software Change Impact Analysis. IEEE CS Press, 1996.
[39] L.V.S.P. Boyd, Convex Optimization. Cambridge Univ. Press, 2004.
[40] G.L. Nemhauser and L.A. Wolsey, Integer and Combinatorial Optimization. Wiley, 1999.
[41] D. Greer and G. Ruhe, "Software Release Planning: An Evolutionary and Iterative Approach," Information and Software Technology, vol. 46, no. 4, pp. 243-253, 2004.
[42] S. Mirarab and L. Tahvildari, "An Empirical Study on Bayesian Network-Based Approach for Test Case Prioritization," Proc. Int'l Conf. Software Testing, Verification and Validation, pp. 278-287, 2008.
[43] T. Ostrand and M.J. Balcer, "The Category-Partition Method for Specifying and Generating Functional Tests," Comm. ACM, vol. 31, no. 6, pp. 676-688, 1988.
[44] J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. Int'l Conf. Software Eng., pp. 402-411, May 2005.
[45] A. Kinneer, M. Dwyer, and G. Rothermel, "Sofya: A Flexible Framework for Development of Dynamic Program Analysis for Java Software," Technical Report TR-UNL-CSE-2006-0006, Univ. of Nebraska-Lincoln, Apr. 2006.
[46] M.S. Christian Collberg, G. Myles, "An Empirical Study of Java Bytecode Programs," Technical Report TR04-11, Dept. of Computer Science, Univ. of Arizona, 2004.
[47] D. Chen and R.G. Batson, Applied Integer Programming: Modeling and Solution. Wiley, 2009.
[48] J. Black, E. Melachrinoudis, and D. Kaeli, "Bi-Criteria Models for All-Uses Test Suite Reduction," Proc. ACM Int'l Conf. Software Eng., pp. 106-115, 2004.
[49] G. Rothermel and M.J. Harrold, "Analyzing Regression Test Selection Techniques," IEEE Trans. Software Eng., vol. 22, no. 8, pp. 529-551, Aug. 1996.
[50] H. Leung and L. White, "A Cost Model to Compare Regression Test Strategies," Proc. Int'l Conf. Software Maintenance, pp. 201-208, 1991.
[51] A.G. Malishevsky, G. Rothermel, and S. Elbaum, "Modeling the Cost-Benefits Tradeoffs for Regression Testing Techniques," Proc. Int'l Conf. Software Maintenance, pp. 204-213, 2002.
[52] J.-W. Lin and C.-Y. Huang, "Analysis of Test Suite Reduction with Enhanced Tie-Breaking Techniques," Information and Software Technology, vol. 51, no. 4, pp. 679-690, 2009.
[53] G.M. Kapfhammer and M.L. Soffa, "Using Coverage Effectiveness to Evaluate Test Suite Prioritizations," Proc. ACM Workshop Empirical Assessment of Software Eng. Languages and Technologies, 2007.
[54] M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. Future of Software Eng., pp. 342-357, 2007.
[55] Y. Zhang, A. Finkelstein, and M. Harman, "Search Based Requirements Optimisation: Existing Work and Challenges," Proc. Int'l Conf. Requirements Eng.: Foundation for Software Quality, vol. 5025, pp. 88-94, 2008.
[56] P. Baker, M. Harman, K. Steinhöfel, and A. Skaliotis, "Search Based Approaches to Component Selection and Prioritization for the Next Release Problem," Proc. Int'l Conf. Software Maintenance, pp. 176-185, 2006.
[57] G. Antoniol, M. Di Penta, and M. Harman, "Search-Based Techniques Applied to Optimization of Project Planning for a Massive Maintenance Project," Proc. Int'l Conf. Software Maintenance, pp. 240-249, 2005.
[58] P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
[59] G. Canfora, M. Di Penta, R. Esposito, and M.L. Villani, "An Approach for QoS-Aware Service Composition Based on Genetic Algorithms," Proc. Conf. Genetic and Evolutionary Computation, pp. 1069-1075, 2005.
[60] L. Zeng, B. Benatallah, A.H.H. Ngu, M. Dumas, J. Kalagnanam, and H. Chang, "QoS-Aware Middleware for Web Services Composition," IEEE Trans. Software Eng., vol. 30, no. 5, pp. 311-327, May 2004.
[61] C. Li, M. van den Akker, S. Brinkkemper, and G. Diepen, "Integrated Requirement Selection and Scheduling for the Release Planning of a Software Product," Proc. Int'l Conf. Requirements Eng.: Foundation for Software Quality, vol. 4542, pp. 93-108, 2007.

Index Terms:
Software regression testing, test case selection, integer programming, Pareto optimality
Citation:
Siavash Mirarab, Soroush Akhlaghi, Ladan Tahvildari, "Size-Constrained Regression Test Case Selection Using Multicriteria Optimization," IEEE Transactions on Software Engineering, vol. 38, no. 4, pp. 936-956, July-Aug. 2012, doi:10.1109/TSE.2011.56
Usage of this product signifies your acceptance of the Terms of Use.