The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - Oct. (2013 vol.39)
pp: 1326-1344
Sreedevi Sampath , University of Maryland, Baltimore
Renee Bryce , University of North Texas, Denton
Atif M. Memon , University of Maryland, Baltimore
ABSTRACT
Regression testing tasks of test case prioritization, test suite reduction/minimization, and regression test selection are typically centered around criteria that are based on code coverage, test execution costs, and code modifications. Researchers have developed and evaluated new individual criteria; others have combined existing criteria in different ways to form what we--and some others--call hybrid criteria. In this paper, we formalize the notion of combining multiple criteria into a hybrid. Our goal is to create a uniform representation of such combinations so that they can be described unambiguously and shared among researchers. We envision that such sharing will allow researchers to implement, study, extend, and evaluate the hybrids using a common set of techniques and tools. We precisely formulate three hybrid combinations, Rank, Merge, and Choice, and demonstrate their usefulness in two ways. First, we recast, in terms of our formulations, others' previously reported work on hybrid criteria. Second, we use our previous results on test case prioritization to create and evaluate new hybrid criteria. Our findings suggest that hybrid criteria of others can be described using our Merge and Rank formulations, and that the hybrid criteria we developed most often outperformed their constituent individual criteria.
INDEX TERMS
Testing, Fault detection, Educational institutions, Genetic algorithms, Vectors, Loss measurement, Minimization,GUI testing, Test case prioritization, test criteria, hybrid test criteria, web testing
CITATION
Sreedevi Sampath, Renee Bryce, Atif M. Memon, "A Uniform Representation of Hybrid Criteria for Regression Testing", IEEE Transactions on Software Engineering, vol.39, no. 10, pp. 1326-1344, Oct. 2013, doi:10.1109/TSE.2013.16
REFERENCES
[1] S. Alspaugh, K.R. Walcott, M. Belanich, G.M. Kapfhammer, and M.L. Soffa, "Efficient Time-Aware Prioritization with Knapsack Solvers," Proc. First ACM Int'l Workshop Empirical Assessment of Software Eng. Languages and Technologies: Held in Conjunction with the 22nd IEEE/ACM Int'l Conf. Automated Software Eng., pp. 13-18, 2007.
[2] J. Black, E. Melachrinoudis, and D. Kaeli, "Bi-Criteria Models for All-Uses Test Suite Reduction," Proc. Int'l Conf. Software Eng., pp. 106-115, May 2004.
[3] R. Bryce, S. Sampath, and A. Memon, "Developing a Single Model and Test Prioritization Strategies for Event-Driven Software," IEEE Trans. Software Eng., vol. 37, no. 1, pp. 48-64, Jan./Feb. 2011.
[4] R. Carlson, H. Do, and A. Denton, "A Clustering Approach to Improving Test Case Prioritization: An Industrial Case Study," Proc. 27th IEEE Int'l Conf. Software Maintenance, pp. 382-391, Sept. 2011.
[5] T.Y. Chen and M.F. Lau, "Dividing Strategies for the Optimization of a Test Suite," Information Processing Letters, vol. 60, no. 3, pp. 135-141, Mar. 1996.
[6] T.Y. Chen and M.F. Lau, "A New Heuristic for Test Suite Reduction," Information and Software Technology, vol. 40, nos. 5/6, pp. 347-354, Sept. 1998.
[7] A.D. Lucia, M.D Penta, R. Oliveto, and A. Panichella, "On the Role of Diversity Measures for Multi-Objective Test Case Selection," Proc. Seventh Int'l Workshop Automation of Software Test, pp. 145-151, June 2012.
[8] L.S. de Souza, P.B.C. de Miranda, R.B.C. Prudencio, and F.A. de Barros, "A Multi-Objective Particle Swarm Optimization for Test Case Selection Based on Functional Requirements Coverage and Execution Effort," Proc. 23rd IEEE Int'l Conf. Tools with Artificial Intelligence, pp. 245-252, Nov. 2011.
[9] H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, "The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments," IEEE Trans. Software Eng., vol. 36, no. 5, pp. 593-617, Oct./Nov. 2010.
[10] H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, "An Empirical Study of the Effect of Time Constraints on the Cost-Benefits of Regression Testing," Proc. 16th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 71-82, 2008.
[11] C. Donghua and Y. Wenjie, "The Research of Test-Suite Reduction Technique," Proc. Int'l Conf. Consumer Electronics, Comm. and Networks, pp. 4552-4554, Apr. 2011.
[12] S. Elbaum, D. Gable, and G. Rothermel, "Understanding and Measuring the Sources of Variation in the Prioritization of Regression Test Suites," Proc. Int'l Software Metrics Symp., pp. 169-179, Apr. 2001.
[13] S. Elbaum, A. Malishevsky, and G. Rothermel, "Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization," Proc. 23rd Int'l Conf. Software Eng., pp. 329-338, 2001.
[14] S. Elbaum, A.G. Malishevsky, and G. Rothermel, "Prioritizing Test Cases for Regression Testing," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 102-112, 2000.
[15] S. Elbaum, A.G. Malishevsky, and G. Rothermel, "Test Case Prioritization: A Family of Empirical Studies," IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002.
[16] S. Elbaum, G. Rothermel, S. Kanduri, and A. Malishevsky, "Selecting a Cost-Effective Test Case Prioritization Technique," Software Quality J., vol. 12, no. 3, pp. 185-210, Sept. 2004.
[17] Y. Fazlalizadeh, A. Khalilian, M.A. Azgomi, and S. Parsa, "Prioritizing Test Cases for Resource Constraint Environments Using Historical Test Case Performance Data," Proc. Second IEEE Int'l Conf. Computer Science and Information Technology, pp. 190-195, Aug. 2009.
[18] Q. Gu, B. Tang, and D. Chen, "Optimal Regression Testing Based on Selective Coverage of Test Requirements," Proc. Int'l Symp. Parallel and Distributed Processing with Applications, pp. 419-426, Sept. 2010.
[19] R.A. Haraty, N. Mansour, and B. Daou, "Regression Testing of Database Applications," Proc. ACM Symp. Applied Computing, pp. 285-289, 2001.
[20] M. Harman, "Making the Case for MORTO: Multi Objective Regression Test Optimization," Proc. First Int'l Workshop Regression Testing, pp. 111-114, 2011.
[21] M.J. Harrold, R. Gupta, and M.L. Soffa, "A Methodology for Controlling the Size of a Test Suite," ACM Trans. Software Eng. and Methodology, vol. 2, no. 3, pp. 270-285, July 1993.
[22] M.P.E. Heimdahl and D. George, "Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing," Proc. Int'l Conf. Automated Software Eng., pp. 176-185, Sept. 2004.
[23] K.H.S. Hla, Y. Choi, and J.S. Park, "Applying Particle Swarm Optimization to Prioritizing Test Cases for Embedded Real Time Software Retesting," Proc. Eighth IEEE Int'l Conf. Computer and Information Technology Workshops, pp. 527-532, July 2008.
[24] S.-S. Hou, L. Zhang, T. Xie, and J.-S. Sun, "Quota-Constrained Test-Case Prioritization for Regression Testing of Service-Centric Systems," Proc. IEEE Int'l Conf. Software Maintenance, pp. 257-266, Oct. 2008.
[25] H.-Y. Hsu and A. Orso, "MINTS: A General Framework and Tool for Supporting Test-Suite Minimization," Proc. 31st IEEE and ACM SIGSOFT Int'l Conf. Software Eng., pp. 419-429, May 2009.
[26] Y.-C. Huang, C.-Y. Huang, J.-R. Chang, and T.-Y. Chen, "Design and Analysis of Cost-Cognizant Test Case Prioritization Using Genetic Algorithm with Test History," Proc. 34th IEEE Ann. Computer Software and Applications Conf. pp. 413-418, July 2010.
[27] M.M. Islam, A. Marchetto, A. Susi, and G. Scanniello, "A Multi-Objective Technique to Prioritize Test Cases Based on Latent Semantic Indexing," Proc. 16th European Conf. Software Maintenance and Reeng., pp. 21-30, Mar. 2012.
[28] D. Jeffrey and N. Gupta, "Test Suite Reduction with Selective Redundancy," Proc. IEEE Int'l Conf. Software Maintenance, pp. 549-558, 2005.
[29] D. Jeffrey and N. Gupta, "Test Case Prioritization Using Relevant Slices," Proc. Int'l Computer Software and Applications Conf., pp. 411-418, Sept. 2006.
[30] D. Jeffrey and N. Gupta, "Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction," IEEE Trans. Software Eng., vol. 33, no. 2, pp. 108-123, Feb. 2007.
[31] J.A. Jones and M.J. Harrold, "Test Suite Reduction and Prioritization for Modified Condition/Decision Coverage," IEEE Trans. Software Eng., vol. 29, no. 3, pp. 195-209, Mar. 2003.
[32] R. Kavitha, V.R. Kavitha, and N.S. Kumar, "Requirement Based Test Case Prioritization," Proc. IEEE Int'l Conf. Comm. Control and Computing Technologies, pp. 826-829, Oct. 2010.
[33] J.M. Kim and A. Porter, "A History-Based Test Prioritization Technique for Regression Testing in Resource Constrained Environments," Proc. Int'l Conf. Software Eng., pp. 119-129, May 2002.
[34] S. Kim and J. Baik, "An Effective Fault Aware Test Case Prioritization by Incorporating a Fault Localization Technique," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng. and Measurement, pp. 5:1-5:10, 2010.
[35] N. Koochakzadeh, V. Garousi, and F. Maurer, "Test Redundancy Measurement Based on Coverage Information: Evaluations and Lessons Learned," Proc. Int'l Conf. Software Testing Verification and Validation, pp. 220-229, Apr. 2009.
[36] J. Lin and C. Huang, "Analysis of Test Suite Reduction with Enhanced Tie-Breaking Techniques," Information and Software Technology, vol. 51, no. 11, pp. 679-690, 2008.
[37] C. Malz and P. Gohner, "Agent-Based Test Case Prioritization," Proc. IEEE Fourth Int'l Conf. Software Testing, Verification and Validation Workshops, pp. 149-152, Mar. 2011.
[38] W. Masri and M. El-Ghali, "Test Case Filtering and Prioritization Based on Coverage of Combinations of Program Elements," Proc. Seventh Int'l Workshop Dynamic Analysis, pp. 29-34, 2009.
[39] S. McMaster and A.M. Memon, "Call-Stack Coverage for GUI Test-Suite Reduction," IEEE Trans. Software Eng., vol. 34, no. 1, pp. 99-115, Jan. 2008.
[40] S. Mirarab and L. Tahvildari, "An Empirical Study on Bayesian Network-Based Approach for Test Case Prioritization," Proc. First Int'l Conf. Software Testing, Verification, and Validation, pp. 278-287, Apr. 2008.
[41] S. Mirarab, S. Akhlaghi, and L. Tahvildari, "Size-Constrained Regression Test Case Selection Using Multi-Criteria Optimization," IEEE Trans. Software Eng., vol. 38, no. 4, pp. 936-956, July/Aug. 2012.
[42] S. Nachiyappan, A. Vimaladevi, and C.B. SelvaLakshmi, "An Evolutionary Algorithm for Regression Test Suite Reduction," Proc. Int'l Conf. Comm. and Computational Intelligence, pp. 503-508, Dec. 2010.
[43] R.L. Ott and M. Longnecker, An Introduction to Statistical Methods and Data Analysis, fifth ed. Duxbury, 2001.
[44] L. Pan, B. Zou, J. Li, and H. Chen, "Bi-Objective Model for Test-Suite Reduction Based on Modified Condition/Decision Coverage," Proc. 11th Pacific Rim Int'l Symp. Dependable Computing, p. 7, Dec. 2005.
[45] J. Prabhu, N. Malmurugan, G. Gunasekaran, and R. Gowtham, "Study of ERP Test-Suite Reduction Based on Modified Condition/Decision Coverage," Proc. Second Int'l Conf. Computer Research and Development, pp. 373-378, May 2010.
[46] X. Qu, M.B. Cohen, and K.M. Woolf, "Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization," Proc. Int'l Conf. Software Maintenance, pp. 255-264, Oct. 2007.
[47] K. Ramasamy and S.A. Mary, "Incorporating Varying Requirement Priorities and Costs in Test Case Prioritization for New and Regression Testing," Proc. Int'l Conf. Computing, Comm., and Networking, pp. 1-9, Dec. 2008.
[48] G. Rothermel, M.J. Harrold, J. von Ronne, and C. Hong, "Empirical Studies of Test Suite Reduction," J. Software Testing, Verification, and Reliability, vol. 12, no. 4, pp. 219-249, Dec. 2002.
[49] G. Rothermel, R.H. Untch, C. Chu, and M.J. Harrold, "Prioritizing Test Cases for Regression Testing," IEEE Trans. Software Eng., vol. 27, no. 10, pp. 929-948, Oct. 2001.
[50] G. Rothermel, R.J. Untch, and C. Chu, "Prioritizing Test Cases for Regression Testing," IEEE Trans. Software Eng., vol. 27, no. 10, pp. 929-948, Oct. 2001.
[51] S. Sampath and R.C. Bryce, "Improving the Effectiveness of Test Suite Reduction for User-Session-Based Testing of Web Applications," Information and Software Technology, vol. 54, no. 7, pp. 724-738, 2012.
[52] S. Sampath, S. Sprenkle, E. Gibson, and L. Pollock, "Integrating Customized Test Requirements with Traditional Requirements in Web Application Testing," Proc. Workshop Testing, Analysis, and Verification of Web Services and Applications, pp. 23-32, 2006.
[53] S. Sampath, S. Sprenkle, E. Gibson, L. Pollock, and A.S. Greenwald, "Applying Concept Analysis to User-Session-Based Testing of Web Applications," IEEE Trans. Software Eng., vol. 33, no. 10, pp. 643-658, Oct. 2007.
[54] Y. Singh, A. Kaur, and B. Suri, "Test Case Prioritization Using Ant Colony Optimization," SIGSOFT Software Eng. Notes, vol. 35, no. 4, pp. 1-7, July 2010.
[55] Software-Artifact Infrastructure Repository (SIR), http://sir.unl.edu/portalindex.php, 2012.
[56] A.M. Smith and G.M. Kapfhammer, "An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization," Proc. ACM Symp. Applied Computing, pp. 461-467, 2009.
[57] S. Sprenkle, E. Gibson, S. Sampath, and L. Pollock, "Automated Replay and Failure Detection for Web Applications," Proc. Int'l Conf. Automated Software Eng., pp. 253-262, Nov. 2005.
[58] S. Sprenkle, S. Sampath, E. Gibson, L. Pollock, and A. Souter, "An Empirical Comparison of Test Suite Reduction Techniques for User-Session-Based Testing of Web Applications," Proc. Int'l Conf. Software Maintenance, pp. 587-596, Sept. 2005.
[59] A. Srivastava and J. Thiagarajan, "Effectively Prioritizing Tests in Development Environment," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 97-106, 2002.
[60] M.G. Stochel and R. Sztando, "Testing Optimization for Mission-Critical, Complex, Distributed Systems," Proc. 32nd Ann. IEEE Int'l Computer Software and Applications, pp. 847-852, Aug. 2008.
[61] P. Tonella, P. Avesani, and A. Susi, "Using the Case-Based Prioritization," Proc. 22nd IEEE Int'l Conf. Software Maintenance, pp. 123-133, Sept. 2006.
[62] M. Utting and B. Legeard, Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., 2007.
[63] K.R. Walcott, M.L. Soffa, G.M. Kapfhammer, and R. Roos, "Time-Aware Test Suite Prioritization," Proc. Int'l Symp. Software Testing and Analysis, pp. 1-12, July 2006.
[64] Z. Wang, Z. Chen, T.-Y. Chen, and B. Xu, "Fault Class Prioritization in Boolean Expressions," Proc. 27th Ann. ACM Symp. Applied Computing, pp. 1191-1196, 2012.
[65] W.E. Wong, J.R. Horgan, S. London, and A.P. Mathur, "Effect of Test Set Minimization on Fault Detection Effectiveness," Proc. Int'l Conf. Software Eng., pp. 41-50, 1995.
[66] X.-y. Ma, Z.-f. He, B.-k. Sheng, and C.-q. Ye, "A Genetic Algorithm for Test-Suite Reduction," Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, vol. 1, pp. 133-139, Oct. 2005.
[67] S. Yoo and M. Harman, "Pareto Efficient Multi-Objective Test Case Selection," Proc. Int'l Symp. Software Testing and Analysis, pp. 140-150, July 2007.
[68] S. Yoo and M. Harman, "Using Hybrid Algorithm for Pareto Efficient Multi-Objective Test Suite Minimisation," J. Systems and Software, vol. 83, no. 4, pp. 689-701, Apr. 2010.
[69] S. Yoo and M. Harman, "Regression Testing Minimisation, Selection and Prioritisation: A Survey," J. Software Testing, Verification, and Reliability, vol. 22, no. 2, pp. 67-120, 2012.
[70] S. Yoo, M. Harman, P. Tonella, and A. Susi, "Clustering Test Cases to Achieve Effective and Scalable Prioritisation Incorporating Expert Knowledge," Proc. 18th Int'l Symp. Software Testing and Analysis, pp. 201-212, 2009.
[71] L. Zhang, J. Zhou, D. Hao, L. Zhang, and H. Mei, "Prioritizing Junit Test Cases in Absence of Coverage Information," Proc. Int'l Conf. Software Maintenance, pp. 19-28, Sept. 2009.
[72] L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei, "Time-Aware Test-Case Prioritization Using Integer Linear Programming," Proc. 18th Int'l Symp. Software Testing and Analysis, pp. 213-224, 2009.
[73] X. Zhang, Q. Gu, X. Chen, J. Qi, and D. Chen, "A Study of Relative Redundancy in Test-Suite Reduction while Retaining or Improving Fault-Localization Effectiveness," Proc. ACM Symp. Applied Computing, pp. 2229-2236, 2010.
[74] H. Zhu, P.A.V. Hall, and J.H.R. May, "Software Unit Test Coverage and Adequacy," ACM Computing Surveys, vol. 29, no. 4, pp. 366-427, Dec. 1997.
48 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool