Issue No. 05 - September/October (2010 vol. 36)
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TSE.2010.58
Hyunsook Do , North Dakota State University, Fargo
Ladan Tahvildari , University of Waterloo, Waterloo
Gregg Rothermel , University of Nebraska—Lincoln, Lincoln
Siavash Mirarab , IBM, Vancouver
Regression testing is an expensive process used to validate modified software. Test case prioritization techniques improve the cost-effectiveness of regression testing by ordering test cases such that those that are more important are run earlier in the testing process. Many prioritization techniques have been proposed and evidence shows that they can be beneficial. It has been suggested, however, that the time constraints that can be imposed on regression testing by various software development processes can strongly affect the behavior of prioritization techniques. If this is correct, a better understanding of the effects of time constraints could lead to improved prioritization techniques and improved maintenance and testing processes. We therefore conducted a series of experiments to assess the effects of time constraints on the costs and benefits of prioritization techniques. Our first experiment manipulates time constraint levels and shows that time constraints do play a significant role in determining both the cost-effectiveness of prioritization and the relative cost-benefit trade-offs among techniques. Our second experiment replicates the first experiment, controlling for several threats to validity including numbers of faults present, and shows that the results generalize to this wider context. Our third experiment manipulates the number of faults present in programs to examine the effects of faultiness levels on prioritization and shows that faultiness level affects the relative cost-effectiveness of prioritization techniques. Taken together, these results have several implications for test engineers wishing to cost-effectively regression test their software systems. These include suggestions about when and when not to prioritize, what techniques to employ, and how differences in testing processes may relate to prioritization cost--effectiveness.
Regression testing, test case prioritization, cost-benefits, Bayesian networks, empirical studies.
Hyunsook Do, Ladan Tahvildari, Gregg Rothermel, Siavash Mirarab, "The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments", IEEE Transactions on Software Engineering, vol. 36, no. , pp. 593-617, September/October 2010, doi:10.1109/TSE.2010.58