The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - Feb. (2013 vol.39)
pp: 258-275
Shifa-e-Zehra Haidry , University of Melbourne, Parkville
Tim Miller , University of Melbourne, Parkville
ABSTRACT
Test case prioritization is the process of ordering the execution of test cases to achieve a certain goal, such as increasing the rate of fault detection. Increasing the rate of fault detection can provide earlier feedback to system developers, improving fault fixing activity and, ultimately, software delivery. Many existing test case prioritization techniques consider that tests can be run in any order. However, due to functional dependencies that may exist between some test cases—that is, one test case must be executed before another—this is often not the case. In this paper, we present a family of test case prioritization techniques that use the dependency information from a test suite to prioritize that test suite. The nature of the techniques preserves the dependencies in the test ordering. The hypothesis of this work is that dependencies between tests are representative of interactions in the system under test, and executing complex interactions earlier is likely to increase the fault detection rate, compared to arbitrary test orderings. Empirical evaluations on six systems built toward industry use demonstrate that these techniques increase the rate of fault detection compared to the rates achieved by the untreated order, random orders, and test suites ordered using existing "coarse-grained” techniques based on function coverage.
INDEX TERMS
Fault detection, Digital signal processing, Schedules, Software, Complexity theory, Testing, Weight measurement, test execution, Software engineering, testing and debugging
CITATION
Shifa-e-Zehra Haidry, Tim Miller, "Using Dependency Structures for Prioritization of Functional Test Suites", IEEE Transactions on Software Engineering, vol.39, no. 2, pp. 258-275, Feb. 2013, doi:10.1109/TSE.2012.26
REFERENCES
[1] J. Bach, "Useful Features of a Test Automation System (Part iii)," Testing Techniques Newsletter, Oct. 1996.
[2] F. Basanieri, A. Bertolino, and E. Marchetti, "The Cow_Suite Approach to Planning and Deriving Test Suites in UML Projects," Proc. Fifth Int'l Conf. Unified Modeling Language, pp. 275-303, 2002.
[3] S. Elbaum, A. Malishevsky, and G. Rothermel, "Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization," Proc. 23rd Int'l Conf. Software Eng., pp. 329-338, 2001.
[4] S. Elbaum, A.G. Malishevsky, and G. Rothermel, "Test Case Prioritization: A Family of Empirical Studies," IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002.
[5] R.W. Floyd, "Algorithm 97: Shortest Path," Comm. ACM, vol. 5, no. 6, p. 345, June 1962.
[6] D. Jeffrey and N. Gupta, "Experiments with Test Case Prioritization Using Relevant Slices," J. Systems and Software, vol. 81, no. 2, pp. 196-221, 2008.
[7] B. Jiang, Z. Zhang, W. Chan, and T. Tse, "Adaptive Random Test Case Prioritization," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 233-244, 2009.
[8] J. Kim and D. Bae, "An Approach to Feature Based Modelling by Dependency Alignment for the Maintenance of the Trustworthy System," Proc. 28th Ann. Int'l Computer Software and Applications Conf., pp. 416-423, 2004.
[9] R. Krishnamoorthi and S.A. Sahaaya Arul Mary, "Factor Oriented Requirement Coverage Based System Test Case Prioritization of New and Regression Test Cases," Information and Software Technology, vol. 51, no. 4, pp. 799-808, 2009.
[10] D. Kundu, M. Sarma, D. Samanta, and R. Mall, "System Testing for Object-Oriented Systems with Test Case Prioritization," Software Testing, Verification, and Reliability, vol. 19, no. 4, pp. 97-333, 2009.
[11] K.S. Lew, T.S. Dillon, and K.E. Forward, "Software Complexity and Its Impact on Software Reliability," IEEE Trans. Software Eng., vol. 14, no. 11, pp. 1645-1655, Nov. 1988.
[12] J. Li, "Prioritize Code for Testing to Improve Code Coverage of Complex Software," Proc. 16th IEEE Int'l Symp. Software Reliability Eng., pp. 75-84, 2005.
[13] J. Li, D. Weiss, and H. Yee, "Code-Coverage Guided Prioritized Test Generation," J. Information and Software Technology, vol. 48, no. 12, pp. 1187-1198, 2006.
[14] Z. Li, M. Harman, and R. Hierons, "Search Algorithms for Regression Test Case Prioritization," IEEE Trans. Software Eng., vol. 33, no. 4, pp. 225-237, Apr. 2007.
[15] Z. Ma and J. Zhao, "Test Case Prioritization Based on Analysis of Program Structure," Proc. 15th Asia-Pacific Software Eng. Conf., pp. 471-478, 2008.
[16] T. Miller and P. Strooper, "A Case Study in Model-Based Testing of Specifications and Implementations," Software Testing, Verification, and Reliability, vol. 22, no. 1, pp. 33-63, 2012.
[17] B. Peirce, "Criterion for the Rejection of Doubtful Observations," Astronomical J. II, vol. 45, pp. 161-163, 1852.
[18] C. Perrow, Normal Accidents: Living with High-Risk Technologies. Princeton Univ Press, 1999.
[19] J. Poole, "A Method to Determine a Basis Set of Paths to Perform Program Testing," Nat'l Inst. of Standards and Tech nology, Nov. 1995.
[20] X. Qu, M.B. Cohen, and K.M. Woolf, "Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization," Proc. IEEE Int'l Conf. Software Maintenance, pp. 255-264, 2007.
[21] S.M. Ross, "Peirce's Criterion for the Elimination of Suspect Experimental Data," J. Eng. Technology, vol. 20, no. 2, pp. 38-41, 2003.
[22] G. Rothermel, S. Elbaum, A. Malishevsky, P. Kallakuri, and B. Davia, "The Impact of Test Suite Granularity on the Cost-Effectiveness of Regression Testing," Proc. 24th Int'l Conf. Sofware Eng., p. 130, 2002.
[23] G. Rothermel, R.H. Untch, C. Chu, and M.J. Harrold, "Prioritizing Test Cases for Regression Testing," IEEE Trans. Software Eng., vol. 27, no. 10, pp. 929-948, Sept. 2001.
[24] J. Ryser and M. Glinz, "Using Dependency Charts to Improve Scenario-Based Testing," Proc. 17th Int'l Conf. Testing Computer Software, 2000.
[25] J.B. Sidney, "Decomposition Algorithms for Single-Machine Sequencing with Precedence Relations and Deferral Costs," Operations Research, vol. 23, no. 2, pp. 283-298, 1975.
[26] H. Srikanth, L. Williams, and J. Osborne, "System Test Case Prioritization of New and Regression Test Cases," Proc. Fourth Int'l Symp. Empirical Software Eng., pp. 62-71, 2005.
[27] P. Stocks and D. Carrington, "A Framework for Specification-Based Testing," IEEE Trans. Software Eng., vol. 22, no. 11, pp. 777-793, Nov. 1996.
[28] P. Tonella, P. Avesani, and A. Susi, "Using the Case-Based Ranking Methodology for Test Case Prioritization," Proc. 22nd IEEE Int'l Conf. Software Maintenance, pp. 123-133, 2006.
[29] S. Warshall, "A Theorem on Boolean Matrices," J. ACM, vol. 9, no. 1, pp. 11-12, Jan. 1962.
[30] W.E. Wong, J.R. Horgan, S. London, and H. Agrawal, "A Study of Effective Regression Testing in Practice," Proc. Eighth Int'l Symp. Software Reliability Eng., pp. 230-238, 1997.
[31] S. Yoo, M. Harman, P. Tonella, and A. Susi, "Clustering Test Cases to Achieve Effective and Scalable Prioritisation Incorporating Expert Knowledge," Proc. 18th Int'l Symp. Software Testing and Analysis, pp. 201-212, 2009.
14 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool