The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - January (2008 vol.34)
pp: 99-115
ABSTRACT
Graphical user interfaces (GUIs) are used as front-ends to most of today's software applications. The event-driven nature of GUIs presents new challenges for testing. One important challenge is test suite reduction. Conventional reduction techniques/tools based on static analysis are not easily applicable due to the increased use of multi-language GUI implementations, callbacks for event handlers, virtual function calls, reflection, and multi-threading. Moreover, many existing techniques ignore code in libraries, and fail to consider the context in which event handlers execute. Consequently, they yield GUI test suites with seriously impaired fault-detection ability. This paper presents a reduction technique based on the call-stack coverage criterion. Call stacks may be collected for any executing program with very little overhead. Empirical studies in this paper compare reduction based on call-stack coverage to reduction based on line, method, and event coverage, including variations that control for size and optional consideration of library methods. These studies show that call-stack-based reduction provides an excellent tradeoff between reduction in test suite size and loss of fault detection effectiveness. Additionally, an analysis of the relationship between coverage elements and fault-revealing test cases is presented.
INDEX TERMS
Testing strategies, Test coverage of code, Test management, Testing tools
CITATION
Scott McMaster, Atif Memon, "Call-Stack Coverage for GUI Test Suite Reduction", IEEE Transactions on Software Engineering, vol.34, no. 1, pp. 99-115, January 2008, doi:10.1109/TSE.2007.70756
REFERENCES
[1] G. Ammons, T. Ball, and J.R. Larus, “Exploiting Hardware Performance Counters with Flow and Context Sensitive Profiling,” Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, 1997.
[2] W. Dickinson, D. Leon, and A. Podgurski, “Finding Failures by Cluster Analysis of Execution Profiles,” Proc. 23rd Int'l Conf. Software Eng., pp. 339-348, 2001.
[3] H. Do and G. Rothermel, “An Empirical Study of Regression Testing Techniques Incorporating Context and Life Cycle Factors and Improved Cost-Benefit Models,” Proc. 14th ACM SIGSOFT Symp. Foundations of Software Eng., Nov. 2006.
[4] S. Elbaum, A. Malishevsky, and G. Rothermel, “Test Case Prioritization: A Family of Empirical Studies,” IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002.
[5] P.G. Frankl and O. Iakounenko, “Further Empirical Studies of Test Effectiveness,” Proc. Sixth ACM SIGSOFT Symp. Foundations of Software Eng., Nov. 1998.
[6] M. Harder, J. Mellen, and M.D. Ernst, “Improving Test Suites via Operational Abstraction,” Proc. 25th Int'l Conf. Software Eng., pp.60-71, 2003.
[7] M.J. Harrold, R. Gupta, and M.L. Soffa, “A Methodology for Controlling the Size of a Test Suite,” ACM Trans. Software Eng. and Methodology, vol. 2, no. 3, July 1993.
[8] J.R. Horgan and S. London, “Data Flow Coverage and the C Language,” Proc. Fourth ACM Symp. Testing, Analysis, and Verification, 1991.
[9] G. Hunt and D. Brubacher, “Detours: Binary Interception of Win32 Functions,” Proc. Third Usenix Windows NT Symp., pp. 135-143, July 1999.
[10] JavaCCTAgent information on the Web, http://sourceforge.net/projectsjavacctagent /, Apr. 2007.
[11] Java Native Interface specification, http://java.sun.com/j2se/1.4.2/docs/guide jni/, Sept. 2006.
[12] jcoverage information on the Web, http:/www.jcoverage.com/, Apr. 2006.
[13] D. Jeffrey and N. Gupta, “Test Suite Reduction with Selective Redundancy,” Proc. 21st IEEE Int'l Conf. Software Maintenance, pp.549-558, 2005.
[14] D. Leon and A. Podgurski, “A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases,” Proc. 14th IEEE Int'l Symp. Software Reliability Eng., Nov. 2003.
[15] S. McMaster and A. Memon, “Call Stack Coverage for GUI Test-Suite Reduction,” Proc. 17th IEEE Int'l Symp. Software Reliability Eng., Nov. 2006.
[16] S. McMaster and A. Memon, “Call Stack Coverage for Test Suite Reduction,” Proc. 21st IEEE Int'l Conf. Software Maintenance, pp.539-548, 2005.
[17] A. Memon, A. Nagarajan, and Q. Xie, “Automating Regression Testing for Evolving GUI Software,” J. Software Maintenance and Evolution: Research and Practice, vol. 17, no. 1, pp. 27-64, 2005.
[18] A. Memon, M. Pollack, and M.L. Soffa, “Automated Test Oracles for GUIs,” Proc. Eighth ACM SIGSOFT Symp. Foundations of Software Eng., pp. 30-39, 2000.
[19] A. Memon, M. Pollack, and M.L. Soffa, “Hierarchical GUI Test Case Generation Using Automated Planning,” IEEE Trans. Software Eng., vol. 27, no. 2, pp. 144-155, Feb. 2001.
[20] A. Memon, M.L. Soffa, and M. Pollack, “Coverage Criteria for GUI Testing,” Proc. Eight European Software Eng. Conf./Ninth ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 256-267, 2001.
[21] A. Memon and Q. Xie, “Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software,” IEEE Trans. Software Eng., vol. 31, no. 10, pp. 884-896, Oct. 2005.
[22] J. Offutt, J. Pan, and J. Voas, “Procedures for Reducing the Size of Coverage-Based Test Sets,” Proc. 12th Int'l Conf. Testing Computer Software, pp. 111-123, June 1995.
[23] S. Rapps and E.J. Weyuker, “Selecting Software Test Data Using Data Flow Information,” IEEE Trans. Software Eng., vol. 11, no. 4, pp. 367-375, Apr. 1985.
[24] G. Rothermel, M.J. Harrold, J. Ostrin, and C. Hong, “An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites,” Proc. 14th Int'l Conf. Software Maintenance, pp. 34-43, Nov. 1998.
[25] G. Rothermel, M.J. Harrold, J. von Ronne, and C. Hong, “Empirical Studies of Test-Suite Reduction,” J. Software Testing, Verification, and Reliability, vol. 12, no. 4, Dec. 2002.
[26] G. Rothermel, R. Untch, C. Chu, and M.J. Harrold, “Test Case Prioritization,” IEEE Trans. Software Eng., vol. 27, no. 10, pp. 929-948, Oct. 2001.
[27] A. Rountev, S. Kagan, and M. Gibas, “Static and Dynamic Analysis of Call Chains in Java,” Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 1-11, July 2004.
[28] S. Sampath, V. Mihaylov, A. Souter, and L. Pollock, “A Scalable Approach to User-Session Based Testing of Web Applications through Concept Analysis,” Proc. 19th IEEE Int'l Conf. Automated Software Eng., pp. 132-141, 2004.
[29] S. Sampath, S. Sprenkle, E. Gibson, and L. Pollock, “Web Application Testing with Customized Test Requirements: An Experimental Comparison Study,” Proc. 17th IEEE Int'l Symp. Software Reliability Eng.), Nov. 2006.
[30] S. Wagner, “A Model and Sensitivity Analysis of the Quality Economics of Defect-Detection Techniques,” Proc. ACM Int'l Symp. Software Testing and Analysis, July 2006.
[31] W.E. Wong, J.R. Horgan, S. London, and A.P. Mathur, “Effect of Test Set Minimization on Fault Detection Effectiveness,” Proc. 17th Int'l Conf. Software Eng., pp. 41-50, 1995.
[32] T. Xie, D. Marinov, and D. Notkin, “Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests,” Proc. 19th IEEE Int'l Conf. Automated Software Eng., pp. 196-205, Sept. 2004.
30 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool