This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback
January/February 2010 (vol. 36 no. 1)
pp. 81-95
Xun Yuan, University of Maryland, College Park
Atif M. Memon, University of Maryland, College Park
This paper presents a fully automatic model-driven technique to generate test cases for Graphical user interfaces (GUIs)-based applications. The technique uses feedback from the execution of a “seed test suite,” which is generated automatically using an existing structural event interaction graph model of the GUI. During its execution, the runtime effect of each GUI event on all other events pinpoints event semantic interaction (ESI) relationships, which are used to automatically generate new test cases. Two studies on eight applications demonstrate that the feedback-based technique 1) is able to significantly improve existing techniques and helps identify serious problems in the software and 2) the ESI relationships captured via GUI state yield test suites that most often detect more faults than their code, event, and event-interaction-coverage equivalent counterparts.

[1] E. Dustin, J. Rashka, and J. Paul, Automated Software Testing: Introduction, Management, and Performance. Addison-Wesley Professional, 1999.
[2] R.K. Shehady and D.P. Siewiorek, "A Method to Automate User Interface Testing Using Variable Finite State Machines," Proc. 27th Int'l Symp. Fault-Tolerant Computing, pp. 80-88, 1997.
[3] L. White and H. Almezen, "Generating Test Cases for GUI Responsibilities Using Complete Interaction Sequences," Proc. 11th Int'l Symp. Software Reliability Eng., p. 110, 2000.
[4] A.M. Memon and Q. Xie, "Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software," IEEE Trans. Software Eng., vol. 31, no. 10, pp. 884-896, Oct. 2005.
[5] X. Yuan and A.M. Memon, "Using GUI Run-Time State as Feedback to Generate Test Cases," Proc. 29th Int'l Conf. Software Eng., pp. 396-405, May 2007.
[6] M. d'Amorim, C. Pacheco, T. Xie, D. Marinov, and M.D. Ernst, "An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing," Proc. 21st IEEE/ACM Int'l Conf. Automated Software Eng., 2006.
[7] T. Xie and D. Notkin, "Tool-Assisted Unit-Test Generation and Selection Based on Operational Abstractions," Automated Software Eng., vol. 13, no. 3, pp. 345-371, 2006.
[8] R. Ferguson and B. Korel, "The Chaining Approach for Software Test Data Generation," ACM Trans. Software Eng. Methodolgy, vol. 5, no. 1, pp. 63-86, 1996.
[9] M.J. Gallagher and V.L. Narasimhan, "Adtest: A Test Data Generation Suite for Ada Software Systems," IEEE Trans. Software Eng., vol. 23, no. 8, pp. 473-484, Aug. 1997.
[10] B. Korel, "Automated Software Test Data Generation," IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990.
[11] C.C. Michael, G. McGraw, and M. Schatz, "Generating Software Test Data by Evolution," IEEE Trans. Software Eng., vol. 27, no. 12, pp. 1085-1110, Dec. 2001.
[12] C. Boyapati, S. Khurshid, and D. Marinov, "Korat: Automated Testing Based on Java Predicates," Proc. 2002 ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 123-133, 2002.
[13] N. Gupta, A.P. Mathur, and M.L. Soffa, "Automated Test Data Generation Using an Iterative Relaxation Method," Proc. Sixth ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 231-244, 1998.
[14] W. Miller and D.L. Spooner, "Automatic Generation of Floating-Point Test Data," IEEE Trans. Software Eng., vol. 2, no. 3, pp. 223-226, Sept. 1976.
[15] C. Pacheco, S.K. Lahiri, M.D. Ernst, and T. Ball, "Feedback-Directed Random Test Generation," Proc. 29th Int'l Conf. Software Eng., pp. 396-405, May 2007.
[16] B.A. Myers and M.B. Rosson, "Survey on User Interface Programming." Proc. ACM SIGCHI Conf. Human Factors in Computing Systems, pp. 195-202, 1992.
[17] F. Belli, C.J. Budnik, and L. White, "Event-Based Modelling, Analysis and Testing of User Interactions: Approach and Case Study: Research Articles," Software Testing, Verification, and Reliability, vol. 16, no. 1, pp. 3-32, 2006.
[18] J.M. Clarke, "Automated Test Generation from a Behavioral Model," Proc. Pacific Northwest Software Quality Conf., May 1998.
[19] S. Esmelioglu and L. Apfelbaum, "Automated Test Generation, Execution, and Reporting," Proc. Pacific Northwest Software Quality Conf., pp. 127-142, Oct. 1997.
[20] P.J. Bernhard, "A Reduced Test Suite for Protocol Conformance Testing," ACM Trans. Software Eng. and Methodology, vol. 3, no. 3, pp. 201-220, July 1994.
[21] W.-H. Chen, C.-S. Lu, E.R. Brozovsky, and J.-T. Wang, "An Optimization Technique for Protocol Conformance Testing Using Multiple UIO Sequences," Information Processing Letters, vol. 36, no. 1, pp. 7-11, 1990.
[22] T.S. Chow, "Testing Software Design Modeled by Finite-State Machines," IEEE Trans. Software Eng., vol. 4, no. 3, pp. 178-187, May 1978.
[23] A. von Mayrhauser, R.T. Mraz, and J. Walls, "Domain Based Regression Testing," Proc. Int'l Conf. Software Maintenance, pp. 26-35, 1994.
[24] P.M. Maurer, "Generating Test Data with Enhanced Context-Free Grammars," IEEE Software, vol. 7, no. 4, pp. 50-55, July 1990.
[25] M. Auguston, J.B. Michael, and M.-T. Shing, "Environment Behavior Models for Scenario Generation and Testing Automation," Proc. First Int'l Workshop Advances in Model-Based Testing, pp. 1-6, 2005.
[26] A.M. Memon, M.E. Pollack, and M.L. Soffa, "Hierarchical GUI Test Case Generation Using Automated Planning," IEEE Trans. Software Eng., vol. 27, no. 2, pp. 144-155, Feb. 2001.
[27] A. Howe, A. von Mayrhauser, and R.T. Mraz, "Test Case Generation as an AI Planning Problem," Automated Software Eng., vol. 4, pp. 77-106, 1997.
[28] D.J. Kasik and H.G. George, "Toward Automatic Generation of Novice User Test Scripts," Proc. Conf. Human Factors in Computing Systems: Common Ground, pp. 244-251, Apr. 1996.
[29] D.M. Woit, "Specifying Operational Profiles for Modules," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 2-10, 1993.
[30] B. Sarikaya, "Conformance Testing: Architectures and Test Sequences," Computer Networks and ISDN Systems, vol. 17, no. 2, pp. 111-126, 1989.
[31] F. Ipate and M. Holcombe, "Complete Testing from a Stream X-Machine Specification," Fundamenta Informaticae, vol. 64, nos. 1-4, pp. 205-216, 2004.
[32] M. Barnett, W. Grieskamp, L. Nachmanson, W. Schulte, N. Tillmann, and M. Veanes, "Towards a Tool Environment for Model-Based Testing with AsmL," Proc. Int'l Workshop Formal Approach to Software Testing, pp. 252-266, 2003.
[33] E. Farchi, A. Hartman, and S.S. Pinter, "Using a Model-Based Test Generator to Test for Standard Conformance," IBM Systems J., vol. 41, no. 1, pp. 89-110, 2002.
[34] H.S. Hong, Y.R. Kwon, and S.D. Cha, "Testing of Object-Oriented Programs Based on Finite State Machines," Proc. Asia Pacific Software Eng. Conf., pp. 234-241, 1995.
[35] L. Apfelbaum, "Automated Functional Test Generation," Proc. Int'l Automatic Testing Conf., 1995.
[36] L. Lucio, L. Pedro, and D. Buchs, "A Methodology and a Framework for Model-Based Testing," Rapid Integration of Software Eng. Techniques, N. Guelfi, ed., pp. 57-70, Springer, 2005.
[37] J.A. Whittaker, "Stochastic Software Testing," Ann. Software Eng., vol. 4, pp. 115-131, 1997.
[38] C. Campbell, W. Grieskamp, L. Nachmanson, W. Schulte, N. Tillmann, and M. Veanes, "Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer," Formal Methods and Testing, pp. 39-76, Springer, 2008.
[39] P.W.M. Koopman, R. Plasmeijer, and P. Achten, "Model-Based Testing of Thin-Client Web Applications," Proc. Int'l Workshop Formal Approaches to Testing and Runtime Verification, pp. 115-132, 2006.
[40] N.H. Lee and S.D. Cha, "Generating Test Sequences from a Set of MSCs," Computer Networks, vol. 42, no. 3, pp. 405-417, 2003.
[41] F. Belli, "Finite-State Testing and Analysis of Graphical User Interfaces," Proc. 12th Int'l Symp. Software Reliability Eng., pp. 34-43, 2001.
[42] D. Hovemeyer and W. Pugh, "Finding Bugs Is Easy," ACM SIGPLAN Notices, vol. 39, no. 12, pp. 92-106, 2004.
[43] X. Yuan and A.M. Memon, "Using GUI Run-Time State as Feedback for Test Automation," technical report, Univ. of Maryland, http://hdl.handle.net/19039416, http://hdl.handle. net/19039416, Aug. 2009.
[44] Q. Xie and A.M. Memon, "Designing and Comparing Automated Test Oracles for GUI-Based Software Applications," ACM Trans. Software Eng. and Methodology, vol. 16, no. 1, p. 4, 2007.
[45] L.C. Briand, Y. Labiche, and Y. Wang, "Using Simulation to Empirically Investigate Test Coverage Criteria Based on Statechart," Proc. 26th Int'l Conf. Software Eng., pp. 86-95, 2004.
[46] G. Rothermel, S. Elbaum, A.G. Malishevsky, P. Kallakuri, and X. Qiu, "On Test Suite Composition and Cost-Effective Regression Testing," ACM Trans. Software Eng. Methodology, vol. 13, no. 3, pp. 277-331, 2004.
[47] D.J. Richardson and M.C. Thompson, "An Analysis of Test Data Selection Criteria Using the Relay Model of Fault Detection," IEEE Trans. Software Eng., vol. 19, no. 6, pp. 533-553, June 1993.
[48] F. Belli and C.J. Budnik, "Test Minimization for Human-Computer Interaction," Applied Intelligence, vol. 26, no. 2, pp. 161-174, 2007.
[49] P.T. Alessandro Marchetto and F. Ricca, "State-Based Testing of Ajax Web Applications" Proc. First Int'l Conf. Software Testing, Verification, and Validation, pp. 121-130, Apr. 2008.

Index Terms:
GUI testing, automated testing, model-based testing, GUITAR testing system.
Citation:
Xun Yuan, Atif M. Memon, "Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback," IEEE Transactions on Software Engineering, vol. 36, no. 1, pp. 81-95, Jan.-Feb. 2010, doi:10.1109/TSE.2009.68
Usage of this product signifies your acceptance of the Terms of Use.