The Community for Technology Leaders
RSS Icon
Issue No.04 - July/August (2011 vol.37)
pp: 559-574
Xun Yuan , Google Kirkland
Myra B. Cohen , University of Nebraska-Lincoln, Lincoln
Atif M. Memon , University of Maryland, College Park
Graphical user interfaces (GUIs), due to their event-driven nature, present an enormous and potentially unbounded way for users to interact with software. During testing, it is important to “adequately cover” this interaction space. In this paper, we develop a new family of coverage criteria for GUI testing grounded in combinatorial interaction testing. The key motivation of using combinatorial techniques is that they enable us to incorporate “context” into the criteria in terms of event combinations, sequence length, and by including all possible positions for each event. Our new criteria range in both efficiency (measured by the size of the test suite) and effectiveness (the ability of the test suites to detect faults). In a case study on eight applications, we automatically generate test cases and systematically explore the impact of context, as captured by our new criteria. Our study shows that by increasing the event combinations tested and by controlling the relative positions of events defined by the new criteria, we can detect a large number of faults that were undetectable by earlier techniques.
GUI testing, automated testing, model-based testing, combinatorial interaction testing, GUITAR testing system.
Xun Yuan, Myra B. Cohen, Atif M. Memon, "GUI Interaction Testing: Incorporating Event Context", IEEE Transactions on Software Engineering, vol.37, no. 4, pp. 559-574, July/August 2011, doi:10.1109/TSE.2010.50
[1] A.P. Mathur, Foundations of Software Testing: Fundamental Algorithms and Techniques. Pearson Education, 2008.
[2] Q. Xie and A.M. Memon, "Using a Pilot Study to Derive a GUI Model for Automated Testing," ACM Trans. Software Eng. and Methodology, vol. 18, pp. 1-35, 2008.
[3] A.M. Memon and Q. Xie, "Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software," IEEE Trans. Software Eng., vol. 31, no. 10, pp. 884-896, Oct. 2005.
[4] A.M. Memon and Q. Xie, "Using Transient/Persistent Errors to Develop Automated Test Oracles for Event-Driven Software," Proc. 19th IEEE Int'l Conf. Automated Software Eng., pp. 186-195, 2004.
[5] A.M. Memon, "Developing Testing Techniques for Event-Driven Pervasive Computing Applications," Proc. Object-Oriented Programming, Systems, Languages, and Applications Workshop Building Software for Pervasive Computing, Oct. 2004.
[6] G.J. Tretmans and H. Brinksma, "TorX: Automated Model-Based Testing," Proc. First European Conf. Model-Driven Software Eng., A. Hartman and K. Dussa-Ziegler, eds., pp. 31-43, Dec. 2003.
[7] M. Veanes, C. Campbell, W. Grieskamp, W. Schulte, N. Tillmann, and L. Nachmanson, "Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer," Formal Methods and Testing, pp. 39-76, Springer, 2008.
[8] H. Ural and B. Yang, "A Test Sequence Selection Method for Protocol Testing," IEEE Trans. Comm., vol. 39, no. 4, pp. 514-523, Apr. 1991.
[9] A. Marchetto and P. Tonella, "Search-Based Testing of Ajax Web Applications," Proc. First Int'l Symp. Search Based Software Eng., pp. 3-12, May 2009.
[10] J. Offutt, S. Liu, A. Abdurazik, and P. Ammann, "Generating Test Data from State-Based Specifications," The J. Software Testing, Verification and Reliability, vol. 13, no. 1, pp. 25-53, 2003.
[11] P. Brooks, B. Robinson, and A.M. Memon, "An Initial Characterization of Industrial Graphical User Interface Systems," Proc. First IEEE Int'l Conf. Software Testing, Verification and Validation, 2009.
[12] "FreeMind—Free Mind-Mapping Software," http:/freemind. 2009.
[13] A.M. Memon, M.L. Soffa, and M.E. Pollack, "Coverage Criteria for GUI Testing," Proc. European Software Eng. Conf./Foundations of Software Eng., pp. 256-267, 2001.
[14] H. Zhu, P.A.V. Hall, and J.H.R. May, "Software Unit Test Coverage and Adequacy," ACM Computing Surveys, vol. 29, no. 4, pp. 366-427, 1997.
[15] X. Yuan, M. Cohen, and A.M. Memon, "Covering Array Sampling of Input Event Sequences for Automated GUI Testing," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 405-408, 2007.
[16] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, "The AETG System: An Approach to Testing Based on Combinatorial Design," IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437-444, July 1997.
[17] M.B. Cohen, C.J. Colbourn, P.B. Gibbons, and W.B. Mugridge, "Constructing Test Suites for Interaction Testing," Proc. Int'l Conf. Software Eng., pp. 38-48, May 2003.
[18] J. Strecker and A.M. Memon, "Relationships between Test Suites, Faults, and Fault Detection in GUI Testing," Proc. First Int'l Conf. Software Testing, Verification, and Validation, pp. 12-21, 2008.
[19] X. Yuan and A.M. Memon, "Using GUI Run-Time State as Feedback to Generate Test Cases," Proc. Int'l Conf. Software Eng., pp. 396-405, 2007.
[20] F.J. Daniels and K.C. Tai, "Measuring the Effectiveness of Method Test Sequences Derived from Sequencing Constraints," Proc. Technology of Object-Oriented Languages and Systems, pp. 74-83, 1999.
[21] U. Farooq, C.P. Lam, and H. Li, "Towards Automated Test Sequence Generation," Proc. Australian Software Eng. Conf., pp. 441-450, 2008.
[22] K. Inkumsah and T. Xie, "Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 297-306, 2008.
[23] L. Gallagher and J. Offutt, "Test Sequence Generation for Integration Testing of Component Software," The Computer J., vol. 52, pp. 514-529, 2007.
[24] A. Gargantini and E. Riccobene, "ASM-Based Testing: Coverage Criteria and Automatic Test Sequence Generation," J. Universal Computer Science, vol. 7, no. 11, pp. 1050-1067, 2001.
[25] R.K. Shehady and D.P. Siewiorek, "A Method to Automate User Interface Testing Using Variable Finite State Machines," Proc. Int'l Symp. Fault-Tolerant Computing, pp. 80-88, 1997.
[26] A.M. Memon, M.E. Pollack, and M.L. Soffa, "Hierarchical GUI Test Case Generation Using Automated Planning," IEEE Trans. Software Eng., vol. 27, no. 2, pp. 144-155, Feb. 2001.
[27] L. White and H. Almezen, "Generating Test Cases for GUI Responsibilities Using Complete Interaction Sequences," Proc. Int'l Symp. Software Reliability Eng., pp. 110-121, 2000.
[28] S. McMaster and A.M. Memon, "Call-Stack Coverage for GUI Test-Suite Reduction," IEEE Trans. Software Eng., vol. 34, no. 1, pp. 99-115, Jan./Feb. 2008.
[29] Q. Xie and A.M. Memon, "Rapid "Crash Testing" for Continuously Evolving GUI-Based Software Applications," Proc. Int'l Conf. Software Maintenance, pp. 473-482, 2005.
[30] F. Belli, "Finite-State Testing and Analysis of Graphical User Interfaces," Proc. Int'l Symp. Software Reliability Eng., pp. 34-43, 2001.
[31] L.J. White, "Regression Testing of GUI Event Interactions," Proc. Int'l Conf. Software Maintenance, pp. 350-358, 1996.
[32] "JUnit, Testing Resources for Extreme Programming," , 2010.
[33] "Mercury Interactive WinRunner," http://www. productswinrunner, 2003.
[34] R. Brownlie, J. Prowse, and M.S. Phadke, "Robust Testing of AT&T PMX/StarMAIL Using OATS," AT&T Technical J., vol. 71, no. 3, pp. 41-47, 1992.
[35] I.S. Dunietz, W.K. Ehrlich, B.D. Szablak, C.L. Mallows, and A. Iannino, "Applying Design of Experiments to Software Testing," Proc. Int'l Conf. Software Eng., 1997, pp. 205-215,
[36] D. Kuhn, D.R. Wallace, and A.M. Gallo, "Software Fault Interactions and Implications for Software Testing," IEEE Trans. Software Eng., vol. 30, no. 6, pp. 418-421, June 2004.
[37] C. Yilmaz, M.B. Cohen, and A. Porter, "Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces," IEEE Trans. Software Eng., vol. 32, no. 1, pp. 20-34, Jan. 2006.
[38] X. Qu, M.B. Cohen, and G. Rothermel, "Configuration-Aware Regression Testing: An Empirical Study of Sampling and Prioritization," Proc. Int'l Symp. Software Testing and Analysis, pp. 75-85, July 2008.
[39] M.B. Cohen, M.B. Dwyer, and J. Shi, "Coverage and Adequacy in Software Product Line Testing," Proc. Workshop Role of Architecture for Testing and Analysis, pp. 53-63, July 2006.
[40] D. Chays, S. Dan, Y. Deng, F.I. Vokolos, P.G. Frankl, and E.J. Weyuker, "AGENDA: A Test Case Generator for Relational Database Applications," technical report, Polytechnic Univ., 2002.
[41] R.C. Bryce, A. Rajan, and M.P.E. Heimdahl, "Interaction Testing in Model-Based Development: Effect on Model-Coverage," Proc. Asia Pacific Software Eng. Conf., pp. 259-268, 2006.
[42] Q. Xie and A.M. Memon, "Model-Based Testing of Community-Driven Open-Source GUI applications," Proc. Int'l Conf. Software Maintenance, pp. 145-154, 2006.
[43] D. Hovemeyer and W. Pugh, "Finding Bugs is Easy," ACM SIGPLAN Notices, vol. 39, no. 12, pp. 92-106, 2004.
[44] X. Yuan, M. Cohen, and A.M. Memon, "Towards Dynamic Adaptive Automated Test Generation for Graphical User Interfaces," Proc. First Int'l Workshop TESTing Techniques & Experimentation Benchmarks for Event-Driven Software, pp. 1-4, 2009.
13 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool