The Community for Technology Leaders
RSS Icon
Issue No.05 - May (2013 vol.39)
pp: 707-724
Juha Itkonen , Aalto University School of Science, Espoo
Mika V. Mäntylä , Aalto University School of Science, Espoo
Casper Lassenius , Aalto University School of Science, Espoo
We present a field study on how testers use knowledge while performing exploratory software testing (ET) in industrial settings. We video recorded 12 testing sessions in four industrial organizations, having our subjects think aloud while performing their usual functional testing work. Using applied grounded theory, we analyzed how the subjects performed tests and what type of knowledge they utilized. We discuss how testers recognize failures based on their personal knowledge without detailed test case descriptions. The knowledge is classified under the categories of domain knowledge, system knowledge, and general software engineering knowledge. We found that testers applied their knowledge either as a test oracle to determine whether a result was correct or not, or for test design, to guide them in selecting objects for test and designing tests. Interestingly, a large number of failures, windfall failures, were found outside the actual focus areas of testing as a result of exploratory investigation. We conclude that the way exploratory testers apply their knowledge for test design and failure recognition differs clearly from the test-case-based paradigm and is one of the explanatory factors of the effectiveness of the exploratory testing approach.
Software testing, Context, Software, Knowledge engineering, Observers, Organizations, and V&V, Software testing, exploratory testing, validation, test execution, test design, human factors, methods for SQA
Juha Itkonen, Mika V. Mäntylä, Casper Lassenius, "The Role of the Tester's Knowledge in Exploratory Software Testing", IEEE Transactions on Software Engineering, vol.39, no. 5, pp. 707-724, May 2013, doi:10.1109/TSE.2012.55
[1] G.J. Myers, The Art of Software Testing. John Wiley & Sons, 1979.
[2] B. Beizer, Software Testing Techniques. Van Nostrand Reinhold, 1990.
[3] C. Kaner, J. Falk, and H.Q. Nguyen, Testing Computer Software. John Wiley & Sons, Inc., 1999.
[4] L. Copeland, A Practitioner's Guide to Software Test Design. Artech House Publishers, 2004.
[5] J.B. Goodenough and S.L. Gerhart, "Toward a Theory of Test Data Selection," IEEE Trans. Software Eng., vol. 1, no. 2, pp. 156-173, Mar. 1975.
[6] C. Andersson and P. Runeson, "Verification and Validation in Industry—A Qualitative Survey on the State of Practice," Proc. Int'l Symp. Empirical Software Eng., pp. 37-47, 2002.
[7] S. Ng, T. Murnane, K. Reed, D. Grant, and T. Chen, "A Preliminary Survey on Software Testing Practices in Australia," Proc. Australian Software Eng. Conf., pp. 116-125. 2004,
[8] E. Engström and P. Runeson, "A Qualitative Survey of Regression Testing Practices," Proc. 11th Int'l Conf. Product-Focused Software Process Improvement, 2010.
[9] M. Fewster and D. Graham, Software Test Automation. Addison-Wesley, 1999.
[10] C. Kaner, J. Bach, and B. Pettichord, Lessons Learned in Software Testing. John Wiley & Sons, Inc., 2002.
[11] S. Berner, R. Weber, and R.K. Keller, "Observations and Lessons Learned from Automated Testing," Proc. Int'l Conf. Software Eng., pp. 571-579, 2005.
[12] A. Beer and R. Ramler, "The Role of Experience in Software Testing Practice," Proc. Euromicro Conf. Software Eng. and Advanced Applications, pp. 258-265, 2008.
[13] J. Bach, "Exploratory Testing," The Testing Practitioner, second ed., E. van Veenendaal, ed., UTN Publishers, pp. 253-265, 2004.
[14] A. Abran, J.W. Moore, P. Bourque, R. Dupuis, and L.L. Tripp, Guide to the Software Engineering Body of Knowledge. IEEE CS, 2004.
[15] A. Tinkham and C. Kaner, "Exploring Exploratory Testing," Proc. Software Testing Analysis & Rev. Conf. East, p. 9, 2003.
[16] J. Itkonen and K. Rautiainen, "Exploratory Testing: A Multiple Case Study," Proc. Int'l Symp. Empirical Software Eng., pp. 84-93, 2005.
[17] J.A. Whittaker, Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design. Addison-Wesley Professional, 2009.
[18] J. Våga and S. Amland, "Managing High-Speed Web Testing," Software Quality and Software Testing in Internet Times, D. Meyerhoff, B. Laibarra, R. van der Pouw Kraan, and A. Wallet, eds., pp. 23-30, Springer-Verlag, 2002.
[19] J. Lyndsay and N. van Eeden, "Adventures in Session-Based Testing," AiSBTv1.2.pdf, May 2003.
[20] B. Wood and D. James, "Applying Session-Based Testing to Medical Software," Medical Device & Diagnostic Industry, vol. 25, no. 5, p. 90, May 2003.
[21] J. Tuomikoski and I. Tervonen, "Absorbing Software Testing into the Scrum Method," Proc. 10th Int'l Conf. Product-Focused Software Process Improvement, 2009.
[22] P. Poon, T.H. Tse, S. Tang, and F. Kuo, "Contributions of Tester Experience and a Checklist Guideline to the Identification of Categories and Choices for Software Testing," Software Quality J., vol. 19, no. 1, pp. 141-163, 2011.
[23] V. Kettunen, J. Kasurinen, O. Taipale, and K. Smolander, "A Study on Agility and Testing Processes in Software Organizations," Proc. 19th Int'l Symp. Software Testing and Analysis, pp. 231-240, 2010.
[24] P.N. Robillard, "The Role of Knowledge in Software Development," Comm. ACM, vol. 42, no. 1, pp. 87-92, 1999.
[25] "IEEE Standard Glossary of Software Engineering Terminology," IEEE Standard 610.12-1990, p. 83, 1990.
[26] D. Martin, J. Rooksby, M. Rouncefield, and I. Sommerville, "'Good' Organisational Reasons for 'Bad' Software Testing: An Ethnographic Study of Testing in a Small Software Company," Proc. Int'l Conf. Software Eng., pp. 602-611, 2007.
[27] J. Rooksby, M. Rouncefield, and I. Sommerville, "Testing in the Wild: The Social and Organisational Dimensions of Real World Practice," Computer Supported Cooperative Work, vol. 18, nos. 5/6, pp. 559-580, 2009.
[28] J.A. Whittaker, How to Break Software: A Practical Guide to Testing. Addison Wesley, 2003.
[29] L. Crispin and J. Gregory, Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley, 2009.
[30] J. Bach, "Session-Based Test Management," Software Testing and Quality Eng., vol. 2, no. 6, 2000.
[31] F. Houdek, T. Schwinn, and D. Ernst, "Defect Detection for Executable Specifications—an Experiment," Int'l J. Software Eng. and Knowledge Eng., vol. 12, no. 6, p. 637, Dec. 2002.
[32] J. Itkonen, M.V. Mäntylä, and C. Lassenius, "Defect Detection Efficiency: Test Case Based vs. Exploratory Testing," Proc. Int'l Symp. Empirical Software Eng. and Measurement, pp. 61-70, 2007.
[33] L.H.O. do Nascimento and P.D.L. Machado, "An Experimental Evaluation of Approaches to Feature Testing in the Mobile Phone Applications Domain," Proc. Workshop Domain Specific Approaches to Software Test Automation, pp. 27-33, 2007.
[34] J. Itkonen, M.V. Mäntylä, and C. Lassenius, "How Do Testers Do It? An Exploratory Study on Manual Testing Practices," Proc. Third Int'l Symp. Empirical Software Eng. and Measurement, pp. 494-497, 2009.
[35] J. Pichler and R. Ramler, "How to Test the Intangible Properties of Graphical User Interfaces?" Proc. First Int'l Conf. Software Testing, Verification, and Validation, pp. 494-497, 2008.
[36] J. Itkonen, K. Rautiainen, and C. Lassenius, "Toward an Understanding of Quality Assurance in Agile Software Development," Int'l J. Agile Manufacturing, vol. 8, no. 2, pp. 39-49, 2005.
[37] J. Kasurinen, O. Taipale, and K. Smolander, "Test Case Selection and Prioritization: Risk-Based or Design-Based?" Proc. Int'l Symp. Empirical Software Eng. and Measurement, article 10, 2010.
[38] A. Tinkham and C. Kaner, "Learning Styles and Exploratory Testing," Proc. Pacific Northwest Software Quality Conf., 2003.
[39] L. Shoaib, A. Nadeem, and A. Akbar, "An Empirical Evaluation of the Influence of Human Personality on Exploratory Software Testing," Proc. 13th IEEE Int'l Multitopic Conf., 2009.
[40] R.T. Turley and J.M. Bieman, "Competencies of Exceptional and Nonexceptional Software Engineers," J. Systems and Software, vol. 28, no. 1, pp. 19-38, Jan. 1995.
[41] B. Adelson and E. Soloway, "The Role of Domain Experience in Software Design," IEEE Trans. Software Eng., vol. 11, no. 11, pp. 1351-1360, Nov. 1985.
[42] S. Sonnentag, "Expertise in Professional Software Design: A Process Study." J. Applied Psychology, vol. 83, no. 5, pp. 703-715, 1998.
[43] J. Sandberg, "Understanding Human Competence at Work: An Interpretative Approach," Academy of Management J., vol. 43, no. 1, pp. 9-25, 2000.
[44] M. Kharlamov, A. Polovinkin, E. Kondrateva, and A. Lobachev, "Beyond Brute Force: Testing Financial Software," IT Professional, vol. 10, no. 3, pp. 14-18, 2008.
[45] C. Engelke and D. Olivier, "Putting Human Factors Engineering into Practice," Medical Device & Diagnostic Industry, vol. 24, no. 7, July 2002.
[46] R. Merkel and T. Kanij, "Does the Individual Matter in Software Testing?" Swinburne Univ. of Technology, Centre for Software Analysis and Testing, Technical Report 2010-001, May 2010.
[47] A. Følstad, "Work-Domain Experts as Evaluators: Usability Inspection of Domain-Specific Work-Support Systems," Int'l J. Human-Computer Interaction, vol. 22, no. 3, pp. 217-245, 2007.
[48] D.F. Galletta, D. Abraham, M.E. Louadi, W. Lekse, Y.A. Pollalis, and J.L. Sampler, "An Empirical Study of Spreadsheet Error-Finding Performance," Accounting, Management and Information Technologies, vol. 3, no. 2, pp. 79-95, 1993.
[49] W. Howden, "Theoretical and Empirical Studies of Program Testing," IEEE Trans. Software Eng., vol. 4, no. 4, pp. 293-298, July 1978.
[50] L. Baresi and M. Young, "Test Oracles," Technical Report CISTR- 01-02, Dept. of Computer and Information Science, Univ. of Oregon, Eugene, Aug. 2001.
[51] J.A. Whittaker, "What Is Software Testing? and Why Is It So Hard?" IEEE Software, vol. 17, no. 1, pp. 70-79, Jan./Feb. 2000.
[52] A. Memon, I. Banerjee, and A. Nagarajan, "What Test Oracle Should I Use for Effective GUI Testing?" Proc. 18th Int'l Conf. Automated Software Eng., pp. 164-173, 2003.
[53] V.R. Basili and R.W. Selby, "Comparing the Effectiveness of Software Testing Strategies," IEEE Trans. Software Eng., vol. 13, no. 12, pp. 1278-1296, Dec. 1987.
[54] E.J. Weyuker, "On Testing Non-Testable Programs," The Computer J., vol. 25, no. 4, pp. 465-470, 1982.
[55] M. Bolton, "Testing without a Map," Better Software, Jan. 2005.
[56] R. Chillarege, I. Bhandari, J. Chaar, M. Halliday, D. Moebus, B. Ray, and M. Wong, "Orthogonal Defect Classification—A Concept for In-Process Measurements," IEEE Trans. Software Eng., vol. 18, no. 11, pp. 943-956, Nov. 1992.
[57] M.V. Mäntylä and C. Lassenius, "What Types of Defects Are Really Discovered in Code Reviews?" IEEE Trans. Software Eng., vol. 35, no. 3, pp. 430-448, May/June 2009.
[58] A. Bondavalli and L. Simoncini, "Failure Classification with Respect to Detection," Proc. Second IEEE Workshop Future Trends of Distributed Computing Systems, pp. 47-53, 1990.
[59] D. Cotroneo, S. Orlando, and S. Russo, "Failure Classification and Analysis of the Java Virtual Machine," Proc. 26th IEEE Int'l Conf. Distributed Computing Systems, pp. 17-26, 2006.
[60] D.R. Wallace and D.R. Kuhn, "Failure Modes in Medical Device Software: An Analysis of 15 Years of Recall Data," Int'l J. Reliability, Quality and Safety Eng., vol. 8, no. 4, pp. 351-371, 2001.
[61] E.T. Hvannberg and L. Law, "Classification of Usability Problems (CUP) Scheme," Proc. Ninth Int'l Conf. Human-Computer Interaction, pp. 655-662, 2003.
[62] N. Juristo and S. Vegas, "Functional Testing, Structural Testing and Code Reading: What Fault Type Do They Each Detect?" Empirical Methods and Studies in Software Eng., Springer, 2003.
[63] B. Freimut, "Developing and Using Defect Classification Schemes," Research Report IESE-Report 072.01/E, Fraunhofer IESE, Kaiserslautern, 2001.
[64] D. Kuhn, D. Wallace, and A. Gallo, "Software Fault Interactions and Implications for Software Testing," IEEE Trans. Software Eng., vol. 30, no. 6, pp. 418-421, June 2004.
[65] C. Seaman, "Qualitative Methods in Empirical Studies of Software Engineering," IEEE Trans. Software Eng., vol. 25, no. 4, pp. 557-572, July/Aug. 1999.
[66] M.Q. Patton, Qualitative Research and Evaluation Methods, third ed. Sage, 2002.
[67] C. Seaman and V. Basili, "Communication and Organization: An Empirical Study of Discussion in Inspection Meetings," IEEE Trans. Software Eng., vol. 24, no. 7, pp. 559-572, July 1998.
[68] T.C. Lethbridge, S.E. Sim, and J. Singer, "Studying Software Engineers: Data Collection Techniques for Software Field Studies," Empirical Software Eng., vol. 10, no. 3, pp. 311-341, 2005.
[69] H. Wu, Y. Guo, and C.B. Seaman, "Analyzing Video Data: A Study of Programming Behavior under Two Software Engineering Paradigms," Proc. Int'l Symp. Empirical Software Eng. and Measurement, pp. 456-459, 2009.
[70] A. Höfer, "Video Analysis of Pair Programming," Proc. Int'l Workshop Scrutinizing Agile Practices or "Shoot-Out at the Agile Corral," pp. 37-41, 2008.
[71] L. Prechelt, U. Stärk, and S. Salinger, "7 Types of Cooperation Episodes in Side-by-Side Programming," Technical Report B-08-17, Freie Universität Berlin, Institut für Informatik, Germany, Dec. 2008.
[72] S. Salinger, L. Plonka, and L. Prechelt, "A Coding Scheme Development Methodology Using Grounded Theory for Qualitative Analysis of Pair Programming," Human Technology, vol. 4, no. 1, pp. 9-25, May 2008.
[73] J. Hughes and S. Parkes, "Trends in the Use of Verbal Protocol Analysis in Software Eng. Research," Behaviour & Information Technology, vol. 22, no. 2, p. 127, 2003.
[74] A.L. Strauss and J.M. Corbin, Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, 1998.
[75] J.C. van Niekerk and J.D. Roode, "Glaserian and Straussian Grounded Theory: Similar or Completely Different?" Proc. Ann. Research Conf. South African Inst. of Computer Scientists and Information Technologists, pp. 96-103, 2009.
[76] M.V. Mäntylä, J. Itkonen, and J. Iivonen, "Who Tested My Software? Testing as an Organizationally Cross-Cutting Activity," Software Quality J., vol. 20, no. 1, pp. 145-172, 2012.
6 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool