This Article 
 Bibliographic References 
 Add to: 
An Experimental Comparison of Usage-Based and Checklist-Based Reading
August 2003 (vol. 29 no. 8)
pp. 687-704

Abstract—Software quality can be defined as the customers' perception of how a system works. Inspection is a method to monitor and control the quality throughout the development cycle. Reading techniques applied to inspections help reviewers to stay focused on the important parts of an artifact when inspecting. However, many reading techniques focus on finding as many faults as possible, regardless of their importance. Usage-based reading helps reviewers to focus on the most important parts of a software artifact from a user's point of view. This paper presents an experiment which compares usage-based and checklist-based reading. The results show that reviewers applying usage-based reading are more efficient and effective in detecting the most critical faults from a user's point of view than reviewers using checklist-based reading. Usage-based reading may be preferable for software organizations that utilize or will start utilizing use cases in their software development.

[1] V.R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sørumgård, and M.V. Zelkowitz, The Empirical Investigation of Perspective-Based Reading Empirical Software Eng.: An Int'l J., vol. 1, no. 2, pp. 133-164, 1996.
[2] V.R. Basili, F. Shull, and F. Lanubile, Building Knowledge through Families of Experiments IEEE Trans. Software Eng., vol. 25, no. 4, pp. 456-473, July/Aug. 1999.
[3] S. Biffl, Software Inspection Techniques to Support Project and Quality Management. Austria: Habilitationsschrift, Shaker Verlag, 2001.
[4] D.B. Bisant and J.R. Lyle, A Two-Person Inspection Method to Improve Programming Productivity IEEE Trans. Software Eng., vol. 15, no. 10, pp. 1294-1304, Oct. 1989.
[5] M. Ciolkowski, C. Differding, O. Laitenberger, and J. Münch, Empirical Investigation of Perspective-Based Reading: A Replicated Experiment ISERN Report no. 97-13, 1997.
[6] S. Strauss and R. Ebenau, Software Inspection Process. McGraw-Hill, 1994.
[7] S.G. Eick, C.R. Loader, M.D. Long, L.G. Votta, and W.S. Vander, Estimating Software Fault Content Before Coding Proc. 14th Int'l Conf. Software Eng., pp. 49-65, 1992.
[8] M.E. Fagan, Design and Code Inspections to Reduce Errors in Program Development IBM System J., vol. 15, no. 3, pp. 182-211, 1976.
[9] P. Fusaro and F. Lanubile, “A Replicated Experiment to Assess Requirements Inspection Techniques,” Empirical Software Eng., vol. 2, no. 1, pp. 39–57, 1997.
[10] T. Gilb and D. Graham, Software Inspection, Addison-Wesley, 1993.
[11] M. Halling, S. Biffl, T. Grechenig, and M. Köhle, Using Reading Techniques to Focus Inspection Performance Proc. 27th Euromicro Workshop Software Process and Product Improvement, pp. 248-257, 2001.
[12] W. Hayes, “Research Synthesis in Software Engineering: A Case for Meta-Analysis,” Proc. Int'l Symp. Software Metrics, 1999.
[13] ITU-T Z.100, Specification and Description Language, SDL, ITU-T Recommendation Z.100, 1993.
[14] ITU-T Z.120, Message Sequence Charts, MSC, ITU-T Recommendation Z.120, 1996.
[15] I. Jacobson, M. Christerson, P. Jonsson, and G. Övergaard, Object-Oriented Software Engineering: A Use Case Driven Approach. Addison-Wesley, 1992.
[16] N. Juristo and A.M. Moreno, Basics of Software Engineering Experimentation. Kluwer Academic, 2001.
[17] J. Knight and E.A. Myers, "An Improved Inspection Technique," Comm. ACM, vol. 36, no. 11, pp. 51-61, Nov. 1993.
[18] O. Laitenberger and J.-M. DeBaud, Perspective-Based Reading of Code Documents at Robert Bosch GmbH Information and Software Technology, vol. 39, no. 11, pp. 781-791, 1997.
[19] O. Laitenberger and C. Atkinson, “Generalizing Perspective-Based Inspection to Handle Object-Oriented Development Artifacts,” Proc. Int'l Conf. Software Eng., 1999.
[20] O. Laitenberger and J. DeBaud, "An Encompassing Life-Cycle Centric Survey of Software Review," J. Systems and Software, vol. 50, no. 1, Jan. 2000, pp. 5-31.
[21] O. Laitenberger, C. Atkinson, M. Schlich, and K. El Emam, An Experimental Comparison of Reading Techniques for Defect Detection in UML Design Documents J. Systems and Software, vol. 53, no. 2, pp. 183-204, 2000.
[22] F. Lanubile and G. Visaggio, Evaluating Defect Detection Techniques for Software Requirements Inspections Technical Report no. 00-08, ISERN, 2000.
[23] S. Lauesen, Software Requirements Styles and Techniques. Addison-Wesley, 2002.
[24] R.C. Linger, Cleanroom Process Model IEEE Software, vol. 11, no. 2, pp. 50-58, 1994.
[25] J. Martin and W.T. Tsai, N-Fold Inspection: A Requirements Analysis Technique Comm. ACM, vol. 33, no. 2, pp. 225-232, 1990.
[26] J. Miller, M. Wood, and M. Roper, “Further Experiences with Scenarios and Checklists,” Empirical Software Eng., vol. 3, no. 1, pp. 37–64, 1998.
[27] D. Montgomery, Design and Analysis of Experiments. John Wiley&Sons, 2000.
[28] J.D. Musa, "Operational Profiles in Software Reliability Engineering," IEEE Software, vol. 10, no. 2, pp. 14-32, 1993.
[29] J.D. Musa, Software Reliability Engineering: More Reliable Software, Faster Development and Testing. McGraw-Hill, 1998.
[30] M. Olofsson and M. Wennberg, Statistical Usage Inspection master's thesis, Dept. of Comm. Systems, Lund Univ., CODEN: LUTEDX (TETS-5244)/1-81/(1996)&local 9, 1996.
[31] D.L. Parnas and D.M. Weiss, "Active Design Reviews: Principles and Practices," Proc. Eighth Int'l Conf. Software Eng., pp. 215-222, Aug. 1985.
[32] H. Petersson, C. Wohlin, and A. Aurum, Effectiveness of Software Inspections as a Guiding Tool to Regulate Team Size to be published.
[33] A.A. Porter, L.G. Votta, and V.R. Basili, "An Experiment to Assess Different Defect Detection Methods for Software Requirements Inspections," Proc. 16th Int'l Conf. Software Eng., 1994, pp. 103-112.
[34] A.A. Porter, L.G. Votta, and V.R. Basili, “Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment,” IEEE Trans. Software Eng., vol. 21, no. 6, pp. 563-575, June 1995.
[35] A. Porter and L. Votta, “Comparing Detection Methods for Software Requirements Inspections: A Replication Using Professional Subjects,” Empirical Software Eng., vol. 3, pp. 355–379, 1998.
[36] B. Regnell, P. Runeson, and T. Thelin, Are the Perspectives Really Different? Further Experimentation on Scenario-Based Reading of Requirements Empirical Software Eng.: An Int'l J., vol. 5, no. 4, pp. 331-356, 2000.
[37] C. Robson, Real World Research. U.K.: Blackwell, 2002.
[38] P. Runeson and C. Wohlin, Statistical Usage Testing for Software Reliability Control Informatica, vol. 19, no. 2, pp. 195-207, 1995.
[39] T.L. Saaty and L.G. Vargas, Models, Methods, Concepts&Applications of the Analytic Hierarchy Process. Kluwer Academic, 2001.
[40] K. Sandahl, O. Blomkvist, J. Karlsson, C. Krysander, M. Lindvall, and N. Ohlsson, “An Extended Replication of an Experiment for Assessing Methods for Software Requirements Inspections,” Empirical Software Eng., vol. 3 pp. 327–254, 1998.
[41] F.J. Shull, “Developing Techniques for Using Software Documents: A Series of Empirical Studies,” PhD thesis, Univ. of Maryland, College Park,, 1998.
[42] F. Shull, I. Rus, and V. Basili, "How Perspective-Based Reading Can Improve Requirements Inspections," Computer, vol. 33, no. 7, July 2000, pp. 73-79.
[43] S. Siegel and N.J. Castellan, Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, 1988.
[44] S. Sørumgård, Verification of Process Conformance in Empirical Studies of Software Development PhD thesis, Dept. of Computer and Information Science, Norwegian Univ. of Science and Tech nology, 1997.
[45] T. Thelin, P. Runeson, and B. Regnell, Usage-Based Reading An Experiment to Guide Reviewers with Use Cases Information and Software Technology, vol. 43, no. 15, pp. 925-938, 2001.
[46] T. Thelin, P. Runeson, C. Wohlin, T. Olsson, and C. Andersson, How Much Information Is Needed for Usage-Based Reading? A Series of Experiments Proc. First Int'l Symp. Empirical Software Eng., pp. 127-138, 2002.
[47] G. Travassos, F. Shull, M. Fredericks, and V. Basili, “Detecting Defects in Object-Oriented Designs: Using Reading Techniques to Increase Software Quality,” Proc. Conf. Object-Oriented Programming Systems, Languages, and Applications (OOPSLA), 1999.
[48] L.G. Votta, "Does Every Inspection Need a Meeting?" ACM Software Eng. Notes, vol. 18, no. 5, Dec. 1993, pp. 107-114.
[49] E.F. Weller, "Lessons from Three Years of Inspection Data," IEEE Software, pp. 38-45, Sept. 1993.
[50] C. Wohlin, B. Regnell, A. Wesslén, and H. Cosmo, User-Centered Software Engineering A Comprehensive View of Software Development Proc. Nordic Seminar on Dependable Computing Systems, pp. 229-240, 1994.
[51] C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering: An Introduction. Kluwer Academic, 2000.
[52] Z. Zhang, V.R. Basili, and B. Schneiderman, Perspective-Based Usability Inspection: An Empirical Validation of Efficacy Empirical Software Eng.: An Int'l J., vol. 4, no. 1, pp. 43-69, 1999.

Index Terms:
Controlled experiment, empirical study, reading technique, software inspection, software review.
Thomas Thelin, Per Runeson, Claes Wohlin, "An Experimental Comparison of Usage-Based and Checklist-Based Reading," IEEE Transactions on Software Engineering, vol. 29, no. 8, pp. 687-704, Aug. 2003, doi:10.1109/TSE.2003.1223644
Usage of this product signifies your acceptance of the Terms of Use.