This Article 
 Bibliographic References 
 Add to: 
The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research
January 2000 (vol. 26 no. 1)
pp. 1-14

Abstract—Software engineers use a number of different types of software development technical review (SDTR) for the purpose of detecting defects in software products. This paper applies the behavioral theory of group performance to explain the outcomes of software reviews. A program of empirical research is developed, including propositions to both explain review performance and identify ways of improving review performance based on the specific strengths of individuals and groups. Its contributions are to clarify our understanding of what drives defect detection performance in SDTRs and to set an agenda for future research. In identifying individuals' task expertise as the primary driver of review performance, the research program suggests specific points of leverage for substantially improving review performance. It points to the importance of understanding software reading expertise and implies the need for a reconsideration of existing approaches to managing reviews.

[1] A.F. Ackerman, P.J. Fowler, and R. Ebenau, “Software Inspections and the Industrial Production of Software,” Software Validation, North-Holland, 1984.
[2] IEEE Standard for Software Reviews and Audits. Software Eng. Tech. Comm. of the IEEE Computer Society, 1989. IEEE Std 1028-1988.
[3] V.R. Basili, G. Caldiera, F. Lanubile, and F. Shull, “Studies on Reading Techniques,” Proc. 21st Ann. Software Eng. Workshop, Dec. 1996.
[4] V.R. Basili, S. Green, O. Laitenburger, F. Lanubile, F. Shull, S. Sorumgard, and M. Zelkowitz, “The Empirical Investigation of Perspective-Based Reading,” J. Empirical Software Eng., vol. 1, no. 2, 1996.
[5] D.B. Bisant and J.R. Lyle, A Two-Person Inspection Method to Improve Programming Productivity IEEE Trans. Software Eng., vol. 15, no. 10, pp. 1294-1304, Oct. 1989.
[6] P.C. Bottger and P.W. Yetton, “Improving Group Performance by Training in Individual Problem Solving,” J. Applied Psychology, vol. 72, no. 4, pp. 651-657, 1987.
[7] P.C. Bottger and P.W. Yetton, “An Integration of Process and Decision Scheme Explanations of Group Problem Solving Performance,” Organizational Behavior and Human Decision Processes, vol. 42, pp. 234-249, 1988.
[8] M.G. Bradac, D.E. Perry, and L.G. Votta, “Prototyping a Process Monitoring Experiment,” Proc. 15th Int'l Conf. Software Eng., May 1993.
[9] T.A. Corbi,“Program understanding: Challenge for the 1990s,” IBM Systems J., vol. 28, no. 2, pp. 294-306, 1989.
[10] A. Diehl and W. Stroebe, “Productivity Loss in Brainstorming Groups: Toward the Solution of a Riddle,” J. Personality and Social Psychology, vol. 53, no. 3, pp. 497-509, 1987.
[11] S.G. Eick, C.R. Loader, M.D. Long, L.G. Votta, and W.S. Vander, Estimating Software Fault Content Before Coding Proc. 14th Int'l Conf. Software Eng., pp. 49-65, 1992.
[12] H.J. Einhorn, R.M. Hogarth, and E. Klempner, “Quality of Group Judgement,” Psychological Bulletin, vol. 84, pp. 158-172, 1977.
[13] M.E. Fagan, “Design and Code Inspections to Reduce Errors in Program Development,” IBM Systems J., vol. 15, no. 3, pp. 182-211, 1976.
[14] M. Fagan, “Advances in Software Inspections,” IEEE Trans. Software Eng., vol. 12, no. 7, pp. 744–751, July 1986.
[15] P. Fowler, “In-Process Inspections of Workproducts at AT&T,” AT&T Technical J., vol. 65, no. 2, pp. 102–112, Mar. 1986.
[16] L.A. Franz and J.C. Shih, “Estimating the Value of Inspections for Early Testing of Software Projects,” Hewlett-Packard J., vol. 45, no. 6, Dec. 1994.
[17] D.P. Freedman and G.M. Weinberg, Handbook of Walkthroughs, Inspections, and Technical Reviews—Evaluating Programs, Projects, and Products. Dorset House, 1990.
[18] R.D. Galliers, Y. Merali, and L. Spearing, “Managing Information Technology? How British Executives Perceive the Key Issues,” J. Information Technology, vol. 9, no. 1, pp. 223-238, 1994.
[19] D.C. Ganster, P. Poppler, and S. Williams, “Does Training in Problem Solving Improve the Quality of Group Decisions?” J. Applied Psychology, vol. 76, no. 3, pp. 479-483, 1991.
[20] T. Gilb, Principles of Software Engineering Management, Addison-Wesley, Wokingham, U.K., 1988.
[21] R. Glazer, J.H. Steckel, and R.S. Winer, “Group Process and Decision Performance in a Simulated Marketing Environment,” J. Business Research, vol. 15, pp. 545-557, 1987.
[22] R.B. Grady, Practical Software Metrics for Project Management and Process Improvement, Prentice Hall, Englewood Cliffs, N.J., 1992.
[23] R.B. Grady and T. van Slack, "Key Lessons in Achieving Widespread Inspection Usage," IEEE Software, vol. 11, no. 4, 1994, pp. 46—57.
[24] H.A. Gurnee, “A Comparison of Collective and Group Judgements of Fact,” J. Experimental Psychology, vol. 3, pp. 437-444, 1937.
[25] J.R. Hackman and C.G. Morris, “Group Tasks, Group Interaction Process and Group Performance Effectiveness: A Review and Partial Integration,” Advances in Experimental Social Psychology, L. Berkowitz, ed., vol. 8, pp. 47-99, New York: Academic Press, 1975.
[26] J. Hall and W.H. Watson, “The Effects of a Normative Intervention on Group Decision-Making Performance,” Human Relations, vol. 23, pp. 299-317, 1970.
[27] R.A. Henry, “Improving Group Judgement Accuracy: Information Sharing and Determining the Best Member,” Organizational Behavior and Human Decision Processes, vol. 62, no. 2, pp. 190-197, May 1995.
[28] C.R. Holloman and H.W. Hendrick, “Individual versus Group Effectiveness in Solving Factual and Nonfactual Problems,” Proc. 78th Ann. Convention, Am. Psychological Assoc., 1970.
[29] P.M. Johnson and D. Tjahjono, “Assessing Software Review Meetings: A Controlled Experimental Study Using CSRS,” Technical Report 96-06, Dept. of Information and Computer Sciences, Univ. of Hawaii, 1996.
[30] L.P.W. Kim, C. Sauer, and R. Jeffery, “A Critical Survey of Software Development Technical Reviews as a Non-Method-Specific Approach to Software Quality Assurance,” ITRC Report #94/34, School of Information Systems, Univ. of New South Wales, Sydney, 1994.
[31] J. Knight and E.A. Myers, "An Improved Inspection Technique," Comm. ACM, vol. 36, no. 11, pp. 51-61, Nov. 1993.
[32] I. Lakatos, “Falsification and the Methodology of Scientific Research Programmes,” Criticism and the Growth of Knowledge, I. Lakatos and A. Musgrave, eds., Cambridge U.K.: Cambridge Univ. Press, 1974.
[33] F. Lanubile and G. Visaggio, “Assessing Defect Detection Methods for Software Requirements Inspections through External Replication,” Technical Report ISERN-96-01, Int'l Software Eng. Research Network, Jan. 1996.
[34] L.P.W. Land, C. Saucer, and R. Jeffery, ”Validating the Defect Detection Performance Advantage of Group Designs for Software Reviews: Report of a Laboratory Experiment Using Program Code,” Proc. European Software Eng. Conf. Foundations of Software Eng., pp. 294–309, Sep. 1997.
[35] P.R. Laughlin, N.L. Kerr, J.H. Davis, H.M. Halff, and K.A. Marcinak, “Group Size, Member Ability, and Social Decision Schemes on an Intellective Task,” J. Personality and Social Psychology, vol. 31, no. 3, pp. 522-535, 1975.
[36] R.E. Levine and R.L. Moreland, “Progress in Small Group Research,” Ann. Review of Psychology, vol. 1, 1990.
[37] R. Libby and R.K. Blashfield, “Performance of a Composite as a Function of the Number of Judges,” Organizational Behavior and Human Performance, vol. 21, pp. 121-129, 1978.
[38] R. Libby, K.T. Trotman, and I. Zimmer, “Member Variation, Recognition of Expertise, and Group Performance” J. Applied Psychology, vol. 72, pp. 81-87, 1987.
[39] Generalising from Laboratory to Field Settings: Research Findings from Industrial-Organizational Psychology, Organizational Behaviour, and Human Resource Management, E.A. Locke, ed., Lexington, Mass.: Heath-Lexington, 1985.
[40] I. Lorge and H. Solomon, “Two Models of Group Behaviour in the Solution of Eureka Type Problems,” Psychometrica, vol. 20, pp. 139-148, 1955.
[41] I. Lorge and H. Solomon, “Individual Performance and Group Performance in Problem Solving Related to Group Size and Previous Exposure to the Problem,” J. Psychology, vol. 48, pp. 107-114, 1959.
[42] I. Lorge and H. Solomon, “Group and Individual Performance in Problem Solving Related to Previous Exposure to the Problem, Level of Aspiration, and Group Size,” Behavioral Science, vol. 5, pp. 28-38, 1960.
[43] I. Lorge, D. Fox, J. Davitz, and M. Brenner, “A Survey of Studies Contrasting the Quality of Group Performance and Individual Performance,” Psychological Bulletin, vol. 55, pp. 337-371, 1958.
[44] F.J. Lukey, “Understanding and Debugging Programs,” Int'l J. Man-Machine Studies, vol. 12, pp. 189-202, 1980.
[45] F. Macdonald, J. Miller, A. Brooks, M. Roper, and M. Wood, “Automating the Software Inspection Process,” Proc. Seventh Int'l Workshop Computer-Aided Software Eng., CASE '95, July 1995.
[46] R. Madachy, L. Little, and S. Fan, “Analysis of a Successful Inspection Program,” Proc. Software Eng. Workshop, SEW, SEL-93-003, pp. 176-188, 1993.
[47] J. Martin and W.T. Tsai, N-Fold Inspection: A Requirements Analysis Technique Comm. ACM, vol. 33, no. 2, pp. 225-232, 1990.
[48] V. Mashayekhi, W. Tsai, J. Drake, and J. Riedl, “Distributed, Collaborative Software Inspection,” IEEE Software, vol. 10, no. 5, pp. 66-75, 1993.
[49] L.K. Michaelsen, W.E. Watson, and R.H. Black, “Realistic Test of Individual versus Group Decision Making,” J. Applied Psychology, vol. 74, pp. 834-839, 1989.
[50] G. Myers, “A Controlled Experiment in Program Testing and Code Walkthroughs/Inspections,” Comm. ACM, vol. 21, no. 9, pp. 760–768, Sept. 1978.
[51] F. Niederman, J.C. Brancheau, and J.C. Wetherbe, “Information Systems Management Issues for the 1990s,” MIS Quarterly, pp. 475-500, Dec. 1991.
[52] G.M. Olson, S. Sheppard, and E. Soloway, eds., "Empirical Studies of Programmers: Second Workshop,"Washington, D.C., Ablex Publishing Corp., Dec. 1987.
[53] D.L. Parnas and D.M. Weiss, "Active Design Reviews: Principles and Practices," Proc. Eighth Int'l Conf. Software Eng., pp. 215-222, Aug. 1985.
[54] N. Pennington, “Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs,” Cognitive Psychology, vol. 19, pp. 295-341, 1987.
[55] A.A. Porter, H. Siy, C.A. Toman, and L.G. Votta, “An Experiment to Assess the Cost-Benefits of Code Inspections in Large-Scale Software Development,”, 1996.
[56] A.A. Porter, L.G. Votta, and V.R. Basili, "An Experiment to Assess Different Defect Detection Methods for Software Requirements Inspections," Proc. 16th Int'l Conf. Software Eng., 1994, pp. 103-112.
[57] A.A. Porter, L.G. Votta, and V.R. Basili, “Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment,” IEEE Trans. Software Eng., vol. 21, no. 6, pp. 563-575, June 1995.
[58] Z.W. Pylyshyn, Computation and Cognition: Toward a Foundation for Cognitive Science. Cambridge, Mass: Bradford Books, MIT Press, 1984.
[59] F. Redmill, “Fagan's Inspection: Achieving Quality in Code and Documentation; Built-In Gauge of Effectiveness,” Managing System Development, vol. 13, no. 3, pp. 1-5, Mar. 1993.
[60] G.M. Schnieder, J. Martin, and W.T. Tsai, "An Experimental Study of Fault Detection in User Requirements," ACM Trans. Software Eng. and Methodology, vol. 1, no. 2, pp. 188-204, Apr. 1992.
[61] M.E. Shaw, “Comparison of Individuals and Small Groups in the Rational Solution of Complex Problems,” Am. J. Psychology, vol. 44, pp. 491-504, 1932.
[62] M.E. Shaw, Group Dynamics: The Psychology of Small Group Behavior. New York: McGraw-Hill, 1981.
[63] H. Siy, “Identifying the Mechanisms Driving Code Inspection Costs and Benefits,” PhD thesis, Univ. of Maryland, College Park, June 1996.
[64] Empirical Studies of Programmers, E. Soloway and S. Iyengar, eds. Norwood, N.J.: Ablex Publishing, 1986.
[65] I.D. Steiner, Group Process and Productivity. New York: Academic Press, 1972.
[66] S. Strauss and R. Ebenau, Software Inspection Process. McGraw-Hill, 1994.
[67] R.L. Thorndike, “In What Type of Task Does the Group Do Well?” J. Abnormal and Social Psychology, vol. 33, pp. 408-412, 1938.
[68] K.T. Trotman, “The Review Process and the Accuracy of Auditor Judgements,” J. Accounting Research, vol. 23, no. 2, pp. 740-752, 1985.
[69] K.T. Trotman and P.W. Yetton, “The Effect of the Review Process on Auditor Judgements,” J. Accounting Research, vol. 23, no. 1, pp. 256-267, 1985.
[70] K.T. Trotman, P.W. Yetton, and I.R. Zimmer, “Group Size and Performance: Prediction of Failure by Loan Officers,” Australian J. Management, pp. 127-136, 1981.
[71] K.T. Trotman, P.W. Yetton, and I.R. Zimmer, “Individual and Group Judgements of Internal Control Systems,” J. Accounting Research, vol. 21, no. 1, pp. 286-292, 1983.
[72] L.G. Votta, "Does Every Inspection Need a Meeting?" ACM Software Eng. Notes, vol. 18, no. 5, Dec. 1993, pp. 107-114.
[73] W. Watson, W. Sharp, and L.K. Michaelsen, “Member Competence, Group Interaction, and Group Decision Making: A Longitudinal Study,” J. Applied Psychology, vol. 76, no. 6, pp. 803-809, 1991.
[74] G.M. Weinberg and D.P. Freedman, “Reviews, Walkthroughs, and Inspections,” IEEE Trans. Software Eng., vol. 10, no. 1, pp. 68-72, Jan. 1984.
[75] E.F. Weller, "Lessons from Three Years of Inspection Data," IEEE Software, pp. 38-45, Sept. 1993.
[76] P.W. Yetton and P.C. Bottger, “Individual versus Group Problem Solving: An Empirical Test of a Best-Member Strategy,” Organizational Behavior and Human Performance, vol. 29, no. 3, pp. 307-321, June 1982.
[77] P.W. Yetton and P.C. Bottger, “The Relationship among Group Size, Member Ability, Social Decision Schemes, and Performance,” Organizational Behavior and Human Performance, vol. 32, pp. 145-159, 1983.
[78] E. Yourdon, Structured Walkthroughs, 4th ed., Prentice Hall, 1989.
[79] R.C. Ziller, “Group Size: A Determinant of the Quality and Stability of Group Decisions,” Sociometry, vol. 20, pp. 165-173, 1957.
[80] S. Rifkin and L. Deimel, “Applying Program Comprehension Techniques to Improve Software Inspections,” Proc. 19th Ann. NASA Software Laboratory Workshop, Nov. 1994.
[81] L. Land, C. Sauer, and R. Jeffery, “The Performance Effects of Process Roles in Code Reviews: A Preliminary Empirical Investigation,” Caesar Technical Report 97/7, Univ. of New South Wales, School of Information Systems, Sydney, 1997.
[82] L.P.W. Land, R. Jeffery, and C. Sauer, “Validating the Defect Detection Performance Advantage of Group Designs for Software Reviews: Report of a Replicated Experiment,” Proc. Australian Software Eng. Conf., P.A. Bailes ed., 1997.
[83] S.H. Kan, V.R. Basili, and L.N. Shapiro, “Software Quality: An Overview from the Perspective of Total Quality Management,” IBM Systems J., vol. 33, no. 1, pp. 4-18, 1994.
[84] E. Yourdon, “Quality: What It Means and How to Achieve It,” Management Information Science, pp. 43-47, Feb. 1993.
[85] M. Paulk et al., "Capability Maturity Model, Version 1.1," IEEE Software, July 1993, pp. 18-27.
[86] A. Onoma and T. Yamaura, "Practical Steps Toward Quality Development," IEEE Software, Sept. 1995, pp. 68-77.
[87] P. Hsia, “Learning to Put Lessons into Practice,” IEEE Software, pp. 14-17, Sept. 1993.
[88] T. Gilb and D. Graham, Software Inspection, Addison-Wesley, 1993.

Index Terms:
Inspections, walkthroughs, technical reviews, defects, defect detection, groups, group process, group size, expertise, reading, training, behavioral research, theory, research program.
Chris Sauer, D. Ross Jeffery, Lesley Land, Philip Yetton, "The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research," IEEE Transactions on Software Engineering, vol. 26, no. 1, pp. 1-14, Jan. 2000, doi:10.1109/32.825763
Usage of this product signifies your acceptance of the Terms of Use.