This Article 
 Bibliographic References 
 Add to: 
Reviewing Software Diagrams: A Cognitive Study
February 2004 (vol. 30 no. 2)
pp. 82-96
Bruce C. Hungerford, IEEE Computer Society

Abstract—Reviews and inspections of software artifacts throughout the development life cycle are effective techniques for identifying defects and improving software quality. While review methods for text-based artifacts (e.g., code) are well understood, very little guidance is available for performing reviews of software diagrams, which are rapidly becoming the dominant form of software specification and design. Drawing upon human cognitive theory, we study how 12 experienced software developers perform individual reviews on a software design containing two types of diagrams: entity-relationship diagrams and data flow diagrams. Verbal protocol methods are employed to describe and analyze defect search patterns among the software artifacts, both text and diagrams, within the design. Results indicate that search patterns that rapidly switch between the two design diagrams are the most effective. These findings support the cognitive theory thesis that how an individual processes information impacts processing success. We conclude with specific recommendations for improving the practice of reviewing software diagrams.

[1] M. Anderson, Some Evidence on the Effect of Verbalization on Process: A Methodological Note J. Accounting Research, vol. 23, no. 2, pp. 843-852, Autumn 1985.
[2] R. Baeker, C. DiFiano, and A. Marcus, Software Visualization for Debugging Comm. ACM, vol. 40, no. 4, pp. 44-54, Apr. 1997.
[3] P. Barber, Applied Cognitive Psychology. London: Methuen, 1988.
[4] S. Biffl and M. Halling, Investigating the Defect Detection Effectiveness and Cost Benefit of Nominal Inspection Teams IEEE Trans. Software Eng., vol. 29, no. 5, pp. 385-397, May 2003.
[5] G. Booch, J. Rumbaugh, and I. Jacobson, The Unified Modeling Language User Guide. Addison-Wesley, 1999.
[6] W. Chase and H. Simon, Perception in Chess Cognitive Psychology, vol. 4, pp. 55-81, 1973.
[7] M. Chi, P. Feltovich, and R. Glaser, Categorization and Representation of Physics Problems by Experts and Novices Cognitive Science, vol. 5, pp. 121-152, 1981.
[8] G. Chroust and H. Lexen, Software Inspections Theory, New Approaches and an Experiment Proc. 25th EUROMICRO Conf., vol. 2, pp. 286-293, Sept. 1999.
[9] N. Cowan, Attention and Memory: An Integrated Framework. Oxford Univ. Press and Clarendon Press, New York and Oxford, 1995.
[10] D. Damos and C. Wickens, The Identification and Transfer of Timesharing Skills Acta Psychologica, vol. 46, pp. 15-39, 1980.
[11] K. Ericsson and H. Simon, Protocol Analysis: Verbal Reports as Data. MIT Press, 1993.
[12] D. Flynn and O. Fragosa Diaz, Information Modelling: An International Perspective. Prentice Hall, 1996.
[13] C. Frederiksen and A. Breuleux, Monitoring Cognitive Processing in Semantically Complex Domains Diagnostic Monitoring of Skill and Knowledge Acquisition, N. Frederiksen, R. Glaser, A. Lesgold, and M. Shafto, eds., pp. 351-391, Hillsdale, N.J.: Lawrence Erlbaum, 1990.
[14] P. Gibbins, Logic with Prolog. Clarendon Press, 1988.
[15] Diagrammatic Reasoning: Cognitive and Computational Perspectives. J. Glasgow, N. Narayanan, and B.Chandrasekaran, eds., MIT Press, 1995.
[16] T.R.G. Green and M. Petre, Usability Analysis of Visual Programming Environments: A‘Cognitive Dimensions’Framework J. Visual Languages and Computing, vol. 7, pp. 131-174, 1996.
[17] T. Hendrix, J. Cross II, S. Maghsoodloo, and M. McKinney, Do Visualizations Improve Program Comprehensibility? Experiments with Control Structure Diagrams for Java Proc. 31st SIGCSE Technical Symp. Computer Science Education, pp. 382-386, 2000.
[18] W. James, The Principles of Psychology. New York: Henry Holt, 1890.
[19] J. Kim, J. Hahn, and H. Hahn, How do we Understand a System with (So) Many Diagrams? Cognitive Integration Processes in Diagrammatic Reasoning Information Systems Research, vol. 11, no. 3, pp. 284-303, Sept. 2000.
[20] G. Klersey and T. Mock, Verbal Protocol Research in Auditing Accounting Organizations and Soc., vol. 14, nos. 1/2, pp. 133-151, 1989.
[21] L.P.W. Land, R. Jeffery, and C. Sauer, Validating the Defect Detection Performance Advantage of Group Designs for Software Reviews: Report of a Replicated Experiment Proc. Software Eng. Conf., pp. 17-26, Sept./Oct. 1997.
[22] J. Larkin and H. Simon, Why a Diagram is (Sometimes) Worth Ten Thousand Words Cognitive Science, vol. 11, pp. 65-99, 1987.
[23] G.L. Lohse, The Role of Working Memory on Graphical Information Processing Behaviour&Information Technology, vol. 16, no. 6, pp. 297-308, 1997.
[24] J. Mao and I. Benbasat, The Use of Explanations in Knowledge-Based Systems: Cognitive Perpectives and a Process-Tracing Analysis J. Management Information Systems, vol. 17, pp. 153-179, Fall 2000.
[25] J. Martin and C. McClure, Structured Techniques for Computing. Prentice-Hall, 1985.
[26] R. Mayer, From Novice to Expert Handbook of Human-Computer Interaction, second ed., M. Helander, T. Landauer, and P. Prabhu, eds., London: Elsevier Science, pp. 781-795, 1997.
[27] F. McDonald and J. Miller, A Comparison of Computer Support Systems for Software Inspection Automated Software Eng., vol. 6, pp. 291-313, 1999.
[28] J. McGrath, Dilemmatics: The Study of Research Choices and Dilemmas Judgment Calls in Research, J. McGrath, J. Martin, and R. Kulka, eds., Beverly Hills, Calif.: Sage, 1982.
[29] A. Newell and H.A. Simon, Human Problem Solving. Englewood Cliffs N.J.: Prentice-Hall, 1972.
[30] D. Norman, Learning and Memory. San Francisco, Calif.: W.H. Freeman, 1982.
[31] J.W. Payne, M.L. Braunstein, and J.S. Carroll, Exploring Predecisional Behavior: An Alternative Approach to Decision Research Organizational Behavior and Human Performance, vol. 22, pp. 17-44, 1978.
[32] M. Petre, Why Looking Isn't Always Seeing: Readership Skills and Graphical Programming Comm. ACM, vol. 38, no. 6, pp. 33-44, June 1995.
[33] A.A. Porter, H. Siy, A. Mockus, and L.G. Votta, Understanding the Sources of Variation in Software Inspections ACM Trans. Software Eng. and Methodology, vol. 7, pp. 41-79, Jan. 1998.
[34] B. Regnell, P. Runeson, and T. Thelin, Are the Perspectives Really Different? Further Experimentation on Scenario-Based Reading of Requirements Empirical Software Eng., vol. 5, pp. 331-356, 2000.
[35] P. Reimann and M. Chi, Human Expertise Human and Machine Problem Solving, K. Gilhooly, ed., pp. 161-191, New York: Plenum Press, 1989.
[36] E. Rogers, A Study of Visual Reasoning in Medical Diagnosis Proc. 18th Ann. Conf. Cognitive Science Soc., pp. 213-218, 1996.
[37] C.D. Rombach, O. Kude, A. Aurum, R. Jeffery, and C. Wohlin, An Empirical Study of an ER-Model Inspection Meeting Proc. 29th Euromicro Conf., Sept. 2003.
[38] J. Russo, E. Johnson, and D. Stephens, The Validity of Verbal Protocols Memory&Cognition, vol. 17, no. 6, pp. 759-769, 1989.
[39] C. Sauer, D.R. Jeffery, L. Land, and P. Yetton, “The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research,” IEEE Trans. Software Eng., vol. 26, no. 1, pp. 1-14, 2000.
[40] G.M. Schneider, J. Martin, and W.T. Tsai, An Experimental Study of Fault Detection in User Requirements Documents ACM Trans. Software Eng. and Methodology, vol. 1, no. 1, pp. 188-204, Apr. 1992.
[41] D. Schweiger, Is the Simultaneous Verbal Protocol a Viable Method for Studying Managerial Problem Solving and Decision Making? Academy of Management J., vol. 26, no. 1, pp. 185-192, Mar. 1983.
[42] R.M. Shiffrin, Attention Stevens' Handbook of Experimental Psychology, second ed., vol. 2, R.C. Atkinson et al., eds., John Wiley&Sons, 1988.
[43] F. Shull, V. Basili, B. Boehm, A.W. Brown, P. Costa, M. Lindvall, D. Port, I. Rus, R. Tesoriero, and M. Zelkowitz, What we Have Learned about Fighting Defects Proc. Eighth IEEE Symp. Software Metrics, 2002.
[44] H. Simon, Invariants of Human Behavior Am. Rev. of Psychology, vol. 41, pp. 1-19, 1990.
[45] H.E.A. Tinsley and D.J. Weiss, Interrater Reliability and Agreement of Subjective Judgments J. Counseling Psychology, vol. 22, pp. 358-376, 1975.
[46] E. Tolman, Principles of Purposive Behaviorism Psychology, A Study of a Science, vol. 2, S. Koch, ed., McGraw-Hill, 1959.
[47] G.H. Travassos, F. Shull, M. Fredericks, and V.R. Basili, Detecting Defects in Object Oriented Designs: Using Reading Techniques to Increase Software Quality Proc. OOPSLA, pp. 47-56, 1999.
[48] E. Tufte, Visual Explanations. Cheshire, Conn.: Graphics Press, 1997.
[49] L. Votta, Does Every Inspection Need a Meeting? Proc. First ACM SIGSOFT Symp. Software Development Eng., 1993.
[50] Software Inspection: An Industry Best Practice. D. Wheeler, B. Brykczynski, and R. Meeson, eds., IEEE Computer Soc. Press, 1996.
[51] C.D. Wickens, Processing Resources in Attention Varieties of Attention, R. Parasuraman and R. Davies, eds., Academic Press, 1984.
[52] C.D. Wickens and C.M. Carswell, The Proximity Compatibility Principle: Its Psychological Function and Relevance to Display Design Human Factors, vol. 37, pp. 473-494, 1995.
[53] R. Wieringa, A Survey of Structured and Object-Oriented Software Specification Methods and Techniques ACM Computing Surveys, vol. 30, no. 4, pp. 459-527, Dec. 1998.
[54] W. Winn, Contributions of Perceptual and Cognitive Processes to the Comprehension of Graphics Comprehension of Graphics, W. Schnotz and R. Kulhavy, eds., North-Holland, 1994.
[55] C. Wohlin and A. Aurum, An Evaluation of Checklist-Based Reading for Entity-Relationship Diagrams Proc. Ninth IEEE Int'l Software Metrics Symp., pp. 286-296, Sept. 2003.
[56] C. Wohlin, A. Aurum, H. Petersson, F. Shull, and M. Ciolkowski, Software Inspection Benchmarking A Qualitative and Quantitative Comparative Opportunity Proc. Eighth IEEE Symp. Software Metrics, 2002.
[57] J. Zhang, The Nature of External Representations in Problem Solving Cognitive Science, vol. 21, no. 2, pp. 179-217, 1997.

Index Terms:
Defects, human cognitive theory, inspections, software diagrams, software quality, software reviews, verbal protocol methods.
Bruce C. Hungerford, Alan R. Hevner, Rosann W. Collins, "Reviewing Software Diagrams: A Cognitive Study," IEEE Transactions on Software Engineering, vol. 30, no. 2, pp. 82-96, Feb. 2004, doi:10.1109/TSE.2004.1265814
Usage of this product signifies your acceptance of the Terms of Use.