This Article 
 Bibliographic References 
 Add to: 
Two Controlled Experiments Assessing the Usefulness of Design Pattern Documentation in Program Maintenance
June 2002 (vol. 28 no. 6)
pp. 595-606

Abstract—Using design patterns is claimed to improve programmer productivity and software quality. Such improvements may manifest both at construction time (in faster and better program design) and at maintenance time (in faster and more accurate program comprehension). This paper focuses on the maintenance context and reports on experimental tests of the following question: Does it help the maintainer if the design patterns in the program code are documented explicitly (using source code comments) compared to a well-commented program without explicit reference to design patterns? Subjects performed maintenance tasks on two programs ranging from 360 to 560 LOC including comments. Both programs contained design patterns. The controlled variable was whether the use of design patterns was documented explicitly or not. The experiments thus tested whether pattern comment lines (PCL) help during maintenance if patterns are relevant and sufficient program comments are already present. It turns out that this question is a challenge for the experimental methodology: A setup leading to relevant results is quite difficult to find. We discuss these issues in detail and suggest a general approach to such situations. The experiment was performed with Java by 74 German graduate students and then repeated with C++ by 22 American undergraduate students. A conservative analysis of the results supports the hypothesis that pattern-relevant maintenance tasks were completed faster or with fewer errors if redundant design pattern information was provided. Redundant means that the information carried in pattern comments is also available in different form in other comments. The contribution of this article is twofold: It provides the first controlled experiment results on design pattern usage and it presents a solution approach to an important class of experiment design problems for experiments regarding documentation.

[1] K. Beck, J.O. Coplien, R. Crocker, L. Dominick, G. Meszaros, F. Paulisch, and J. Vlissides, “Industrial Experience with Design Patterns,” Proc. 18th Int'l Conf. Software Eng., pp. 103–114, Mar. 1996.
[2] R. Brooks, “Using a Behavioral Theory of Program Comprehension in Software Engineering,” Proc. Third Int'l Conf. Software Eng., pp. 196–201, 1978.
[3] R. Brooks, “Towards a Theory of the Comprehension of Computer Programs,” Int'l J. Man-Machine Studies, vol. 18, no. 6, pp. 543–554, June 1983.
[4] F. Buschmann et al., Pattern-Oriented Software Architecture: A System of Patterns, John Wiley, Chichester, UK, 1996.
[5] L.B. Christensen, Experimental Methodology, sixth ed. Needham Heights, Mass.: Allyn and Bacon, 1994.
[6] Pattern Languages of Program Design. J.O. Coplien and D.C. Schmidt, eds. Monticello, Ill.: Addison-Wesley, 1995.
[7] B. Efron and R. Tibshirani, An Introduction to the Bootstrap. New York: Chapman and Hall, 1993.
[8] EIR, Web pages of the Karlsruhe Empirical Informatics Research Group.http://wwwipd.ira.uka.deEIR/, 2002.
[9] E. Gamma et al., Design Patterns: Elements of Object-Oriented Software, Addison-Wesley, Reading, Mass., 1994.
[10] E.M. Gellenbeck and C.R. Cook, “An Investigation of Procedure and Variable Names as Beacons During Program Comprehension,” Empirical Studies of Programmers: Proc. Fourth Workshop, pp. 65–81, 1991.
[11] O. Gramberg, “Counting the Use of Software Design Patterns in Java AWT and NeXTstep,” Technical Report 19/1997, Fakultät für Informatik, Univ. Karlsruhe, Germany, Dec. 1997,
[12] Empirical Studies of Programmers: Proc. Fourth Workshop, J. Koenemann-Belliveau, T.G. Mohrer and S.P. Robertson, eds., Dec. 1991.
[13] S. Letovsky, “Cognitive Processes in Program Comprehension,” Proc. First Workshop Empirical Studies of Programmers, Ablex Publishing, Norwood, N.J., 1986, pp. 58-79.
[14] Pattern Languages of Program Design 3, R. Martin, D. Riehle, and F. Buschmann, eds. Monticello, Ill.: Addison-Wesley, 1997.
[15] N. Pennington, “Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs,” Cognitive Psychology, vol. 19, pp. 295–341, 1987.
[16] D. E. Perry and A. L. Wolf,“Foundations for the study of software architecture,”inACM SIGSOFT Software Eng. Notes, Oct. 1992, vol. 17, no. 4, pp. 40–52.
[17] L. Prechelt, “An Experiment on the Usefulness of Design Patterns: Detailed Description and Evaluation,” Technical Report 9/1997, Fakultät für Informatik, Univ. Karlsruhe, Germany, June 1997,
[18] L. Prechelt, B. Unger, and D. Schmidt, “Replication of the First Controlled Experiment on the Usefulness of Design Patterns: Detailed Description and Evaluation,” Technical Report wucs-97-34, Washington Univ., Dept. of Computer Science, St. Louis, Dec. 1997.
[19] M. Shaw and D. Garlan, Software Architecture: Perspectives on an Emerging Discipline. Prentice Hall, 1996.
[20] B. Shneiderman and R. Mayer, “Syntactic/Semantic Interactions in Programmer Behavior: A Model and Experimental Results,” Int'l J. Computer and Information Sciences, vol. 8, no. 3, pp. 219–238, 1979.
[21] E. Soloway, J. Pinto, S. Letovsky, D. Littman, and R. Lampert, “Designing Documentation to Compensate for Delocalized Plans,” Comm. ACM, vol. 31, no. 11, pp. 1259–1267, Nov. 1988.
[22] Empirical Studies of Programmers: Proc. First Workshop, E. Soloway and S. Iyengar, eds., June 1986.
[23] B. Unger and W.F. Tichy, “Do Design Patterns Improve Communication? An Experiment with Pair Design,” Proc. Int'l Workshop Empirical Studies of Software Maintenance, George Stark, ed., , Oct. 2000.
[24] W.N. Venables and B.D. Ripley, Modern Applied Statistics with S-PLUS, second ed. Springer-Verlag, 1997.
[25] Pattern Languages of Program Design 2, J.M. Vlissides, J.O. Coplien, and N.L. Kerth, eds. Monticello, Ill.: Addison-Wesley, 1996.
[26] A. von Mayrhauser and S. Lang, “A Coding Scheme to Support Systematic Analysis of Software Comprehension,” IEEE Trans. Software Eng., vol. 25, no. 4, pp. 526–540, July/Aug. 1999.
[27] N. Wilde, P. Matthews, and R. Huitt, “Maintaining Object-Oriented Software,” IEEE Software, vol. 10, no. 1, pp. 75–80, Jan. 1993.
[28] Proc. Fifth Int'l Workshop Program Comprehension, Mar. 1997.
[29] Proc. Sixth Int'l Workshop Program Comprehension, June 1998.
[30] Proc. Seventh Int'l Workshop Program Comprehension, May 1999.
[31] Proc. Eighth Int'l Workshop Program Comprehension, June 2000.

Index Terms:
Controlled experiment, design pattern, comments, documentation, maintenance.
Lutz Prechelt, Barbara Unger-Lamprecht, Michael Philippsen, Walter F. Tichy, "Two Controlled Experiments Assessing the Usefulness of Design Pattern Documentation in Program Maintenance," IEEE Transactions on Software Engineering, vol. 28, no. 6, pp. 595-606, June 2002, doi:10.1109/TSE.2002.1010061
Usage of this product signifies your acceptance of the Terms of Use.