The Community for Technology Leaders
RSS Icon
Issue No.09 - Sept. (2013 vol.39)
pp: 1264-1282
Walid Maalej , University of Hamburg, Germany
Martin P. Robillard , McGill University, Montréal
Reading reference documentation is an important part of programming with application programming interfaces (APIs). Reference documentation complements the API by providing information not obvious from the API syntax. To improve the quality of reference documentation and the efficiency with which the relevant information it contains can be accessed, we must first understand its content. We report on a study of the nature and organization of knowledge contained in the reference documentation of the hundreds of APIs provided as a part of two major technology platforms: Java SDK 6 and .NET 4.0. Our study involved the development of a taxonomy of knowledge types based on grounded methods and independent empirical validation. Seventeen trained coders used the taxonomy to rate a total of 5,574 randomly sampled documentation units to assess the knowledge they contain. Our results provide a comprehensive perspective on the patterns of knowledge in API documentation: observations about the types of knowledge it contains and how this knowledge is distributed throughout the documentation. The taxonomy and patterns of knowledge we present in this paper can be used to help practitioners evaluate the content of their API documentation, better organize their documentation, and limit the amount of low-value content. They also provide a vocabulary that can help structure and facilitate discussions about the content of APIs.
Documentation, Taxonomy, Encoding, Reliability, Java, Software, Sociology, .NET, API documentation, software documentation, empirical study, content analysis, grounded method, data mining, pattern mining, Java
Walid Maalej, Martin P. Robillard, "Patterns of Knowledge in API Reference Documentation", IEEE Transactions on Software Engineering, vol.39, no. 9, pp. 1264-1282, Sept. 2013, doi:10.1109/TSE.2013.12
[1] G. Butler, P. Grogono, and F. Khendek, "A Reuse Case Perspective on Documenting Frameworks," Proc. Fifth Asia Pacific Software Eng. Conf., pp. 94-101, 1998.
[2] G. Butler, R.K. Keller, and H. Mili, "A Framework for Framework Documentation," ACM Computing Surveys, vol. 32, Mar. 2000.
[3] J. Corbin and A. Strauss, Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, third ed. Sage Publications, 2007.
[4] B. Dagenais and M.P. Robillard, "Creating and Evolving Developer Documentation: Understanding the Decisions of Open Source Contributors," Proc. 18th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 127-136, Nov. 2010.
[5] C.R.B. de Souza, D. Redmiles, L.-T. Cheng, D. Millen, and J. Patterson, "How a Good Software Practice Thwarts Collaboration: The Multiple Roles of APIs in Software Development," Proc. 12th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 221-230, 2004.
[6] B. Ellis, J. Stylos, and B. Myers, "The Factory Pattern in API Design: A Usability Evaluation," Proc. 29th ACM/IEEE Int'l Conf. Software Eng., pp. 302-312, May 2007.
[7] A. Erdem, W. Johnson, and S. Marsella, "Task Oriented Software Understanding," Proc. 13th IEEE Int'l Conf. Automated Software Eng., pp. 230-239, 1998.
[8] K. Erdös and H. Sneed, "Partial Comprehension of Complex Programs (Enough to Perform Maintenance)," Proc. Sixth Int'l Workshop Program Comprehension, pp. 98-105, 1998.
[9] T. Fritz and G.C. Murphy, "Using Information Fragments to Answer the Questions Developers Ask," Proc. 32nd ACM/IEEE Int'l Conf. Software Eng., pp. 175-184, 2010.
[10] F.J. Gravetter and L.B. Wallnau, Essentials of Statistics for the Behavioral Sciences, fifth ed. Thomson Wadsworth, 2005.
[11] M. Gunderloy, Understanding and Using Assemblies and Namespaces in .NET, Microsoft Corp., . 2002.
[12] M. Hahsler, B. Gruen, and K. Hornik, "arules—A Computational Environment for Mining Association Rules and Frequent Item Sets," J. Statistical Software, vol. 14, no. 15, pp. 1-25, Oct. 2005.
[13] J.D. Herbsleb and E. Kuwana, "Preserving Knowledge in Design Projects: What Designers Need to Know," Proc. Joint INTERACT '93 and CHI '93 Conf. Human Factors in Computing Systems, pp. 7-14, 1993.
[14] D. Hou, K. Wong, and J.H. Hoover, "What Can Programmer Questions Tell Us about Frameworks?" Proc. 13th Int'l Workshop Program Comprehension, pp. 87-96, 2005.
[15] S.Y. Jeong, Y. Xie, J. Beaton, B.A. Myers, J. Stylos, R. Ehret, J. Karstens, A. Efeoglu, and D.K. Busse, "Improving Documentation for sSOA APIs through User Studies," Proc. Second Int'l Symp. End-User Development, vol. 5435, pp. 86-105, 2009.
[16] W. Johnson and A. Erdem, "Interactive Explanation of Software Systems," Proc. 10th Conf. Knowledge-Based Software Eng., pp. 155-164, 1995.
[17] G. Kiczales, "Beyond the Black Box: Open Implementation," IEEE Software, vol. 13, no. 1, pp. 8-11, Jan. 1996.
[18] D. Kirk, M. Roper, and M. Wood, "Identifying and Addressing Problems in Object-Oriented Framework Reuse," Empirical Software Eng., vol. 12, pp. 243-274, June 2007.
[19] A.J. Ko, R. DeLine, and G. Venolia, "Information Needs in Collocated Software Development Teams," Proc. 29th Int'l Conf. Software Eng., pp. 344-353, 2007.
[20] D. Kramer, "API Documentation from Source Code Comments: A Case Study of Javadoc," Proc. Conf. ACM Special Interest Group for Design of Comm., pp. 147-153, 1999.
[21] J.R. Landis and G.G. Koch, "The Measurement of Observer Agreement for Categorical Data," Biometrics, vol. 33, no. 1, pp. 159-174, Mar. 1977.
[22] W. Maalej and H.-J. Happel, "Can Development Work Describe Itself?" Proc. Seventh IEEE Int'l Working Conf. Mining Software Repositories, pp. 191-200, 2010.
[23] M. Monperrus, M. Eichberg, E. Tekes, and M. Mezini, "What Should Developers Be Aware of? An Empirical Study on the Directives of API Documentation," Empirical Software Eng., vol. 17, no. 6, pp. 703-737, 2012.
[24] J. Mylopoulos, A. Borgida, and E. Yu, "Representing Software Engineering Knowledge," Automated Software Eng., vol. 4, no. 3, pp. 291-317, 1997.
[25] K.A. Neuendorf, The Content Analysis Guidebook. Sage Publications, 2002.
[26] J. Nykaza, R. Messinger, F. Boehme, C.L. Norman, M. Mace, and M. Gordon, "What Programmers Really Want: Results of a Needs Assessment for SDK Documentation," Proc. 20th Ann. ACM SIGDOC Int'l Conf. Computer Documentation, pp. 133-141, 2002.
[27] D. Pagano and W. Maalej, "How Do Open Source Communities Blog?" Empirical Software Eng., pp. 1-35, 2012.
[28] C. Parnin and C. Treude, "Measuring API Documentation on the Web," Proc. Second Int'l Workshop Web 2.0 for Software Eng., 2011.
[29] M.P. Robillard, "What Makes APIs Hard to Learn? Answers from Developers," IEEE Software, vol. 26, no. 6, pp. 26-34, Nov./Dec. 2009.
[30] M.P. Robillard and R. DeLine, "A Field Study of API Learning Obstacles," Empirical Software Eng., vol. 16, no. 6, pp. 703-732, 2011.
[31] R.L. Rosnow and R. Rosenthal, Beginning Behavioral Research: A Conceptual Primer, sixth ed. Pearson, Apr. 2007.
[32] L. Shi, H. Zhong, T. Xie, and M. Li, "An Empirical Study on Evolution of API Documentation," Proc. Conf. Fundamental Approaches to Software Eng., pp. 416-431, 2011.
[33] J. Sillito, G.C. Murphy, and K.D. Volder, "Asking and Answering Questions during a Programming Change Task," IEEE Trans. Software Eng., vol. 34, no. 4, pp. 434-451, July/Aug. 2008.
[34] J. Stylos, B. Graf, D.K. Busse, C. Ziegler, and R.E.J. Karstens, "A Case Study of API Redesign for Improved Usability," Proc. Symp. Visual Languages and Human-Centric Computing, pp. 189-192, 2008.
[35] J. Stylos and B.A. Myers, "The Implications of Method Placement on API Learnability," Proc. 16th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 105-112, 2008.
6 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool