This Article 
 Bibliographic References 
 Add to: 
User Interface Evaluation and Empirically-Based Evolution of a Prototype Experience Management Tool
September 2003 (vol. 29 no. 9)
pp. 838-850

Abstract—Experience management refers to the capture, structuring, analysis, synthesis, and reuse of an organization's experience in the form of documents, plans, templates, processes, data, etc. The problem of managing experience effectively is not unique to software development, but the field of software engineering has had a high-level approach to this problem for some time. The Experience Factory is an organizational infrastructure whose goal is to produce, store, and reuse experiences gained in a software development organization. This paper describes The Q-Labs Experience Management System (Q-Labs EMS), which is based on the Experience Factory concept and was developed for use in a multinational software engineering consultancy . A critical aspect of the Q-Labs EMS project is its emphasis on empirical evaluation as a major driver of its development and evolution. The initial prototype requirements were grounded in the organizational needs and vision of Q-Labs, as were the goals and evaluation criteria later used to evaluate the prototype. However, the Q-Labs EMS architecture, data model, and user interface were designed to evolve, based on evolving user needs. This paper describes this approach, including the evaluation that was conducted of the initial prototype and its implications for the further development of systems to support software experience management.

[1] K.-D. Althoff, A. Birk, S. Hartkopf, W. Muller, M. Nick, D. Surmann, and C. Tautz, Systematic Population, Utilization, and Maintenance of a Repository for Comprehensive Reuse Proc. Workshop Learning Software Organizations: Methodology and Applications (SEKE '99), pp. 25-50, June 1999.
[2] K.-D. Althoff, M. Nick, and C. Tautz, Systematically Diagnosing and Improving the Perceived Usefulness of Organizational Memories Proc. Workshop Learning Software Organizations: Methodology and Applications (SEKE '99), pp. 72-86, June 1999.
[3] K.-D. Althoff, B. Decker, S. Hartkopf, A. Jeditschka, M. Nick, and J. Rech, Experience Management: The Fraunhofer IESE Experience Factory Proc. Industrial Conf. Data Mining, July 2001.
[4] K.-D. Althoff, U. Becker-Kornstaedt, B. Decker, A. Klotz, E. Leopold, J. Rech, and A. Voss, Enhancing Experience Management and Process Learning with Moderated Discourses: The indiGo Approach Proc. European Conf. Artificial Intelligence (ECAI '02) Workshop Knowledge Management and Organizational Memory, 2002.
[5] C. Bailey and M. Clarke, Managing Knowledge for Personal and Organisational Benefit J. Knowledge Management, vol. 5, no. 1, pp. 58-67, 2001.
[6] V. Basili, "Software Development: A Paradigm for the Future," Proc. 13th Int'l Comp. Software Applications Conf., IEEE CS Press, Los Alamitos, Calif., 1989, pp. 471-485.
[7] V.R. Basili, “The Experience Factory and its Relationship to Other Improvement Paradigms,” Proc. Fourth European Software Eng. Conf. (ESEC), Sept. 1993.
[8] V.R. Basili and G. Caldiera, Improve Software Quality by Reusing Knowledge and Experience Sloan Management Rev., vol. 37, no. 1, Fall 1995.
[9] V.R. Basili, G. Caldiera, F. McGarry, R. Pajerski, G. Page, and S. Waligora, "The Software Engineering Laboratory: An Operational Software Experience Factory," Proc. 14th Int'l Conf. Software Eng.,Australia, pp. 370-381, 1992.
[10] V.R. Basili, M. Daskalantonakis, and R. Yacobellis, Technology Transfer at Motorola IEEE Software, pp. 70-76, Mar. 1994.
[11] V.R. Basili, M. Lindvall, and P. Costa, Implementing the Experience Factory Concepts as a Set of Experience Bases Proc. 13th Int'l Conf. Software Eng. and Knowledge Eng., pp. 102-109, June 2001.
[12] M. Brandt and M. Nick, Computer-Supported Reuse of Project Management Experience with an Experience Base Proc. Advances in Learning Software Organizations, pp. 178-189, Sept. 2001.
[13] M. Broome and P. Runeson, Technical Requirements for the Implemenation of an Experience Base Proc. Workshop Learning Software Organizations: Methodology and Applications (SEKE '99), pp. 87-102, June 1999.
[14] R. Baronide Carvalho and M.A.T. Ferreira, Using Information Technology to Support Knowledge Conversion Processes Information Research, vol. 7, no. 1, 2001, also available at .
[15] R.L. Chase, Knowledge Management Benchmarks J. Knowledge Management, vol. 1, no. 1, pp. 83-92, Sept. 1997.
[16] T.H. Davenport, D.W. De Long, and M.C. Beers, Successful Knowledge Management Projects Sloan Management Rev., pp. 43-57, Winter 1998.
[17] T. Dingsoyr, An Evaluation of Research on Experience Factory Proc. Workshop Learning Software Organizations (PROFES 2000), pp. 55-66, June 2000.
[18] R.L. Feldmann, M. Nick, and M. Frey, Towards Industrial-Strength Measurement Programs for Reuse and Experience Repository Systems Proc. Workshop Learning Software Organizations (PROFES 2000), pp. 7-18, June 2000.
[19] B.G. Glaser and A.L. Strauss, The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company, 1967.
[20] S. Greene, E. Tanin, C. Plaisant, B. Shneiderman, L. Olsen, G. Major, and S. Johns, The End of Zero-Hit Queries: Query Previews for NASA's Global Change Master Directory Int'l J. Digital Libraries, vol. 2, nos. 2 and 3, pp. 79-90, 1999.
[21] J.T. Hackos and J.D. Redish, User and Task Analysis for Interface Design. pp. 258–259, New York: John Wiley&Sons, ch. 9, 1998.
[22] F. Houdek, K. Schneider, and E. Wieser, Establishing Experience Factories at Daimler-Benz: An Experience Report Proc. 20th Int'l Conf. Software Eng., pp. 443-447, Apr. 1998.
[23] T. Käpylä, I. Niemi, and A. Lehtola, Towards an Accessible Web by Applying PUSH Technology Proc. Fourth ERCIM Workshop, User Interfaces for All, pp. 19-21, Oct. 1998, also available at Ercim_WS_ UI_paper.htm.
[24] T. Kotnour, C. Orr, J. Spaulding, and J. Guidi, Determining the Benefit of Knowledge Management Activities Proc. IEEE Int'l Conf. Systems, Man, Cybernetics, pp. 94-99, Oct. 1997.
[25] G. Lawton, Knowledge Management: Ready for Prime Time? Computer, vol. 34, no. 2, pp. 12-14, 2001.
[26] J. Liebowitz and T. Beckman, Knowledge Organizations: What Every Manager Should Know. Washington: St. Lucie Press, 1998.
[27] M. Lindvall, M. Frey, P. Costa, and R. Tesoriero, An Analysis of Three Experiences Bases Learning Software Organizations, 2001.
[28] M. Lindvall, I. Rus, and S. Sinha, Technology Support for Knowledge Management Proc. Fourth Int'l Workshop Learning Software Organizations (LSO '02), Aug. 2002.
[29] R. Mack, Y. Ravin, and R.J. Byrd, Knowledge Portals and the Emerging Digital Knowledge Workplace IBM Systems J., vol. 40, no. 4, pp. 925-955, 2001.
[30] A.D. Marwick, Knowledge Management Technology IBM Systems J., vol. 40, no. 4, pp. 814-830, 2001.
[31] M. Mendonça, C.B. Seaman, V. Basili, and Y.-M. Kim, A Prototype Experience Management System for a Software Consulting Organization Proc. 13th Int'l Conf. Software Eng. and Knowledge Eng., pp. 29-36, June 2001.
[32] M.B. Miles and A.M. Huberman, Qualitative Data Analysis: An Expanded Sourcebook, second ed. Thousand Oaks, Calif: Sage, 1994.
[33] G. Miller, The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information Psychological Rev., vol. 101, no. 2, pp. 343-352, Apr. 1994.
[34] I. Nonaka and H. Takeuchi, The Knowledge Creating Company. Oxford Univ. Press, 1995.
[35] R. Prieto-Diaz, Classifying of Reusable Modules Software Reusability, T.J. Biggerstaff and A. Perlis, eds., vol. I, 1990.
[36] R. Roy, F.M. del Rey, B. van Wegen, and A. Steele, A Framework to Create Performance Indicators in Knowledge Management Proc. Third Int'l Conf. Practical Aspects of Knowledge Management (PAKM2000), pp. 18-1:18-8, Oct. 2000.
[37] G. Ruhe and F. Bomarius, Introduction and Motivation Proc. Workshop Learning Software Organizations: Methodology and Applications (SEKE '99), pp. 3-22, June 1999.
[38] K. Schneider, J. von Hunnius, and V.R. Basili, Experience in Implementing a Learning Software Organization IEEE Software, pp. 46-49, June 2002.
[39] C.B. Seaman, M. Mendonca, V. Basili, and Y.-M. Kim, An Experience Management System for a Software Consulting Organization Proc. Software Eng. Workshop, NASA/Goddard Software Eng. Laboratory, Dec. 1999.
[40] B.C. Seaman, “Qualitative Methods in Empirical Studies of Software Engineering,” IEEE Trans. Software Eng., vol. 25, no. 4, pp. 557-572, July 1999.
[41] B. Shneiderman, "Dynamic Queries for Visual Information Seeking," IEEE Software, Nov./Dec. 1994, pp. 70-77.
[42] D.J. Skyrme and D.M. Amidon, New Measures of Success J. Business Strategy, pp. 20-24, Jan./Feb. 1998.
[43] C. Tautz, Customizing Software Engineering Experience Management Systems to Organizational Needs PhD dissertation, Dept. of Computer Science, Univ. of Kaiserslautern, Germany, 2000.
[44] A. Tiwana, The Knowledge Management Toolkit: Practical Techniques for Building a Knowledge Management System, Prentice-Hall, Upper Saddle River, N.J., 2000.
[45] A. von Mayrhauser and A.M. Mans, “Identification of Dynamic Comprehension Processes During Large Scale Maintenance,” IEEE Trans. Software Eng., vol. 22, no. 6, pp. 424–437, 1996.
[46] C.G. von Wangenheim, K.-D. Althoff, and R.M. Barcia, Goal-Oriented and Similarity-Based Retrieval of Software Engineering Experienceware Proc. Workshop Learning Software Organizations: Methodology and Applications (SEKE '99), pp. 118-141, June 1999.
[47] C.G. von Wangenheim and M.R. Rodrigues, Case-Based Management of Software Engineering Experienceware Proc. IBERAMIA-SBIA 2000, pp. 12-22, 2000.
[48] R. Webby, C. Seaman, M. Mendonça, V.R. Basili, and Y. Kim, Implementing an Internet-Enabled Software Experience Factory: Work in Progress Proc. Second Workshop Software Eng. over the Internet (ICSE '99), May 1999.

Index Terms:
Experience management, knowledge management, experience reuse, user interface evaluation, empirical study.
Carolyn B. Seaman, Manoel G. Mendon?, Victor R. Basili, Yong-Mi Kim, "User Interface Evaluation and Empirically-Based Evolution of a Prototype Experience Management Tool," IEEE Transactions on Software Engineering, vol. 29, no. 9, pp. 838-850, Sept. 2003, doi:10.1109/TSE.2003.1232288
Usage of this product signifies your acceptance of the Terms of Use.