This Article
	Subscribers, please Login Purchase article: $19 PDF HTML RSS feed
	Share
	Email this Article to a friend
	Bibliographic References
	ASCII Text BibTex RefWorks Procite/RefMan/EndNote
	Add to:
	Digg Furl Spurl Blink Simpy Google Del.icio.us Y!MyWeb
	Search
	Similar Articles Articles by Oscar Dieste Articles by Natalia Juristo

Systematic Review and Aggregation of Empirical Studies on Elicitation Techniques

March/April 2011 (vol. 37 no. 2)

pp. 283-304

[1] L. Adelman, “Measurement Issues in Knowledge Engineering,” IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 3, pp. 483-488, May/June 1989.
[2] R. Agarwal and M.R. Tanniru, “Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation,” J. Management Information Systems, vol. 7, pp. 123-141, 1990.
[3] T. Bech-Larsen and N.A. Nielsen, “A Comparison of Five Elicitation Techniques for Elicitation of Attributes of Low Involvement Products,” J. Economic Psychology, vol. 20, pp. 315-341, 1999.
[4] E. Breivik and M. Supphellen, “Elicitation of Product Attributes in an Evaluation Context: A Comparison of Three Elicitation Techniques,” J. Economic Psychology, vol. 24, pp. 77-98, 2003.
[5] G.J. Browne and M.B. Rogich, “An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques,” J. Management Information Systems, vol. 17, pp. 223-249, 2001.
[6] A.M. Burton, N.R. Shadbolt, A.P. Hedgecock, and G. Rugg, “A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1,” Proc. Expert Systems '87 Research and Development in Expert Systems IV, The Ann. Technical Conf. British CS Specialist Group on Expert Systems, D.S. Moralee, ed., 1987.
[7] B. Corbridge, G. Rugg, N.P. Major, N.R. Shadbolt, and A.M. Burton, “Laddering—Technique and Tool Use in Knowledge Acquisition,” Knowledge Acquisition, vol. 6, pp. 315-341, 1994.
[8] B. Crandall, “A Comparative Study of Think-Aloud and Critical Decision Knowledge Elicitation Methods,” ACM SIGART Bull., vol. 108, pp. 144-146, 1989.
[9] J.M. Moore and F.M.I. Shipman, “A Comparison of Questionnaire-Based and GUI-Based Requirements,” Proc. 15th IEEE Int'l Conf. Automated Software Eng., pp. 35-43, 2000.
[10] T.J. Larsen and J.D. Naumann, “An Experimental Comparison of Abstract and Concrete Representations in Systems Analysis,” Information & Management, vol. 22, pp. 29-40, 1992.
[11] J.E. Fowlkes, E. Salas, and D.P. Baker, “The Utility of Event-Based Knowledge Elicitation,” Human Factors, vol. 42, pp. 24-35, 2000.
[12] V. Goodrich and L. Olfman, “An Experimental Evaluation of Task and Methodology Variables for Requirements Definition Phase Success,” Proc. 23rd Ann. Hawaii Int'l Conf. System Sciences, pp. 201-209, 1990.
[13] E. Hudlicka, “Requirements Elicitation with Indirect Knowledge Elicitation: Comparison of Three Methods,” Proc. Second IEEE Int'l Conf. Requirements Eng., pp. 4-11, 1996.
[14] S.R. Jones and J.C. Miles, “The Use of a Prototype System for Evaluating Knowledge Elicitation Techniques,” Expert Systems, vol. 15, pp. 83-97, 1998.
[15] A.M. Burton, N.R. Shadbolt, G. Rugg, and A.P. Hedgecock, “The Efficacy of Knowledge Acquisition Techniques: A Comparison Across Domains and Levels of Expertise,” Knowledge Acquisition, vol. 2, pp. 167-178, 1990.
[16] J.W. Moody, R.P. Will, and J.E. Blanton, “Enhancing Knowledge Elicitation Using the Cognitive Interview,” Expert Systems with Applications, vol. 10, pp. 127-133, 1996.
[17] A.L. Rowe, N.J. Cooke, E.P. Hall, and T.L. Halgren, “Toward an On-Line Knowledge Assessment Methodology: Building on the Relationship between Knowing and Doing,” J. Experimental Psychology: Applied, vol. 2, pp. 31-47, 1996.
[18] R.W. Zmud, W.P. Anthony, and R.M. Stair, “The Use of Mental Imagery to Facilitate Information Identification in Requirements Analysis,” J. Management Information Systems, vol. 9, pp. 175-191, 1993.
[19] M. Eva, “Requirements Acquisition for Rapid Applications Development,” Information & Management, vol. 39, pp. 101-107, Dec. 2001.
[20] G.M. Marakas and J.J. Elam, “Semantic Structuring in Analyst and Representation of Facts in Requirements Analysis,” Information Systems Research, vol. 9, pp. 37-63, 1998.
[21] G. Rugg, C. Corbridge, N.P. Major, A.M. Burton, and N. Shadbolt, “A Comparison of Sorting Techniques in Knowledge Acquisition,” J. Knowledge Acquisition, vol. 4, pp. 279-291, 1992.
[22] M.G. Pitts and G.J. Browne, “Stopping Behavior of Systems Analysts during Information Requirements Elicitation,” J. Management Information Systems, vol. 21, pp. 203-226, 2004.
[23] A. Griffin and J.R. Hauser, “The Voice of the Customer,” Marketing Science, vol. 12, pp. 1-27, 1993.
[24] R. Schweickert, A.M. Burton, N.K. Taylor, E.N. Corlett, N.R. Shadbolt, and A.P. Hedgecock, “Comparing Knowledge Elicitation Techniques: A Case Study,” Artificial Intelligence Rev., vol. 1, pp. 245-253, 1987.
[25] J.A. Silver and J.C. Thompson, “Understanding Customer Needs—A Systematic Approach to the ‘Voice of the Customer’,” MS thesis, Massachusetts Inst. of Technology, Sloan School of Management, 1991.
[26] L.A. Freeman, “The Effects of Concept Maps on Requirements Elicitation and System Models During Information Systems Development,” Concept Maps: Theory, Methodology, Technology, Proc. First Int'l Conf. Concept Mapping, 2004.
[27] L. Johnson and N.E. Johnson, “Knowledge Elicitation Involving Teachback Interviewing,” Knowledge Acquisition for Expert Systems: A Practical Handbook, A.L. Kidd, ed., pp. 91-108, Plenum Press, 1987.
[28] K.A. Ericson and H.A. Simon, Protocol Analysis: Verbal Reports as Data. Bradford Books/MIT Press, 1984.
[29] M. LaFrance, “The Knowledge Acquisition Grid: A Method for Training Knowledge Engineers,” Proc. Knowledge Acquisition for Knowledge-Based Systems Workshop, 1986.
[30] J. Wood and D. Silver, Joint Application Development. Wiley, 1995.
[31] L.J. Korhonen, “Nominal Group Technique,” Adult Learning Methods, N.W. Galbraith, ed., pp. 247-259, Krieger Publishing Company, 1990.
[32] A.P. Massey and W.A. Wallace, “Focus Groups as a Knowledge Elicitation Technique: An Exploratory Study,” IEEE Trans. Knowledge and Data Eng., vol. 3, no. 2, pp. 193-200, June 1991.
[33] L. Adelman, P.J. Sticha, and M.L. Donnel, “An Experimental Investigation of the Relative Effectiveness of Two Techniques for Structuring Multiattributed Hierarchies,” Organizational Behavior and Human Decision Processes, vol. 37, pp. 188-196, 1986.
[34] S. Frankel, “NGT+MDS: An Adaptation of the Nominal Group Technique for Ill-Structured Problems,” J. Applied Behavioral Science, vol. 23, pp. 543-551, 1987.
[35] A. Sutcliffe, “A Technique Combination Approach to Requirements Engineering,” Proc. Third IEEE Int'l Symp. Requirements Eng., pp. 65-74, 1997.
[36] R.R. Hoffman, “The Problem of Extracting the Knowledge of Experts from the Perspective of Experimental Psychology,” AI Magazine, vol. 8, pp. 53-67, 1987.
[37] N.R. Shadbolt and A.M. Burton, “The Empirical Study of Knowledge Elicitation Techniques,” ACM SIGART Bull., vol. 108, pp. 15-18, 1989.
[38] R.R. Hoffman, N.R. Shadbolt, A.M. Burton, and G. Klein, “Eliciting Knowledge from Experts: A Methodological Analysis,” Organizational Behavior and Human Decision Processes, vol. 62, pp. 129-158, 1995.
[39] A. Blandford and G. Rugg, “A Case Study on Integrating Contextual Information with Analytical Usability Evaluation,” Int'l J. Human-Computer Studies, vol. 57, pp. 75-99, 2002.
[40] M.A. Musen, L.M. Fagan, D.M. Combs, and E.H. Shortliffe, “Use of a Domain Model to Drive an Interactive Knowledge-Editing Tool,” Int'l J. Man-Machine Studies, vol. 26, pp. 105-121, 1987.
[41] J. Coughlan and R.D. Macredie, “Effective Communication in Requirements Elicitation: A Comparison of Methodologies,” Requirements Eng. J., vol. 7, pp. 47-60, 2002.
[42] R. Martinez-Bejar, R. Benjamins, F. Martin, and V. Castillo, “Deriving Formal Parameters for Comparing Knowledge Elicitation Techniques Based on Mathematical Functions,” Proc. 10th Knowledge Acquisition for Knowledge-Based Systems Workshop, 1996.
[43] J.G. Gammack and A. Anderson, “Constructive Interaction in Knowledge Engineering,” Expert Systems, vol. 7, pp. 19-26, 1990.
[44] B. Fischhoff, “Eliciting Knowledge for Analytical Representation,” IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 3, pp. 448-461, May/June 1989.
[45] S.E. Gordon, K.A. Schmierer, and R.T. Gill, “Conceptual Graph Analysis: Knowledge Acquisition for Instructional System Design,” Human Factors, vol. 35, pp. 459-481, 1993.
[46] C.W. Holsapple and V.S. Raj, “An Exploratory Study of Two KA Methods,” Expert Systems, vol. 11, pp. 77-87, 1994.
[47] L.E. Wood, T.C. Davis, S.L. Clay, J.M. Ford, and S. Lammersen, “Evaluation of Interviewing Methods and Mediating Representations for Knowledge Acquisition,” Int'l J. Expert Systems, vol. 8, pp. 1-23, 1995.
[48] T.A. Byrd, K.L. Cossick, and R.W. Zmud, “A Synthesis of Research on Requirements Analysis and Knowledge Acquisition Techniques,” MIS Quarterly, vol. 16, pp. 117-138, 1992.
[49] M.C. Munro and G.B. Davis, “Determining Management Information News: A Comparison of Methods,” MIS Quarterly, vol. 1, pp. 55-67, 1977.
[50] A. Sutcliffe, “Requirements Rationales: Integrating Approaches to Requirements Analysis,” Proc. Designing Interactive Systems, pp. 33-42, 1995.
[51] N.J. Cooke, “Varieties of Knowledge Elicitation Techniques,” Int'l J. Human-Computer Studies, vol. 41, pp. 801-849, 1994.
[52] A. Hart, Knowledge Acquisition for Expert Systems. Anchor Brendon Ltd., 1986.
[53] J.G. Gammack and R.M. Young, “Psychological Techniques for Eliciting Expert Knowledge,” Knowledge Acquisition for Expert Systems, A. Kidd, ed., pp. 137-163, Plenum Press, 1985.
[54] J.H. Boose, “A Survey of Knowledge Acquisition Techniques and Tools,” Knowledge Acquisition, vol. 1, pp. 3-37, 1989.
[55] N.A.M. Maiden and G. Rugg, “ACRE: Selecting Methods for Requirements Acquisition,” Software Eng. J., vol. 11, pp. 183-192, 1996.
[56] G. Kotonya and I. Sommerville, Requirements Engineering: Processes and Techniques. Wiley, 1998.
[57] L. Macaulay, Requirements Engineering, Springer, 1996.
[58] S. Lauesen, Software Requirements: Styles and Techniques. Addison-Wesley, 2002.
[59] J.A. Goguen and C. Linde, “Techniques for Requirements Elicitation,” Proc. IEEE Int'l Symp. Requirements Eng., pp. 152-164, 1993.
[60] S. Robertson, “Requirements Trawling: Techniques for Discovering Requirements,” Int'l J. Human-Computer Studies, vol. 55, pp. 405-421, 2001.
[61] A.M. Hickey and A.M. Davis, “A Tale of Two Ontologies: The Basis for Systems Analysis Technique Selection,” Proc. Ninth Ann. Am. Conf. Information Systems, 2003.
[62] N.J. Cooke, “Eliciting Semantic Relations for Empirically Derived Networks,” Int'l J. Man-Machine Studies, vol. 37, pp. 721-750, 1992.
[63] B.A. Kitchenham, “Procedures for Performing Systematic Reviews,” Keele Univ. Technical Report TR/SE-0401, 2004.
[64] C.C. Ragin, The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Univ. of California Press, 1987.
[65] R.K. Yin and K.A. Heald, “Using the Case Survey Method to Analyze Policy Studies,” Administrative Science Quarterly, vol. 20, pp. 371-381, 1975.
[66] A. Davis, O. Dieste, A. Hickey, N. Juristo, and A.M. Moreno, “Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review,” Proc. 14th IEEE Int'l Conf. Requirements Eng., 2006.
[67] P.G. Benson, S.P. Curley, and G.F. Smith, “Belief Assessment: An Underdevelopment Phase of Probability Elicitation,” Management Science, vol. 41, pp. 1639-1653, 1995.
[68] G.J. Browne, S.P. Curley, and P.G. Benson, “Evoking Information in Probability Assessment: Knowledge Maps and Reasoning-Based Directed Questions,” Management Science, vol. 43, pp. 1-14, 1997.
[69] G.P. Hodgkinson, A.J. Maule, and N.J. Bown, “Causal Cognitive Mapping in the Organizational Strategy Field: A Comparison of Alternative Elicitation Procedures,” Organizational Research Methods, vol. 7, pp. 3-26, Jan. 2004.
[70] R. Kanwar, J.C. Olson, and L.S. Sims, “Toward Conceptualizing and Measuring Cognitive Structures,” Advances in Consumer Research, vol. 8, pp. 122-127, 1981.
[71] E.W. Duggan and C.S. Thachenkary, “Higher Quality Requirements: Supporting Joint Application Development with the Nominal Group Technique,” Information Technology and Management, vol. 4, pp. 391-408, 2003.
[72] W.J. Lloyd, M.B. Rosson, and J.D. Arthur, “Effectiveness of Elicitation Techniques in Distributed Requirements Engineering,” Proc. IEEE Joint Int'l Conf. Requirements Eng., pp. 311-318, 2002.
[73] R. Ocker and S.R. Hiltz, “The Effects of Distributed Group Support and Process Structuring on Software Requirements Development Teams: Results on Creativity and Quality,” J. Management Information Systems, vol. 12, pp. 127-154, 1996.
[74] A. Sutcliffe, “Requirements Rationales: Integrating Approaches to Requirements Analysis,” Proc. Designing Interactive Systems, pp. 33-42, 1995.
[75] M. Tawbi, C. Ben Achour, and F. Velez, “Guiding the Process of Requirements Elicitation through Scenario,” Proc. 10th Int'l Workshop Database and Expert Systems Applications, pp. 345-349, 1999.
[76] G.W. Noblit and R.D. Hare, Meta-Ethnography: Synthesizing Qualitative Studies. Sage, 1988.
[77] B.L. Paterson, S.E. Thorne, C. Canam, and C. Jillings, Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis. Sage, 2001.
[78] M. Dixon-Woods, S. Agarwal, D. Jones, B. Young, and A. Sutton, “Synthesising Qualitative and Quantitative Evidence: A Review of Possible Methods,” J. Health Services Research and Policy, vol. 10, pp. 45-53, 2005.
[79] M. Mahmood, “System Development Methods—A Comparative Investigation,” MIS Quarterly, vol. 11, pp. 293-311, 1987.
[80] P.D. Chatzoglou, “Factors Affecting Completion of the Requirements Capture Stage of Projects with Different Characteristics,” Information and Software Technology, vol. 39, pp. 627-640, 1997.
[81] M. Tan, “Establishing Mutual Understanding in Systems Design: An Empirical Study,” J. Management Information Systems, vol. 10, pp. 159-182, 1994.
[82] J.W. Moody, J.E. Blanton, and P.H. Cheney, “A Theoretically Grounded Approach to Assist Memory Recall during Information Requirements Determination,” J. Management Information Systems, vol. 15, pp. 79-89, 1998.
[83] R.W. Zmud, W.P. Anthony, and R.M. Stair, “The Use of Mental Imagery as a Requirements Analysis Technique,” Challenges and Strategies for Research in Systems Development, W.W. Cotterman and J.A. Senn, eds., pp. 337-350, John Wiley & Sons, 1992.
[84] R. Lines, E. Breivik, and M. Supphellen, “Elicitation of Attributes: A Comparison of Preference Model Structures Derived from Two Elicitation Techniques,” Proc. 24th European Marketing Academy Conf., pp. 641-656, 1995.
[85] A.M. Burton, N.R. Shadbolt, A.P. Hedgecock, and G.A. Rugg, “A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1,” Proc. First European Workshop Knowledge Acquisition for Knowledge-Based Systems, 1987.
[86] R. Lines, E. Breivik, and M. Supphellen, “Method Effects in Attribute Elicitation: A Multidimensional Comparison of Four Techniques,” 1995.
[87] A.M. Burton and N. Shadbolt, “A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1,” Proc. SERC Workshop Knowledge Acquisition for Eng. Applications, pp. 20-28, 1987.
[88] O. Dieste, A. Griman, and N. Juristo, “Developing Search Strategies for Detecting Relevant Experiments,” Empirical Software Eng., vol. 14, pp. 513-539, 2009.
[89] S.L. Pfleeger, “Soup or Art? The Role of Evidential Force in Empirical Software Engineering,” IEEE Software, vol. 22, no. 1, pp. 66-73, Jan./Feb. 2005.
[90] S.E. Straus, W.S. Richardson, P. Glasziou, and R.B. Haynes, Evidence-Based Medicine. Churchill Livingstone, 2005.
[91] L.V. Hedges and I. Olkin, Statistical Methods for Meta-Analysis. Academic Press, 1985.
[92] N. Juristo, A.M. Moreno, S. Vegas, and M. Solari, “In Search of What We Empirically Know about Unit Testing,” IEEE Software, vol. 23, no. 6, pp. 72-80, Nov.-Dec. 2006.
[93] T. Dyba, V. Kampenes, and D. Sjoberg, “A Systematic Review of Statistical Power in Software Engineering Experiments,” Information and Software Technology, vol. 48, pp. 745-755, 2006.
[94] N. Juristo, A.M. Moreno, and S. Vegas, “Reviewing 25 Years of Testing Technique Experiments,” J. Empirical Software Eng., vol. 9, pp. 7-44, 2004.
[95] P.G. Shekelle, M.L. Hardy, S.C. Morton, M. Maglione, W.A. Mojica, M.J. Suttorp, S.L. Rhodes, L. Jungvig, and J. Gagne, “Efficacy and Safety of Ephedra and Ephedrine for Weight Loss and Athletic Performance: A Meta-Analysis,” J. Am. Medical Assoc., vol. 289, pp. 1537-1545, Mar. 2003.
[96] D. Moher, A.R. Jadad, G. Nichol, M. Penman, P. Tugwell, and S. Walsh, “Assessing the Quality of Randomized Controlled Trials: An Annotated Bibliography of Scales and Checklists,” Controlled Clinical Trials, vol. 16, pp. 62-73, Feb. 1995.
[97] J.P.T. Higgins and S. Green, Cochrane Handbook for Systematic Reviews of Interventions. John Wiley and Sons, 2006.
[98] O.P. Brereton, B. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Employing Systematic Literature Review: An Experience Report,” Technical Report TR/05-01, School of Computing & Math., Keele Univ., 2005.
[99] B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg, “Preliminary Guidelines for Empirical Research in Software Engineering,” IEEE Trans. Software Eng., vol. 28, no. 8, pp. 721-734, Aug. 2002.
[100] W. Shadish, T. Cook, and D.T. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, 2001.
[101] Construx Software, “Context Free Requirements Questions,” http://www.construx.comPage.aspx?hid=1742 , 2008.
[102] D.T. Campbell, “Reforms as Experiments,” Am. Psychologist, vol. 24, pp. 409-429, 1969.