This Article 
 Bibliographic References 
 Add to: 
Systematic Review and Aggregation of Empirical Studies on Elicitation Techniques
March/April 2011 (vol. 37 no. 2)
pp. 283-304
Oscar Dieste, Universidad Politécnica de Madrid, Boadilla del Monte
Natalia Juristo, Universidad Politécnica de Madrid, Boadilla del Monte
We have located the results of empirical studies on elicitation techniques and aggregated these results to gather empirically grounded evidence. Our chosen surveying methodology was systematic review, whereas we used an adaptation of comparative analysis for aggregation because meta-analysis techniques could not be applied. The review identified 564 publications from the SCOPUS, IEEEXPLORE, and ACM DL databases, as well as Google. We selected and extracted data from 26 of those publications. The selected publications contain 30 empirical studies. These studies were designed to test 43 elicitation techniques and 50 different response variables. We got 100 separate results from the experiments. The aggregation generated 17 pieces of knowledge about the interviewing, laddering, sorting, and protocol analysis elicitation techniques. We provide a set of guidelines based on the gathered pieces of knowledge.

[1] L. Adelman, “Measurement Issues in Knowledge Engineering,” IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 3, pp. 483-488, May/June 1989.
[2] R. Agarwal and M.R. Tanniru, “Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation,” J. Management Information Systems, vol. 7, pp. 123-141, 1990.
[3] T. Bech-Larsen and N.A. Nielsen, “A Comparison of Five Elicitation Techniques for Elicitation of Attributes of Low Involvement Products,” J. Economic Psychology, vol. 20, pp. 315-341, 1999.
[4] E. Breivik and M. Supphellen, “Elicitation of Product Attributes in an Evaluation Context: A Comparison of Three Elicitation Techniques,” J. Economic Psychology, vol. 24, pp. 77-98, 2003.
[5] G.J. Browne and M.B. Rogich, “An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques,” J. Management Information Systems, vol. 17, pp. 223-249, 2001.
[6] A.M. Burton, N.R. Shadbolt, A.P. Hedgecock, and G. Rugg, “A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1,” Proc. Expert Systems '87 Research and Development in Expert Systems IV, The Ann. Technical Conf. British CS Specialist Group on Expert Systems, D.S. Moralee, ed., 1987.
[7] B. Corbridge, G. Rugg, N.P. Major, N.R. Shadbolt, and A.M. Burton, “Laddering—Technique and Tool Use in Knowledge Acquisition,” Knowledge Acquisition, vol. 6, pp. 315-341, 1994.
[8] B. Crandall, “A Comparative Study of Think-Aloud and Critical Decision Knowledge Elicitation Methods,” ACM SIGART Bull., vol. 108, pp. 144-146, 1989.
[9] J.M. Moore and F.M.I. Shipman, “A Comparison of Questionnaire-Based and GUI-Based Requirements,” Proc. 15th IEEE Int'l Conf. Automated Software Eng., pp. 35-43, 2000.
[10] T.J. Larsen and J.D. Naumann, “An Experimental Comparison of Abstract and Concrete Representations in Systems Analysis,” Information & Management, vol. 22, pp. 29-40, 1992.
[11] J.E. Fowlkes, E. Salas, and D.P. Baker, “The Utility of Event-Based Knowledge Elicitation,” Human Factors, vol. 42, pp. 24-35, 2000.
[12] V. Goodrich and L. Olfman, “An Experimental Evaluation of Task and Methodology Variables for Requirements Definition Phase Success,” Proc. 23rd Ann. Hawaii Int'l Conf. System Sciences, pp. 201-209, 1990.
[13] E. Hudlicka, “Requirements Elicitation with Indirect Knowledge Elicitation: Comparison of Three Methods,” Proc. Second IEEE Int'l Conf. Requirements Eng., pp. 4-11, 1996.
[14] S.R. Jones and J.C. Miles, “The Use of a Prototype System for Evaluating Knowledge Elicitation Techniques,” Expert Systems, vol. 15, pp. 83-97, 1998.
[15] A.M. Burton, N.R. Shadbolt, G. Rugg, and A.P. Hedgecock, “The Efficacy of Knowledge Acquisition Techniques: A Comparison Across Domains and Levels of Expertise,” Knowledge Acquisition, vol. 2, pp. 167-178, 1990.
[16] J.W. Moody, R.P. Will, and J.E. Blanton, “Enhancing Knowledge Elicitation Using the Cognitive Interview,” Expert Systems with Applications, vol. 10, pp. 127-133, 1996.
[17] A.L. Rowe, N.J. Cooke, E.P. Hall, and T.L. Halgren, “Toward an On-Line Knowledge Assessment Methodology: Building on the Relationship between Knowing and Doing,” J. Experimental Psychology: Applied, vol. 2, pp. 31-47, 1996.
[18] R.W. Zmud, W.P. Anthony, and R.M. Stair, “The Use of Mental Imagery to Facilitate Information Identification in Requirements Analysis,” J. Management Information Systems, vol. 9, pp. 175-191, 1993.
[19] M. Eva, “Requirements Acquisition for Rapid Applications Development,” Information & Management, vol. 39, pp. 101-107, Dec. 2001.
[20] G.M. Marakas and J.J. Elam, “Semantic Structuring in Analyst and Representation of Facts in Requirements Analysis,” Information Systems Research, vol. 9, pp. 37-63, 1998.
[21] G. Rugg, C. Corbridge, N.P. Major, A.M. Burton, and N. Shadbolt, “A Comparison of Sorting Techniques in Knowledge Acquisition,” J. Knowledge Acquisition, vol. 4, pp. 279-291, 1992.
[22] M.G. Pitts and G.J. Browne, “Stopping Behavior of Systems Analysts during Information Requirements Elicitation,” J. Management Information Systems, vol. 21, pp. 203-226, 2004.
[23] A. Griffin and J.R. Hauser, “The Voice of the Customer,” Marketing Science, vol. 12, pp. 1-27, 1993.
[24] R. Schweickert, A.M. Burton, N.K. Taylor, E.N. Corlett, N.R. Shadbolt, and A.P. Hedgecock, “Comparing Knowledge Elicitation Techniques: A Case Study,” Artificial Intelligence Rev., vol. 1, pp. 245-253, 1987.
[25] J.A. Silver and J.C. Thompson, “Understanding Customer Needs—A Systematic Approach to the ‘Voice of the Customer’,” MS thesis, Massachusetts Inst. of Technology, Sloan School of Management, 1991.
[26] L.A. Freeman, “The Effects of Concept Maps on Requirements Elicitation and System Models During Information Systems Development,” Concept Maps: Theory, Methodology, Technology, Proc. First Int'l Conf. Concept Mapping, 2004.
[27] L. Johnson and N.E. Johnson, “Knowledge Elicitation Involving Teachback Interviewing,” Knowledge Acquisition for Expert Systems: A Practical Handbook, A.L. Kidd, ed., pp. 91-108, Plenum Press, 1987.
[28] K.A. Ericson and H.A. Simon, Protocol Analysis: Verbal Reports as Data. Bradford Books/MIT Press, 1984.
[29] M. LaFrance, “The Knowledge Acquisition Grid: A Method for Training Knowledge Engineers,” Proc. Knowledge Acquisition for Knowledge-Based Systems Workshop, 1986.
[30] J. Wood and D. Silver, Joint Application Development. Wiley, 1995.
[31] L.J. Korhonen, “Nominal Group Technique,” Adult Learning Methods, N.W. Galbraith, ed., pp. 247-259, Krieger Publishing Company, 1990.
[32] A.P. Massey and W.A. Wallace, “Focus Groups as a Knowledge Elicitation Technique: An Exploratory Study,” IEEE Trans. Knowledge and Data Eng., vol. 3, no. 2, pp. 193-200, June 1991.
[33] L. Adelman, P.J. Sticha, and M.L. Donnel, “An Experimental Investigation of the Relative Effectiveness of Two Techniques for Structuring Multiattributed Hierarchies,” Organizational Behavior and Human Decision Processes, vol. 37, pp. 188-196, 1986.
[34] S. Frankel, “NGT+MDS: An Adaptation of the Nominal Group Technique for Ill-Structured Problems,” J. Applied Behavioral Science, vol. 23, pp. 543-551, 1987.
[35] A. Sutcliffe, “A Technique Combination Approach to Requirements Engineering,” Proc. Third IEEE Int'l Symp. Requirements Eng., pp. 65-74, 1997.
[36] R.R. Hoffman, “The Problem of Extracting the Knowledge of Experts from the Perspective of Experimental Psychology,” AI Magazine, vol. 8, pp. 53-67, 1987.
[37] N.R. Shadbolt and A.M. Burton, “The Empirical Study of Knowledge Elicitation Techniques,” ACM SIGART Bull., vol. 108, pp. 15-18, 1989.
[38] R.R. Hoffman, N.R. Shadbolt, A.M. Burton, and G. Klein, “Eliciting Knowledge from Experts: A Methodological Analysis,” Organizational Behavior and Human Decision Processes, vol. 62, pp. 129-158, 1995.
[39] A. Blandford and G. Rugg, “A Case Study on Integrating Contextual Information with Analytical Usability Evaluation,” Int'l J. Human-Computer Studies, vol. 57, pp. 75-99, 2002.
[40] M.A. Musen, L.M. Fagan, D.M. Combs, and E.H. Shortliffe, “Use of a Domain Model to Drive an Interactive Knowledge-Editing Tool,” Int'l J. Man-Machine Studies, vol. 26, pp. 105-121, 1987.
[41] J. Coughlan and R.D. Macredie, “Effective Communication in Requirements Elicitation: A Comparison of Methodologies,” Requirements Eng. J., vol. 7, pp. 47-60, 2002.
[42] R. Martinez-Bejar, R. Benjamins, F. Martin, and V. Castillo, “Deriving Formal Parameters for Comparing Knowledge Elicitation Techniques Based on Mathematical Functions,” Proc. 10th Knowledge Acquisition for Knowledge-Based Systems Workshop, 1996.
[43] J.G. Gammack and A. Anderson, “Constructive Interaction in Knowledge Engineering,” Expert Systems, vol. 7, pp. 19-26, 1990.
[44] B. Fischhoff, “Eliciting Knowledge for Analytical Representation,” IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 3, pp. 448-461, May/June 1989.
[45] S.E. Gordon, K.A. Schmierer, and R.T. Gill, “Conceptual Graph Analysis: Knowledge Acquisition for Instructional System Design,” Human Factors, vol. 35, pp. 459-481, 1993.
[46] C.W. Holsapple and V.S. Raj, “An Exploratory Study of Two KA Methods,” Expert Systems, vol. 11, pp. 77-87, 1994.
[47] L.E. Wood, T.C. Davis, S.L. Clay, J.M. Ford, and S. Lammersen, “Evaluation of Interviewing Methods and Mediating Representations for Knowledge Acquisition,” Int'l J. Expert Systems, vol. 8, pp. 1-23, 1995.
[48] T.A. Byrd, K.L. Cossick, and R.W. Zmud, “A Synthesis of Research on Requirements Analysis and Knowledge Acquisition Techniques,” MIS Quarterly, vol. 16, pp. 117-138, 1992.
[49] M.C. Munro and G.B. Davis, “Determining Management Information News: A Comparison of Methods,” MIS Quarterly, vol. 1, pp. 55-67, 1977.
[50] A. Sutcliffe, “Requirements Rationales: Integrating Approaches to Requirements Analysis,” Proc. Designing Interactive Systems, pp. 33-42, 1995.
[51] N.J. Cooke, “Varieties of Knowledge Elicitation Techniques,” Int'l J. Human-Computer Studies, vol. 41, pp. 801-849, 1994.
[52] A. Hart, Knowledge Acquisition for Expert Systems. Anchor Brendon Ltd., 1986.
[53] J.G. Gammack and R.M. Young, “Psychological Techniques for Eliciting Expert Knowledge,” Knowledge Acquisition for Expert Systems, A. Kidd, ed., pp. 137-163, Plenum Press, 1985.
[54] J.H. Boose, “A Survey of Knowledge Acquisition Techniques and Tools,” Knowledge Acquisition, vol. 1, pp. 3-37, 1989.
[55] N.A.M. Maiden and G. Rugg, “ACRE: Selecting Methods for Requirements Acquisition,” Software Eng. J., vol. 11, pp. 183-192, 1996.
[56] G. Kotonya and I. Sommerville, Requirements Engineering: Processes and Techniques. Wiley, 1998.
[57] L. Macaulay, Requirements Engineering, Springer, 1996.
[58] S. Lauesen, Software Requirements: Styles and Techniques. Addison-Wesley, 2002.
[59] J.A. Goguen and C. Linde, “Techniques for Requirements Elicitation,” Proc. IEEE Int'l Symp. Requirements Eng., pp. 152-164, 1993.
[60] S. Robertson, “Requirements Trawling: Techniques for Discovering Requirements,” Int'l J. Human-Computer Studies, vol. 55, pp. 405-421, 2001.
[61] A.M. Hickey and A.M. Davis, “A Tale of Two Ontologies: The Basis for Systems Analysis Technique Selection,” Proc. Ninth Ann. Am. Conf. Information Systems, 2003.
[62] N.J. Cooke, “Eliciting Semantic Relations for Empirically Derived Networks,” Int'l J. Man-Machine Studies, vol. 37, pp. 721-750, 1992.
[63] B.A. Kitchenham, “Procedures for Performing Systematic Reviews,” Keele Univ. Technical Report TR/SE-0401, 2004.
[64] C.C. Ragin, The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Univ. of California Press, 1987.
[65] R.K. Yin and K.A. Heald, “Using the Case Survey Method to Analyze Policy Studies,” Administrative Science Quarterly, vol. 20, pp. 371-381, 1975.
[66] A. Davis, O. Dieste, A. Hickey, N. Juristo, and A.M. Moreno, “Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review,” Proc. 14th IEEE Int'l Conf. Requirements Eng., 2006.
[67] P.G. Benson, S.P. Curley, and G.F. Smith, “Belief Assessment: An Underdevelopment Phase of Probability Elicitation,” Management Science, vol. 41, pp. 1639-1653, 1995.
[68] G.J. Browne, S.P. Curley, and P.G. Benson, “Evoking Information in Probability Assessment: Knowledge Maps and Reasoning-Based Directed Questions,” Management Science, vol. 43, pp. 1-14, 1997.
[69] G.P. Hodgkinson, A.J. Maule, and N.J. Bown, “Causal Cognitive Mapping in the Organizational Strategy Field: A Comparison of Alternative Elicitation Procedures,” Organizational Research Methods, vol. 7, pp. 3-26, Jan. 2004.
[70] R. Kanwar, J.C. Olson, and L.S. Sims, “Toward Conceptualizing and Measuring Cognitive Structures,” Advances in Consumer Research, vol. 8, pp. 122-127, 1981.
[71] E.W. Duggan and C.S. Thachenkary, “Higher Quality Requirements: Supporting Joint Application Development with the Nominal Group Technique,” Information Technology and Management, vol. 4, pp. 391-408, 2003.
[72] W.J. Lloyd, M.B. Rosson, and J.D. Arthur, “Effectiveness of Elicitation Techniques in Distributed Requirements Engineering,” Proc. IEEE Joint Int'l Conf. Requirements Eng., pp. 311-318, 2002.
[73] R. Ocker and S.R. Hiltz, “The Effects of Distributed Group Support and Process Structuring on Software Requirements Development Teams: Results on Creativity and Quality,” J. Management Information Systems, vol. 12, pp. 127-154, 1996.
[74] A. Sutcliffe, “Requirements Rationales: Integrating Approaches to Requirements Analysis,” Proc. Designing Interactive Systems, pp. 33-42, 1995.
[75] M. Tawbi, C. Ben Achour, and F. Velez, “Guiding the Process of Requirements Elicitation through Scenario,” Proc. 10th Int'l Workshop Database and Expert Systems Applications, pp. 345-349, 1999.
[76] G.W. Noblit and R.D. Hare, Meta-Ethnography: Synthesizing Qualitative Studies. Sage, 1988.
[77] B.L. Paterson, S.E. Thorne, C. Canam, and C. Jillings, Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis. Sage, 2001.
[78] M. Dixon-Woods, S. Agarwal, D. Jones, B. Young, and A. Sutton, “Synthesising Qualitative and Quantitative Evidence: A Review of Possible Methods,” J. Health Services Research and Policy, vol. 10, pp. 45-53, 2005.
[79] M. Mahmood, “System Development Methods—A Comparative Investigation,” MIS Quarterly, vol. 11, pp. 293-311, 1987.
[80] P.D. Chatzoglou, “Factors Affecting Completion of the Requirements Capture Stage of Projects with Different Characteristics,” Information and Software Technology, vol. 39, pp. 627-640, 1997.
[81] M. Tan, “Establishing Mutual Understanding in Systems Design: An Empirical Study,” J. Management Information Systems, vol. 10, pp. 159-182, 1994.
[82] J.W. Moody, J.E. Blanton, and P.H. Cheney, “A Theoretically Grounded Approach to Assist Memory Recall during Information Requirements Determination,” J. Management Information Systems, vol. 15, pp. 79-89, 1998.
[83] R.W. Zmud, W.P. Anthony, and R.M. Stair, “The Use of Mental Imagery as a Requirements Analysis Technique,” Challenges and Strategies for Research in Systems Development, W.W. Cotterman and J.A. Senn, eds., pp. 337-350, John Wiley & Sons, 1992.
[84] R. Lines, E. Breivik, and M. Supphellen, “Elicitation of Attributes: A Comparison of Preference Model Structures Derived from Two Elicitation Techniques,” Proc. 24th European Marketing Academy Conf., pp. 641-656, 1995.
[85] A.M. Burton, N.R. Shadbolt, A.P. Hedgecock, and G.A. Rugg, “A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1,” Proc. First European Workshop Knowledge Acquisition for Knowledge-Based Systems, 1987.
[86] R. Lines, E. Breivik, and M. Supphellen, “Method Effects in Attribute Elicitation: A Multidimensional Comparison of Four Techniques,” 1995.
[87] A.M. Burton and N. Shadbolt, “A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems: Domain 1,” Proc. SERC Workshop Knowledge Acquisition for Eng. Applications, pp. 20-28, 1987.
[88] O. Dieste, A. Griman, and N. Juristo, “Developing Search Strategies for Detecting Relevant Experiments,” Empirical Software Eng., vol. 14, pp. 513-539, 2009.
[89] S.L. Pfleeger, “Soup or Art? The Role of Evidential Force in Empirical Software Engineering,” IEEE Software, vol. 22, no. 1, pp. 66-73, Jan./Feb. 2005.
[90] S.E. Straus, W.S. Richardson, P. Glasziou, and R.B. Haynes, Evidence-Based Medicine. Churchill Livingstone, 2005.
[91] L.V. Hedges and I. Olkin, Statistical Methods for Meta-Analysis. Academic Press, 1985.
[92] N. Juristo, A.M. Moreno, S. Vegas, and M. Solari, “In Search of What We Empirically Know about Unit Testing,” IEEE Software, vol. 23, no. 6, pp. 72-80, Nov.-Dec. 2006.
[93] T. Dyba, V. Kampenes, and D. Sjoberg, “A Systematic Review of Statistical Power in Software Engineering Experiments,” Information and Software Technology, vol. 48, pp. 745-755, 2006.
[94] N. Juristo, A.M. Moreno, and S. Vegas, “Reviewing 25 Years of Testing Technique Experiments,” J. Empirical Software Eng., vol. 9, pp. 7-44, 2004.
[95] P.G. Shekelle, M.L. Hardy, S.C. Morton, M. Maglione, W.A. Mojica, M.J. Suttorp, S.L. Rhodes, L. Jungvig, and J. Gagne, “Efficacy and Safety of Ephedra and Ephedrine for Weight Loss and Athletic Performance: A Meta-Analysis,” J. Am. Medical Assoc., vol. 289, pp. 1537-1545, Mar. 2003.
[96] D. Moher, A.R. Jadad, G. Nichol, M. Penman, P. Tugwell, and S. Walsh, “Assessing the Quality of Randomized Controlled Trials: An Annotated Bibliography of Scales and Checklists,” Controlled Clinical Trials, vol. 16, pp. 62-73, Feb. 1995.
[97] J.P.T. Higgins and S. Green, Cochrane Handbook for Systematic Reviews of Interventions. John Wiley and Sons, 2006.
[98] O.P. Brereton, B. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Employing Systematic Literature Review: An Experience Report,” Technical Report TR/05-01, School of Computing & Math., Keele Univ., 2005.
[99] B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg, “Preliminary Guidelines for Empirical Research in Software Engineering,” IEEE Trans. Software Eng., vol. 28, no. 8, pp. 721-734, Aug. 2002.
[100] W. Shadish, T. Cook, and D.T. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, 2001.
[101] Construx Software, “Context Free Requirements Questions,” http://www.construx.comPage.aspx?hid=1742 , 2008.
[102] D.T. Campbell, “Reforms as Experiments,” Am. Psychologist, vol. 24, pp. 409-429, 1969.

Index Terms:
Elicitation methods, performance measures, experimentation, systematic literature review.
Oscar Dieste, Natalia Juristo, "Systematic Review and Aggregation of Empirical Studies on Elicitation Techniques," IEEE Transactions on Software Engineering, vol. 37, no. 2, pp. 283-304, March-April 2011, doi:10.1109/TSE.2010.33
Usage of this product signifies your acceptance of the Terms of Use.