This Article 
 Bibliographic References 
 Add to: 
Elicitation of Knowledge from Multiple Experts Using Network Inference
September-October 1997 (vol. 9 no. 5)
pp. 688-696

Abstract—Eliciting knowledge from multiple experts usually entails the use of groups, and thus is subject to the problems inherent in group dynamics. We present a technique for multiple expert knowledge acquisition that does not rely upon the use of groups and can take advantage of technological advances in communications and computing, i.e., the Internet. The approach uses influence diagrams to represent the individual expert's understanding of the problem situation and develops a Multiple Expert Influence Diagram (MEID), a composite representation of the experts' knowledge. Following a review of present methods for multiple expert knowledge elicitation, we formally define the MEID, describe its manner of construction, and discuss its interpretation. We continue with a review of the issues to be faced in implementation of the technique, and give an illustrative example. Finally, we emphasize the need to provide users of decision aids with defensible measures of the quality of the rules produced by these aids. The MEID-approach is intended to serve as a first step in this direction.

[1] O. Gur-Ali and W.A. Wallace, "Induction of Rules Subject to a Quality Constraint: Probabilistic Inductive Learning," IEEE Trans. Knowledge and Data Eng., vol. 5, no. 6, pp. 979-984, Dec. 1993.
[2] A. Massey and W.A. Wallace, "Focus Groups as a Knowledge Elicitation Technique: An Exploratory Study," IEEE Trans. Knowledge and Data Eng., vol. 3, no. 2, pp. 193-200, June 1991.
[3] P.R. Kleindorfer, H.C. Kunreuther, and P.J.H. Schoemaker, Decision Sciences: An Integrative Perspective.New York: Cambridge Univ. Press, 1993.
[4] S.R. LeClair, "Interactive Learning: A Multiexpert Paradigm for Acquiring New Knowledge," Sigart Newsletter (knowledge acquisition special issue), no. 108, pp. 34-44, Apr. 1989.
[5] J. Boose,Expertise Transfer for Expert System Design, Elsevier Science, Amsterdam, 1986.
[6] E. Turban and M. Tan, "Methods for Knowledge Acquisition from Multiple Experts—An Assessment," Int'l J. Applied Expert Systems, vol. 1, pp 101-119, 1993.
[7] L. Medsjer, M. Tan, and E. Turban, "Knowledge Acquisition from Multiple Experts: Problems and Issues," Expert Systems with Applications, vol. 9, no. 1, pp. 35-40, 1995.
[8] S. Mittal and C.L. Dym, "Knowledge Acquisition from Multiple Experts," AI Magazine, vol. 6, pp. 32-36, Summer 1985.
[9] C.J. Moore and J.C. Miles, "Knowledge Acquisition Using More than One Expert to Cover the Same Domain," Artificial Intelligence Rev., vol. 5, pp. 255-271, 1991.
[10] K.L. McGraw and M.R. Seale, "Knowledge Elicitation with Multiple Experts: Considerations and Techniques," Artificial Intelligence Rev., vol. 2, pp. 31-44, 1988.
[11] S.M. Alexander and G.W. Evans, "The Integration of Multiple Experts: A Review of Methodologies," Applied Expert Systems, E. Turban and P. Watkins, eds., North-Holland, 1988.
[12] A. Massey and W.A. Wallace, "Understanding and Facilitating Group Problem Structuring and Formulation: Mental Representations Presentation Aids," Decision Support Systems, vol. 17, pp. 253-274, 1996.
[13] S.R. LeClair, "A Multi-Expert Knowledge System Architecture for Manufacturing Decision Analysis," unpublished PhD dissertation, Arizona State Univ., 1995.
[14] A.J. LaSalle and L.R. Medsker, "Computerized Conferencing for Knowledge Acquisition from Multiple Experts," Expert Systems With Applications, vol. 3, pp. 517-522, 1991.
[15] J. Gallagher and R.E. Kraut, "Computer-Mediated Communication for Intellectual Teamwork: A Field Experiment in Group Writing," Proc. Conf. Computer-Supported Cooperative Work,Los Angeles, 1990.
[16] S. Easterbrook, "Distributed Knowledge Acquisition as a Model for Requirements Elicitation," Proc. Third European Knowledge Acquisition Workshop, pp. 530-543,Paris, 1989.
[17] D.M. Steier, R.L. Lewis, J.F. Lehman, and A.L. Zacheri, "Combining Multiple Knowledge Sources in an Integrated Intelligent System," IEEE Expert, vol. 8, no. 3, pp. 35-43, 1993.
[18] R. Dieng, A. Giboin, P.-A. Tourtier, and O. Corby, "Knowledge Acquisition for Explainable, Multi-Expert Knowledge-Based Design Systems," Proc. EKAW-92, Current Developments in Knowledge, pp. 298-317,Berlin/Heidelberg, 1992.
[19] J. McGovern, D. Samson, and A. Wirth, "Using Case-Based Reasoning for Basis Development in Intelligent Decision Systems," European J. Operational Research, vol. 77, pp 40-59, 1994.
[20] H. Raiffa, Decision Analysis. Addison-Wesley, 1968.
[21] R. Shachter, "Evaluating Influence Diagrams," Operations Research, vol. 34, no. 6, pp. 871-882, Nov-Dec. 1986.
[22] J.W. Egar and M.A. Musen, "Graph-Grammar Assistance for Automated Generation of Influence Diagrams," IEEE Trans. Systems, Man, and Cybernetics, vol. 24, pp. 1,625-1,642, 1994.
[23] D. Banks and K. Carley, "Metric Inference for Social Networks," J. Classification, vol. 11, pp. 121-149, 1994.
[24] K Langfield-Smith and A. Wirth, "Measuring Differences Between Cognitive Maps," J. Operational Research Soc., vol. 43, pp. 1,135-1,150, 1992.
[25] C.L. Mallows, "Non-Null Ranking Models I," Biometrics, vol. 44, pp. 114-130, 1957.
[26] D. Banks, "Bootstrapping II," Encyclopedia of Statistical Science, S. Kotz, N. Johnson, and C. Read, eds., Wiley, 1989.
[27] R.L Keeney, Value-Focused Thinking: A Path to Creative Decision-Making, Harvard Univ. Press, 1992.
[28] D.G. Johnson and J.M. Mulvey, "Accountability and Computer Decision Systems," Comm. ACM, vol. 38, no. 12, pp. 58-64, 1995.
[29] W.A. Wallace, "Introduction," Ethics in Modeling, W.A. Wallace, ed., Pergamon Press, Oxford, 1994.

Index Terms:
Knowledge acquisition, multiple experts, network inference, quality of rules, influence diagram.
Robert Rush, William A. Wallace, "Elicitation of Knowledge from Multiple Experts Using Network Inference," IEEE Transactions on Knowledge and Data Engineering, vol. 9, no. 5, pp. 688-696, Sept.-Oct. 1997, doi:10.1109/69.634748
Usage of this product signifies your acceptance of the Terms of Use.