This Article 
 Bibliographic References 
 Add to: 
Dealing with the Expert Inconsistency in Probability Elicitation
July/August 2000 (vol. 12 no. 4)
pp. 499-508

Abstract—In this paper, we present and discuss our experience in the task of probability elicitation from experts for the purpose of belief network construction. In our study, we applied four techniques. Three of these techniques are available from the literature, whereas the fourth one is a technique that we developed by adapting a method for the assessment of preferences to the task of probability elicitation. The new technique is based on the Analytic Hierarchy Process (AHP) proposed by Saaty [12], [13], and it allows for the quantitative assessment of the expert inconsistency. The method is, in our opinion, very promising and lends itself to be applied more extensively to the task of probability elicitation.

[1] B.G. Buchanan, J. Moore, D. Forsythe, G. Carenini, G. Banks, and S. Ohlsson, “An Intelligent Interactive System for Delivering Individualized Information to Patients,” Artificial Intelligence in Medicine, vol. 7, no. 2, pp. 117–154, 1995.
[2] G. Carenini, S. Monti, and G. Banks, “An Information-based Bayesian Approach to History-taking,” Proc. Fifth Conf. AI in Medicine, Europe, pp. 129–138, 1995.
[3] G.F. Cooper and E. Herskovits, “A Bayesian Method for the Induction of Probabilistic Networks from Data,” Machine Learning, vol. 9, pp. 309–347, 1992.
[4] V. Coupé and L.C. van der Gaag, “Practicable Sensitivity Analysis of Bayesian Belief Networks,” Proc. Joint Session Sixth Prague Symp. Asymptotic Statistics and the 13th Prague Conf. Information Theory, Statistical Decision Functions and Random Processes, pp. 81–86, 1998.
[5] E.H. Forman and T.L. Saaty, Expert Choice, Inc.,
[6] , B.L. Golden, E.A. Wasil, and P.T. Harker, eds. The Analytic Hierarchy Process: Applications and Studies. Springer Verlag, 1989.
[7] D. Heckerman, D. Geiger, and D.M. Chickering, “Learning Bayesian Networks: The Combination of Knowledge and Statistical Data,” Machine Learning, vol. 20, pp. 197–243, 1995.
[8] M.I. Jordan, ed., Learning in Graphical Models. NATO Science Series, Kluwer Academic, 1998.
[9] W. Lam and F. Bacchus, “Learning Bayesian Belief Networks—An Approach Based on the MDL Principle,” Computational Intelligence, vol. 10, no. 4, pp. 269–293, 1994.
[10] R.A. Miller, H. Pople, and J. Meyers, “Internist-1, An Experimental Computer-Based Diagnostic Consultant for General Internal Medicine,” New England J. Medicine, vol. 307, no. 8 1982. Also in Readings in Medical Artificial Intelligence: The First Decade, W.J. Clancey and E.H. Shortliffe, eds. Reading, Massachusetts: Addison-Wesley, pp. 190–209.
[11] M. Pradhan, M. Henrion, G. Provan, B. Del Favero, and K. Huang, “The Sensitivity of Belief Networks to Imprecise Probabilities: An Experimental Investigation,” Artificial Intelligence, vol. 85,no. 1–2, pp. 363–397, 1996.
[12] T.L. Saaty, The Analytic Hierarchy Process. McGraw-Hill, 1980.
[13] T.L. Saaty, Fundamentals of Decision Making and Priority Theory with the Analytic Hierarchy Process. Pittsburgh, Pennsylvania: RWS Publications, vol. VI, 1994.
[14] J. Saper, Handbook of Headache Management. Williams and Wilkins, 1993.
[15] S. Schocken, “Ratio-scale Elicitation of Subjective Degrees of Support,” NYU-CRIS Working Paper IS-93-30, 1993.
[16] S. Solomon and S. Fraccaro, The Headache Book. Consumers Union of the United States, 1991.
[17] D. Spiegelhalter, A. Dawid, S. Lauritzen, and R. Cowell, “Bayesian Analysis in Expert Systems,” Statistical Science, vol. 8, no. 3, pp. 219–283, 1993.
[18] D. Spiegelhalter, R.C.G. Franklin, and K. Bull, “Assessment, Criticism, and Improvement of Imprecise Subjective Probabilities for a Medical Expert System,” Uncertainty in Artificial Intelligence, M. Henrion, R. Shachter, L. Kanal, and J. Lemmer, eds., NorthHolland, vol. 5, pp. 285–294, 1990.
[19] D. von Winterfeldt and W. Edwards, Decision Analysis and Behavioral Research. Cambridge University Press, 1986.

Index Terms:
Bayesian belief networks, analytic hierarchy process, probability elicitation.
Stefano Monti, Giuseppe Carenini, "Dealing with the Expert Inconsistency in Probability Elicitation," IEEE Transactions on Knowledge and Data Engineering, vol. 12, no. 4, pp. 499-508, July-Aug. 2000, doi:10.1109/69.868903
Usage of this product signifies your acceptance of the Terms of Use.