This Article 
 Bibliographic References 
 Add to: 
Sequential Model Criticism in Probabilistic Expert Systems
March 1993 (vol. 15 no. 3)
pp. 209-219

Probabilistic expert systems based on Bayesian networks require initial specification of both qualitative graphical structure and quantitative conditional probability assessments. As (possibly incomplete) data accumulate on real cases, the parameters of the system may adapt, but it is also essential that the initial specifications be monitored with respect to their predictive performance. A range of monitors based on standardized scoring rules that are designed to detect both qualitative and quantitative departures from the specified model is presented. A simulation study demonstrates the efficacy of these monitors at uncovering such departures.

[1] S. K. Andersen, K. G. Olesen, F. V. Jensen, and F. Jensen, "HUGIN--A shell for building Bayesian belief universes for expert systems," inProc. 11th Int. Joint Conf. Artificial Intell.(Detroit), 1989, pp. 1080-1085.
[2] J. M. Bernardo, "Expected information as expected utility,"Ann. Stat., vol. 7, pp. 686-690, 1979.
[3] G. Cooper and E. Herskovits, "A Bayesian method for the induction of probabilistic networks from data,"Machine Learning, to be published.
[4] R. G. Cowell, "Calculating moments of decomposable functions in Bayesian networks," Preprint, 1992.
[5] R. G. Cowell, "BAIES--A probabilistic expert system shell with qualitative and quantitative learning," inBayesian Statistics 4(J. M. Bernardo, J. Berger, A. P. Dawid, and A. F. M. Smith, Eds.). Oxford, UK: Oxford University Press, 1992, pp. 595-600.
[6] R. G. Cowell and A. P. Dawid, "Fast retraction of evidence in a probabilistic expert system,"Stat. Comput., vol. 2, pp. 37-40, 1992.
[7] A. P. Dawid, "Present position and potential developments: Some personal views. Statistical theory. The prequential approach (with discussion),"J. Royal Stat. Soc., ser. A, vol. 174, pp. 278-292, 1984.
[8] A. P. Dawid, "Probability forecasting," inEncyclopedia of statistical Sciences(S. Kotz, N. L. Johnson, and C. B. Read, Eds.). New York: Wiley, 1986, pp. 210-218, vol. 7.
[9] A. P. Dawid "Fisherian inference in likelihood and prequential frames of reference (with discussion),"J. Royal Stat. Soc., ser. B, vol. 53, pp. 79-109, 1991.
[10] A. P. Dawid, "Applications of a general propagation algorithm for probabilistic expert systems,"Stat. Comput., vol. 2, pp. 25-36, 1992.
[11] A. P. Dawid, "Prequential analysis, stochastic complexity and Bayesian inference," inBayesian Statistics 4(J. M. Bernardo, J. Berger, A. P. Dawid, and A. F. M. Smith, Eds.). Oxford, UK: Oxford University Press, 1992, pp. 109-126.
[12] A. P. Dempster, N. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm (with discussion),"J. Royal Stat. Soc., ser. B, vol. 39, pp. 1-38, 1977.
[13] D. Edwards and T. Havránek, "A fast model selection procedure for large families of models,"J. Amer. Stat. Assoc., vol. 82, pp. 205-11, 1987.
[14] C. Glymour, R. Scheines, P. Spirtes, and K. Kelley,Discovering Causal Structure. New York: Academic, 1987.
[15] F. V. Jensen, S. L. Lauritzen, and K. G. Olesen, "Bayesian updating in causal probabilistic networks by local computations,"Comput. Stat. Quar., vol. 4, pp. 269-282, 1990.
[16] S. L. Lauritzen, "The EM-algorithm for graphical association models with missing data," Res. Rep. R-91-05, Inst. Elec. Syst., Aalborg University, Denmark, 1991.
[17] S. L. Lauritzen and D. J. Spiegelhalter, "Local computations with probabilities on graphical structures and their application to expert systems (with discussion),"J. Royal Stat. Soc., ser. B, vol. 50, pp. 157-224, 1988.
[18] A. H. Murphy and R. L. Winkler, "Probability forecasting in meteorology,"J. Amer. Stat. Assoc., vol. 79, pp. 489-500, 1984.
[19] J. Pearl,Probabilistic Inference in Intelligent Systems. San Mateo: Morgan Kaufmann, 1988.
[20] J. Pearl and T. Verma, "A theory of inferred causality," inProc. 2nd Int. Conf. Principles Knowledge Represent., 1991, pp. 441-52.
[21] F. Seillier-Moiseiwitsch and A. P. Dawid, "On testing the validity of sequential probability forecasts,"J. Amer. Stat. Assoc., to be published.
[22] F. Seillier-Moiseiwitsch, T. J. Sweeting, and A. P. Dawid, "Prequential tests of model fit,"Scand. J. Stat., vol. 19, pp. 45-60, 1992.
[23] D. J. Spiegelhalter and R. G. Cowell, "Learning in probabilistic expert systems,"Bayesian Statistics 4(J. M. Bernardo, J. Berger, A. P. Dawid, and A. F. M. Smith, Eds.). Oxford, UK: Oxford University Press, 1992, pp. 447-465.
[24] D. J. Spiegelhalter, N. L. Harris, K. Bull, and R. C. G. Franklin, "Empirical evaluation of prior beliefs about frequencies: methodology and a case study in congenital heart disease," Tech. Rep. 91-4, MRC Biostat. Unit, Cambridge, UK, 1991.
[25] D. J. Spiegelhalter, A. P. Dawid, S. L. Lauritzen, and R. G. Cowell, "Bayesian analysis in expert systems," to be published inStat. Sci.
[26] D. J. Spiegelhalter and S. L. Lauritzen, "Sequential updating of conditional probabilities on directed graphical structures,"Networks, vol. 20, pp. 579-605, 1990.
[27] B. Thiesson, "(G)EM algorithms for recursive graphical association models with missing data," Thesis for the degree Cand. Scient., Aalborg Univ., Denmark, 1991.

Index Terms:
sequential model criticism; probabilistic expert systems; Bayesian networks; qualitative graphical structure; quantitative conditional probability assessments; scoring rules; Bayes methods; expert systems; probabilistic logic
R.G. Cowell, A.P. Dawid, D.J. Spiegelhalter, "Sequential Model Criticism in Probabilistic Expert Systems," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 15, no. 3, pp. 209-219, March 1993, doi:10.1109/34.204903
Usage of this product signifies your acceptance of the Terms of Use.