This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Variational Maximum A Posteriori by Annealed Mean Field Analysis
November 2005 (vol. 27 no. 11)
pp. 1747-1761
Gang Hua, IEEE
Ying Wu, IEEE
This paper proposes a novel probabilistic variational method with deterministic annealing for the maximum a posteriori (MAP) estimation of complex stochastic systems. Since the MAP estimation involves global optimization, in general, it is very difficult to achieve. Therefore, most probabilistic inference algorithms are only able to achieve either the exact or the approximate posterior distributions. Our method constrains the mean field variational distribution to be multivariate Gaussian. Then, a deterministic annealing scheme is nicely incorporated into the mean field fix-point iterations to obtain the optimal MAP estimate. This is based on the observation that when the covariance of the variational Gaussian distribution approaches to zero, the infimum point of the Kullback-Leibler (KL) divergence between the variational Gaussian and the real posterior will be the same as the supreme point of the real posterior. Although global optimality may not be guaranteed, our extensive synthetic and real experiments demonstrate the effectiveness and efficiency of the proposed method.

[1] C. Andrieu and A. Doucet, “Joint Bayesian Model Selection and Esitimation of Noisy Sinusoids via Reversible Jump MCMC,” IEEE Trans. Signal Processing, vol. 47, no. 10, pp. 2667-2676, 1999.
[2] Z. Tu and S.-C. Zhu, “Image Segmentation by Data-Driven Markov Chain Monte Carlo,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 657-673, May 2002.
[3] A. Barbu and S.-C. Zhu, “Graph Partition by Swendsen-Wang Cut,” Proc. IEEE Int'l Conf. Computer Vision, 2003.
[4] W.T. Freeman and E.C. Pasztor, “Learning Low-Level Vision,” Proc. IEEE Int'l Conf. Computer Vision, pp. 1182-1189, 1999.
[5] W.T. Freeman and E.C. Pasztor, “Markov Network for Low-Level Vision,” technical report, MERL, Mitsubish Electric Research Laboratory, 1999.
[6] Y. Wu and T.S. Huang, “Robust Visual Tracking by Co-Inference Learning,” Proc. IEEE Int'l Conf. Computer Vision, vol. 2, pp. 26-33, 2001.
[7] Y. Wu, G. Hua, and T. Yu, “Tracking Articulated Body by Dynamic Markov Network,” Proc. IEEE Int'l Conf. Computer Vision, pp. 1094-1101, 2003.
[8] L. Sigal, M. Isard, B. Sigelman, and M. Black, “Attractive People: Assembling Loose-Limbed Models Using Non-Parametric Belief Propagation,” Advances in Neural Information Processing System 16, MIT Press, 2004.
[9] J. Yedidia, W. Freeman, and Y. Weiss, “Understanding Belief Propagation and Its Generalization,” Exploring Artificial Intelligence in the New Millenium, chapter 8, pp. 239-286, Elsevier Science and Technology Books, 2003.
[10] G. Hua, Y. Wu, and T. Yu, “Analyzing Structured Deformable Shapes via Mean Field Monte Carlo,” Proc. IEEE Asia Conf. Computer Vision, 2004.
[11] M.J. Beal, “Variational Algorithms for Approximate Bayesian Inference,” PhD Thesis, Gatsby Computational Neuroscience Unit, Univ. College, London, 2003.
[12] J.M. Winn, “Variational Message Passing and Its Application,” PhD thesis, Dept. of Physics, Univ. of Cambridge, 2003.
[13] N. Metropolis, A.W. Rosenbluth, M.N. Rosenbluth, A.H. Teller, and E. Teller, “Equations of State Calculations by Fast Computing Machine,” J. Chemical Physics, vol. 21, pp. 1087-1091, 1953.
[14] J.M.P.V.S. Kirkpatrick and C.D. Gelatt, “Optimization by Simulated Annealing,” Science, vol. 220, no. 4598, pp. 671-680, 1983.
[15] S. Geman and D. Geman, “Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 6, pp. 721-741, June 1984.
[16] A.L. Yuille and J.J. Kosowsky, “Statistical Physics Algorithms that Converge,” Neural Computation, vol. 6, no. 3, pp. 341-356, June 1994.
[17] J. Puzicha, T. Hofmann, and J.M. Buhmann, “Deterministic Annealing: Fast Physical Heuristics for Real-Time Optimization of Large Systems,” Proc. 15th IMACS World Conf. Scientific Computation, Modelling and Applied Math., 1997.
[18] J. Puzicha and J.M. Buhmann, “Multiscale Annealing for Grouping and Unsupervised Texture Segmentation,” Computer Vision and Image Understanding (CVIU), vol. 76, no. 3, pp. 213-230, 1999.
[19] S.Z. Li, “Robustizing Robust M-Estimation Using Deterministic Annealing,” Pattern Recognition, vol. 29, no. 1, pp. 159-166, 1996.
[20] K.P. Murphy, “Dynamic Bayesian Networks: Representation, Inference and Learning,” PhD thesis, Computer Science Division, Univ. of California, Berkeley, 2002.
[21] V.I. Pavlovic, “Dynamic Bayesian Networks for Information Fusion with Application to Human-Computer Interfaces,” PhD thesis, Dept. of Electrical and Computer Eng., Univ. of Illinois at Urbana-Champaign, 1999.
[22] G. Hua and Y. Wu, “Multi-Scale Visual Tracking by Sequential Belief Propagation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 826-833, 2004.
[23] M. Jordan and Y. Weiss, “Graphical Models: Probabilistic Inference,” The Handbook of Brain Theory and Neural Network, second ed. MIT Press, pp. 243-266, 2002.
[24] Y. Wang, T. Tan, and K.-F. Loe, “Video Segmentation Based on Graphical Models,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 335-342, 2003.
[25] Y. Wu, G. Hua, and T. Yu, “Switching Observation Models for Contour Tracking in Clutter,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 295-302, 2003.
[26] Y. Wu, T. Yu, and G. Hua, “Tracking Appearances with Occlusions,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 789-795, 2003.
[27] K. Murphy, Y. Weiss, and M. Jordan, “Loopy-Belief Propagation for Approximate Inference: An Empirical Study,” Proc. 15th Conf. Uncertainty in Artificial Intelligence, 1999.
[28] W.T. Freeman and H. Zhang, “Shape-Time Photography,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2003.
[29] A. Blake and M. Isard, Active Contours. Springer-Verlag, 1998.
[30] M. Isard and A. Blake, “Contour Tracking by Stochastic Propagation of Conditional Density,” Proc. European Conf. Computer Vision, vol. 1, pp. 343-356, 1996.
[31] M. Isard and A. Blake, “Condensation-Conditional Density Propagation for Visual Tracking,” Int'l J. Computer Vision, vol. 29, no. 1, pp. 5-28, 1998.
[32] T.S. Jaakkola, “Tutorial on Variational Approximation Method,” Advanced Mean Field Methods: Theory and Practice, MIT Press, 2000.
[33] E. Sudderth, A. Ihler, W. Freeman, and A. Willsky, “Nonparametric Belief Propagation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 605-612, 2003.
[34] M. Isard, “Pampas: Real-Valued Graphical Models for Computer Vision,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 613-620, 2003.
[35] A.V. Rao, D.J. Miller, K. Rose, and A. Gersho, “A Deterministic Annealing Approach for Parsimonious Design of Piecewise Regression Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 2, pp. 159-173, Feb. 1999.
[36] D. Doll and W. von Seelen, “Object Recognition by Deterministic Annealing,” Image and Vision Computing, vol. 15, pp. 855-860, 1997.
[37] J. Deutscher, A. Blake, and I. Reid, “Articulated Body Motion Capture by Annealed Particle Filtering,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000.
[38] T.M. Cover and J.A. Thomas, Elements of Information Theory, D.L. Schilling, ed., second ed. New York: John Wiley and Sons, 1991.
[39] J.B. Rosen, “The Gradient Projection Method for Nonlinear Programming. Part I. Linear Constraints,” J. Soc. Industrial and Applied Math., vol. 8, no. 1, pp. 181-217, Mar. 1960.
[40] J.B. Rosen, “The Gradient Projection Method for Nonlinear Programming. Part II. Nonlinear Constraints,” J. Soc. Industrial and Applied Math., vol. 9, no. 4, pp. 514-532, Dec. 1961.

Index Terms:
Index Terms- Mean field variational analysis, deterministic annealing, maximum a posteriori estimation, graphical model, Markov network.
Citation:
Gang Hua, Ying Wu, "Variational Maximum A Posteriori by Annealed Mean Field Analysis," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 11, pp. 1747-1761, Nov. 2005, doi:10.1109/TPAMI.2005.229
Usage of this product signifies your acceptance of the Terms of Use.