This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Recursive Unsupervised Learning of Finite Mixture Models
May 2004 (vol. 26 no. 5)
pp. 651-656
Zoran Zivkovic, IEEE Computer Society
Ferdinand van der Heijden, IEEE Computer Society

Abstract—There are two open problems when finite mixture densities are used to model multivariate data: the selection of the number of components and the initialization. In this paper, we propose an online (recursive) algorithm that estimates the parameters of the mixture and that simultaneously selects the number of components. The new algorithm starts with a large number of randomly initialized components. A prior is used as a bias for maximally structured models. A stochastic approximation recursive learning algorithm is proposed to search for the maximum a posteriori (MAP) solution and to discard the irrelevant components.

[1] H. Akaike, A New Look at the Statistical Model Identification IEEE Trans. Automatic Control, vol. 19, no. 6, pp. 716-723, 1974.
[2] E. Anderson, The Irises of the Gaspe Peninsula Bull. of the Am. Iris Soc., vol. 59, 1935.
[3] M.E. Brand, Structure Learning in Conditional Probability Models via an Entropic Prior and Parameter Extinction Neural Computation J., vol. 11, no. 5, pp. 1155-1182, 1999.
[4] A.P. Dempster, N. Laird, and D.B. Rubin, Maximum Likelihood from Incomplete Data via the EM Algorithm J. Royal Statistical Soc., Series B (Methodological), vol. 1, no. 39, pp. 1-38, 1977.
[5] V. Fabian, On Asymptotically Efficient Recursive Estimation Annals of Statistics, vol. 6, pp. 854-866, 1978.
[6] M.A.T. Figueiredo and A.K. Jain, Unsupervised Learning of Finite Mixture Models IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, pp. 381-396, 2002.
[7] A. Gelman, J.B. Carlin, H.S. Stern, and D.B. Rubin, Bayesian Data Analysis. Chapman and Hall, 1995.
[8] G. McLachlan and D. Peel, Finite Mixture Models. John Wiley and Sons, 2000.
[9] R.M. Neal and G.E. Hinton, A New View of the EM Algorithm that Justifies Incremental, Sparse and Other Variants Learning in Graphical Models, pp. 355-368, M.I. Jordan ed., 1998.
[10] C. Rasmussen, The Infinite Gaussian Mixture Model Advances in Neural Information Processing Systems, vol. 12, pp. 554-560, 2000.
[11] S. Richardson and P. Green, On Bayesian Analysis of Mixture Models with Unknown Number of Components J. Royal Statistical Soc., Series B (Methodological), vol. 59, no. 4, pp. 731-792, 1997.
[12] J. Rissansen, Stochastic Complexity J. Royal Statistical Soc., Series B (Methodological), vol. 49, no. 3, pp. 223-239, 1987.
[13] J. Sacks, Asymptotic Distribution of Stochastic Approximation Procedures Annals of Math. Statistics, vol. 29, pp. 373-405, 1958.
[14] G. Schwarz, Estimating the Dimension of a Model Annals of Statistics, vol. 6, no. 2, pp. 461-464, 1978.
[15] D.M. Titterington, Recursive Parameter Estimation Using Incomplete Data J. Royal Statistical Soc., Series B (Methodological), vol. 2, no. 46, pp. 257-267, 1984.
[16] D.M. Titterington, A.F.M. Smith, and U.E. Makov, Statistical Analysis of Finite Mixture Distributions. John Wiley and Sons, 1985.
[17] N. Ueda and R. Nakano, Deterministic Annealing EM Algorithm Neural Networks, vol. 11, pp. 271-282, 1998.
[18] N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton, SMEM Algorithm for Mixture Models Neural Computation, vol. 12, no. 9, pp. 2109-2128, 2000.
[19] J.J. Verbeek, N. Vlassis, and B. Krose, Efficient Greedy Learning of Gaussian Mixture Models Neural Computation, vol. 15, no. 1, 2003.
[20] C. Wallace and P. Freeman, Estimation and Inference by Compact Coding J. Royal Statistical Soc., Series B (Methodological), vol. 49, no. 3, pp. 240-265, 1987.

Index Terms:
Online (recursive) estimation, unsupervised learning, finite mixtures, model selection, EM-algorithm.
Citation:
Zoran Zivkovic, Ferdinand van der Heijden, "Recursive Unsupervised Learning of Finite Mixture Models," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 5, pp. 651-656, May 2004, doi:10.1109/TPAMI.2004.1273970
Usage of this product signifies your acceptance of the Terms of Use.