The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.05 - May (2013 vol.35)
pp: 1121-1134
Jia Zeng , Sch. of Comput. Sci. & Technol., Soochow Univ., Suzhou, China
W. K. Cheung , Dept. of Comput. Sci., Hong Kong Baptist Univ., Hong Kong, China
Jiming Liu , Dept. of Comput. Sci., Hong Kong Baptist Univ., Hong Kong, China
ABSTRACT
Latent Dirichlet allocation (LDA) is an important hierarchical Bayesian model for probabilistic topic modeling, which attracts worldwide interest and touches on many important applications in text mining, computer vision and computational biology. This paper represents the collapsed LDA as a factor graph, which enables the classic loopy belief propagation (BP) algorithm for approximate inference and parameter estimation. Although two commonly used approximate inference methods, such as variational Bayes (VB) and collapsed Gibbs sampling (GS), have gained great success in learning LDA, the proposed BP is competitive in both speed and accuracy, as validated by encouraging experimental results on four large-scale document datasets. Furthermore, the BP algorithm has the potential to become a generic scheme for learning variants of LDA-based topic models in the collapsed space. To this end, we show how to learn two typical variants of LDA-based topic models, such as author-topic models (ATM) and relational topic models (RTM), using BP based on the factor graph representations.
INDEX TERMS
Indexes, Approximation algorithms, Hidden Markov models, Approximation methods, Joints, Inference algorithms, Computational modeling, variational Bayes, Latent Dirichlet allocation, topic models, belief propagation, message passing, factor graph, Bayesian networks, Markov random fields, hierarchical Bayesian models, Gibbs sampling
CITATION
Jia Zeng, W. K. Cheung, Jiming Liu, "Learning Topic Models by Belief Propagation", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.35, no. 5, pp. 1121-1134, May 2013, doi:10.1109/TPAMI.2012.185
REFERENCES
[1] D.M. Blei, A.Y. Ng, and M.I. Jordan, "Latent Dirichlet Allocation," J. Machine Learning Research, vol. 3, pp. 993-1022, 2003.
[2] T.L. Griffiths and M. Steyvers, "Finding Scientific Topics," Proc. Nat'l Academy of Sciences USA, vol. 101, pp. 5228-5235, 2004.
[3] M. Rosen-Zvi, T. Griffiths, M. Steyvers, and P. Smyth, "The Author-Topic Model for Authors and Documents," Proc. 20th Conf. Uncertainty in Artificial Intelligence, pp. 487-494. 2004,
[4] J. Chang and D.M. Blei, "Hierarchical Relational Models for Document Networks," Ann. Applied Statistics, vol. 4, no. 1, pp. 124-150, 2010.
[5] T. Minka and J. Lafferty, "Expectation-Propagation for the Generative Aspect Model," Proc. 18th Conf. Uncertainty in Artificial Intelligence, pp. 352-359, 2002.
[6] Y.W. Teh, D. Newman, and M. Welling, "A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation," Proc. Neural Information Processing Systems, pp. 1353-1360, 2007.
[7] A. Asuncion, M. Welling, P. Smyth, and Y.W. Teh, "On Smoothing and Inference for Topic Models," Proc. 25th Conf. Uncertainty in Artificial Intelligence, pp. 27-34, 2009.
[8] I. Pruteanu-Malinici, L. Ren, J. Paisley, E. Wang, and L. Carin, "Hierarchical Bayesian Modeling of Topics in Time-Stamped Documents," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 6, pp. 996-1011, June 2010.
[9] X.G. Wang, X.X. Ma, and W.E.L. Grimson, "Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 3, pp. 539-555, Mar. 2009.
[10] F.R. Kschischang, B.J. Frey, and H.-A. Loeliger, "Factor Graphs and the Sum-Product Algorithm," IEEE Trans. Information Theory, vol. 47, no. 2, pp. 498-519, Feb. 2001.
[11] C.M. Bishop, Pattern Recognition and Machine Learning. Springer, 2006.
[12] G. Heinrich, "Parameter Estimation for Text Analysis," technical report, Univ. of Leipzig, 2008.
[13] A.U. Asuncion, "Approximate Mean Field for Dirichlet-Based Models" Proc. ICML Workshop Topic Models, 2010.
[14] M. Welling, M. Rosen-Zvi, and G. Hinton, "Exponential Family Harmoniums with an Application to Information Retrieval," Proc. Neural Information Processing Systems, pp. 1481-1488, 2004.
[15] T. Hofmann, "Unsupervised Learning by Probabilistic Latent Semantic Analysis," Machine Learning, vol. 42, pp. 177-196, 2001.
[16] B.J. Frey and D. Dueck, "Clustering by Passing Messages between Data Points," Science, vol. 315, no. 5814, pp. 972-976, 2007.
[17] M.F. Tappen and W.T. Freeman, "Comparison of Graph Cuts with Belief Propagation for Stereo, Using Identical MRF Parameters," Proc. Ninth IEEE Int'l Conf. Computer Vision, pp. 900-907, 2003.
[18] M. Hoffman, D. Blei, and F. Bach, "Online Learning for Latent Dirichlet Allocation," Proc. Neural Information Processing Systems, pp. 856-864, 2010.
[19] J. Zeng and Z.-Q. Liu, "Markov Random Field-Based Statistical Character Structure Modeling for Handwritten Chinese Character Recognition," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 5, pp. 767-780, May 2008.
[20] J. Shi and J. Malik, "Normalized Cuts and Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.
[21] A.P. Dempster, N.M. Laird, and D.B. Rubin, "Maximum Likelihood from Incomplete Data via the EM Algorithm," J. Royal Statistical Soc., Series B, vol. 39, pp. 1-38, 1977.
[22] J. Zeng, L. Xie, and Z.-Q. Liu, "Type-2 Fuzzy Gaussian Mixture Models," Pattern Recognition, vol. 41, no. 12, pp. 3636-3643, 2008.
[23] J. Zeng and Z.-Q. Liu, "Type-2 Fuzzy Hidden Markov Models and Their Application to Speech Recognition," IEEE Trans. Fuzzy Systems, vol. 14, no. 3, pp. 454-467, June 2006.
[24] J. Zeng and Z.Q. Liu, "Type-2 Fuzzy Markov Random Fields and Their Application to Handwritten Chinese Character Recognition," IEEE Trans. Fuzzy Systems, vol. 16, no. 3, pp. 747-760, June 2008.
[25] J. Winn and C.M. Bishop, "Variational Message Passing," J. Machine Learning Research, vol. 6, pp. 661-694, 2005.
[26] W.L. Buntine, "Variational Extensions to EM and Multinomial PCA," Proc. 13th European Conf. Machine Learning, pp. 23-34. 2002.
[27] D.D. Lee and H.S. Seung, "Learning the Parts of Objects by Non-Negative Matrix Factorization," Nature, vol. 401, pp. 788-791, 1999.
[28] J. Zeng, W.K. Cheung, C.-H. Li, and J. Liu, "Coauthor Network Topic Models with Application to Expert Finding," Proc. IEEE/WIC/ACM Int'l Conf. Web Intelligence and Intelligent Agent Technology, pp. 366-373, 2010.
[29] J. Zeng, W.K. Cheung, C.-H. Li, and J. Liu, "Multirelational Topic Models," Proc. IEEE Ninth Int'l Conf. Data Mining, pp. 1070-1075. 2009,
[30] J. Zeng, W. Feng, W.K. Cheung, and C.-H. Li, "Higher-Order Markov Tag-Topic Models for Tagged Documents and Images," arXiv:1109.5370v1 [cs.CV], 2011.
[31] A.K. McCallum, K. Nigam, J. Rennie, and K. Seymore, "Automating the Construction of Internet Portals with Machine Learning," Information Retrieval, vol. 3, no. 2, pp. 127-163, 2000.
[32] S. Zhu, J. Zeng, and H. Mamitsuka, "Enhancing MEDLINE Document Clustering by Incorporating MeSH Semantic Similarity," Bioinformatics, vol. 25, no. 15, pp. 1944-1951, 2009.
[33] I. Porteous, D. Newman, A. Ihler, A. Asuncion, P. Smyth, and M. Welling, "Fast Collapsed Gibbs Sampling for Latent Dirichlet Allocation," Proc. 14th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, pp. 569-577, 2008.
[34] J. Eisenstein and E. Xing, "The CMU 2008 Political Blog Corpus," technical report, Carnegie Mellon Univ., 2010.
[35] J. Zeng, "A Topic Modeling Toolbox Using Belief Propagation," J. Machine Learning Research, vol. 13, pp. 2233-2236, 2012.
[36] K. Zhai, J. Boyd-Graber, and N. Asadi, "Using Variational Inference and Mapreduce to Scale Topic Modeling," arXiv:1107.3765v1 [cs.AI], 2011.
[37] J. Chang, J. Boyd-Graber, S. Gerris, C. Wang, and D. Blei, "Reading Tea Leaves: How Humans Interpret Topic Models," Proc. Neural Information Processing Systems, pp. 288-296, 2009.
[38] D. Ramage, D. Hall, R. Nallapati, and C.D. Manning, "Labeled LDA: A Supervised Topic Model for Credit Attribution in Multi-Labeled Corpora," Proc. Conf. Empirical Methods in Natural Language Processing, pp. 248-256. 2009.
[39] C.-C. Chang and C.-J. Lin, "LIBSVM: A Library for Support Vector Machines," ACM Trans. Intelligent Systems and Technology, vol. 2, pp. 27:1-27:27, 2011.
51 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool