The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.08 - Aug. (2013 vol.35)
pp: 1887-1901
Bo Chen , Duke University, Durham
Gungor Polatkan , Princeton University, Princeton
Guillermo Sapiro , Duke University, Durham
David Blei , Princeton University, Princeton
David Dunson , Duke University, Durham
Lawrence Carin , Duke University, Durham
ABSTRACT
Unsupervised multilayered (“deep”) models are considered for imagery. The model is represented using a hierarchical convolutional factor-analysis construction, with sparse factor loadings and scores. The computation of layer-dependent model parameters is implemented within a Bayesian setting, employing a Gibbs sampler and variational Bayesian (VB) analysis that explicitly exploit the convolutional nature of the expansion. To address large-scale and streaming data, an online version of VB is also developed. The number of dictionary elements at each layer is inferred from the data, based on a beta-Bernoulli implementation of the Indian buffet process. Example results are presented for several image-processing applications, with comparisons to related models in the literature.
INDEX TERMS
Dictionaries, Convolution, Computational modeling, Mathematical model, Analytical models, Load modeling, Bayesian methods, factor analysis, Bayesian, deep learning, convolutional, dictionary learning
CITATION
Bo Chen, Gungor Polatkan, Guillermo Sapiro, David Blei, David Dunson, Lawrence Carin, "Deep Learning with Hierarchical Convolutional Factor Analysis", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.35, no. 8, pp. 1887-1901, Aug. 2013, doi:10.1109/TPAMI.2013.19
REFERENCES
[1] M. Zeiler, D. Krishnan, G. Taylor, and R. Fergus, "Deconvolution Networks," Proc. IEEE Conf. Computer Vision Pattern Recognition, 2010.
[2] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-Based Learning Applied to Document Recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
[3] G. Hinton and R. Salakhutdinov, "Reducing the Dimensionality of Data with Neural Networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
[4] K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, "What Is the Best Multi-Stage Architecture for Object Recognition?" Proc. IEEE Int'l Conf. Computer Vision, 2009.
[5] M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun, "Efficient Learning of Sparse Representations with an Energy-Based Model," Proc. Neural Information Processing Systems, 2006.
[6] P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol, "Extracting and Composing Robust Features with Denoising Autoencoders," Proc. Int'l Conf. Machine Learning, 2008.
[7] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng, "Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations," Proc. Int'l Conf. Machine Learning, 2009.
[8] H. Lee, Y. Largman, P. Pham, and A. Ng, "Unsupervised Feature Learning for Audio Classification Using Convolutional Deep Belief Networks," Proc. Neural Information Processing Systems, 2009.
[9] M.R.M. Norouzi and G. Mori, "Stacks of Convolutional Restricted Boltzmann Machines for Shift-Invariant Feature Learning," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[10] H. Lee, C. Ekanadham, and A.Y. Ng, "Sparse Deep Belief Network Model for Visual Area V2," Proc. Neural Information Processing Systems, 2008.
[11] Y. Boureau, F. Bach, Y. LeCun, and J. Ponce, "Learning Mid-Level Features for Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[12] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, "Locality-Constrained Linear Coding for Image Classification," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[13] Y.-L. Boureau, J. Ponce, and Y. LeCun, "A Theoretical Analysis of Feature Pooling in Vision Algorithms," Proc. Int'l Conf. Machine Learning, 2010.
[14] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online Dictionary Learning for Sparse Coding," Proc. Int'l Conf. Machine Learning, 2009.
[15] T.L. Griffiths and Z. Ghahramani, "Infinite Latent Feature Models and the Indian Buffet Process," Proc. Advances in Neural Information Processing Systems, pp. 475-482, 2005.
[16] R. Thibaux and M. Jordan, "Hierarchical Beta Processes and the Indian Buffet Process," Proc. Int'l Conf. Artificial Intelligence and Statistics, 2007.
[17] J. Paisley and L. Carin, "Nonparametric Factor Analysis with Beta Process Priors," Proc. Int'l Conf. Machine Learning, 2009.
[18] M. Zhou, H. Chen, J. Paisley, L. Ren, G. Sapiro, and L. Carin, "Non-Parametric Bayesian Dictionary Learning for Sparse Image Representations," Proc. Neural Information Processing Systems, 2009.
[19] R. Adams, H. Wallach, and Z. Ghahramani, "Learning the Structure of Deep Sparse Graphical Models," Proc. Int'l Conf. Artificial Intelligence and Statistics, 2010.
[20] B. Chen, G. Polatkan, G. Sapiro, D. Dunson, and L. Carin, "The Hierarchical Beta Process for Convolutional Factor Analysis and Deep Learning," Proc. Int'l Conf. Machine Learning, 2011.
[21] M. West, "Bayesian Factor Regression Models in the 'Large p, Small n' Paradigm," Bayesian Statistics 7, J.M. Bernardo, M. Bayarri, J. Berger, A. Dawid, D. Heckerman, A. Smith, and M. West, eds., pp. 723-732, Oxford Univ. Press, 2003.
[22] C. Carvalho, J. Chang, J. Lucas, J.R. Nevins, Q. Wang, and M. West, "High-Dimensional Sparse Factor Modelling: Applications in Gene Expression Genomics," J. Am. Statistical Assoc., vol. 103, pp. 1438-1456, 2008.
[23] M. Tipping, "Sparse Bayesian Learning and the Relevance Vector Machine," J. Machine Learning Research, vol. 1, pp. 211-244, 2001.
[24] R. Adams, H. Wallach, and Z. Ghahramani, "Learning the Structure of Deep Sparse Graphical Models," Proc. Int'l Conf. Artificial Intelligence and Statistics, 2010.
[25] M. Hoffman, D. Blei, and F. Bach, "Online Learning for Latent Dirichlet Allocation," Proc. Advances in Neural Information Processing Systems, 2010.
[26] N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines. Cambridge Univ. Press, 2000.
[27] R.C.J. Weston and F. Ratle, "Deep Learning via Semi-Supervised Embedding," Proc. Int'l Conf. Machine Learning, 2008.
[28] Y.-L. Boureau, M. Ranzato, and F.-J. Huang, "Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[29] P.O. Hoyer, "Non-Negative Matrix Factorization with Sparseness Constraints," J. Machine Learning Research, vol. 5, pp. 1457-1469, 2004.
[30] D.D. Lee and H.S. Seung, "Learning the Parts of Objects by Non-Negative Matrix Factorization," Nature, vol. 401, no. 6755 pp. 788-791, 1999.
[31] S. Lazebnik, C. Schmid, and J. Ponce, "Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006.
[32] H. Zhang, A.C. Berg, M. Maire, and J. Malik, "SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006.
[33] K. Yu, Y. Lin, and J. Lafferty, "Learning Image Representations from the Pixel Level via Hierarchical Sparse Coding," Proc. IEEE Conf, Computer Vision and Pattern Recognition, Dec. 2011.
[34] L. Bo, X. Ren, and D. Fox, "Hierarchical Matching Pursuit for Image Classification: Architecture and Fast Algorithms," Proc. Advances in Neural Information Processing Systems, Dec. 2011.
[35] M.D. Zeiler, G.W. Taylor, and R. Fergus, "Adaptive Deconvolutional Networks for Mid and High Level Feature Learning," Proc. IEEE Int'l Conf. Computer Vision, 2011.
[36] Y. Boureau, F. Bach, Y. LeCun, and J. Ponce, "Learning Mid-Level Features for Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[37] K. Kavukcuoglu, P. Sermanet, Y. Boureau, K. Gregor, M. Mathieu, and Y. LeCun, "Learning Convolutional Feature Hierachies for Visual Recognition," Proc. Advances in Neural Information Processing Systems, 2010.
[38] C.R. Johnson, E. Hendriks, I. Berezhnoy, E. Brevdo, S.M. Hughes, I. Daubechies, J. Li, E. Postma, and J.Z. Wang, "Image Processing for Artist Identification: Computerized Analysis of Vincent van Gogh's Brushstrokes," IEEE Signal Processing Magazine, vol. 25, no. 4, pp. 37-48, July 2008.
[39] D. Marr, Vision. Freeman, 1982.
[40] D.L. Donoho, "Nature vs. Math: Interpreting Independent Component Analysis in Light of Recent work in Harmonic Analysis," Proc. Int'l Workshop Independent Component Analysis and Blind Signal Separation, pp. 459-470, 2000.
[41] D. Knowles and Z. Ghahramani, "Infinite Sparse Factor Analysis and Infinite Independent Components Analysis," Proc. Seventh Int'l Conf. Independent Component Analysis and Signal Separation, 2007.
21 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool