Issue No. 09 - Sept. (2014 vol. 36)
Aaron Courville , Department of Computer Science and Operations Research, University of Montreal, Montreal, Canada
Guillaume Desjardins , DIRO, University of Montreal, 2920 chemin de la Tour, Montreal, Canada
James Bergstra , Centre for Theoretical Neuroscience , University of Waterloo, Waterloo, Canada
Yoshua Bengio , DIRO, University of Montreal, 2920 chemin de la Tour, Montreal, Canada
The spike-and-slab restricted Boltzmann machine (ssRBM) is defined to have both a real-valued “slab” variable and a binary “spike” variable associated with each unit in the hidden layer. The model uses its slab variables to model the conditional covariance of the observation—thought to be important in capturing the statistical properties of natural images. In this paper, we present the canonical ssRBM framework together with some extensions. These extensions highlight the flexibility of the spike-and-slab RBM as a platform for exploring more sophisticated probabilistic models of high dimensional data in general and natural image data in particular. Here, we introduce the subspace-ssRBM focused on the task of learning invariant features. We highlight the behaviour of the ssRBM and its extensions through experiments with the MNIST digit recognition task and the CIFAR-10 object classification task.
Slabs, Data models, Vectors, Feature extraction, Covariance matrices, Training, Standards
A. Courville, G. Desjardins, J. Bergstra and Y. Bengio, "The Spike-and-Slab RBM and Extensions to Discrete and Sparse Data Distributions," in IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 36, no. 9, pp. 1874-1887, 2014.