We present a spectral representation for appearance-based image classification and object recognition. Based on a generative process, the representation is derived by partitioning the frequency domain into small disjoint regions. This gives rise to a set of filters and a representation consisting of marginal distributions of those filter responses. We use a neural network to learn a classifier through training examples. We propose a filter selection algorithm by maximizing the performance over training data. A distinct advantage of our representation is that it can be effectively used for different classification and recognition tasks, which is demonstrated by experiments and comparisons in texture classification, face recognition, and appearance-based 3D object recognition.