Representation of images using the distribution of local features on a visual codebook is an effective method for object categorization. Typically, discriminative capability of the codebook can lead to a better performance. However, conventional methods usually use clustering algorithms to learn codebooks without considering this. This paper presents a novel approach of learning optimal compact codebooks by selecting a subset of discriminative codes from a large codebook. Firstly, the Gaussian models of object categories based on a single code are learned from the distribution of local features within each image. Then two discriminative criteria, i.e. likelihood ratio and Fisher, are introduced to evaluate how each code contributes to the categorization. We evaluate the optimal codebooks constructed by these two criteria on Caltech-4 dataset, and report superior performance of object categorization compared with traditional K-means method with the same size of codebook.