Face representations based on Gabor features have achieved great success in face recognition, such as Elastic Graph Matching, Gabor Fisher Classifier (GFC), and AdaBoosted Gabor Fisher Classifier (AGFC). In GFC and AGFC, either down-sampled or selected Gabor features are analyzed in holistic mode by a single classifier. In this paper, we propose a novel patch-based GFC (PGFC) method, in which Gabor features are spatially partitioned into a number of patches, and on each patch one GFC is constructed as component classifier to form the final ensemble classifier using sum rule. The positions and sizes of the patches are learned from a training data using AdaBoost. Experiments on two large-scale face databases (FERET and CAS-PEAL-R1) show that the proposed PGFC with only tens of patches outperforms the GFC and AGFC impressively.