Gabor feature has been widely recognized as one of the best representations for face recognition. However, traditionally, it has to be reduced in dimension due to curse of dimensionality. In this paper, an Ensemble based Gabor Fisher Classifier (EGFC) method is proposed, which is an ensemble classifier combining multiple Fisher Discriminant Analysis (FDA)-based component classifiers learnt using different segments of the entire Gabor feature. Since every dimension of the entire Gabor feature is exploited by one component FDA classifier, we argue that EGFC makes better use of the discriminability implied in all the Gabor features by avoiding the dimension reduction procedure. In addition, by carefully controlling the dimension of each feature segment, Small Sample Size (3S) problem commonly confronting FDA is artfully avoided. Experimental results on FERET show that the proposed EGFC significantly outperforms the known best results so far. Furthermore, to speed up, hierarchical EGFC (HEGFC) is proposed based on pyramid-based Gabor representation. Our experiments show that, by using the hierarchical method, the time cost of the HEGFC can be dramatically reduced without much accuracy lost.