This paper proposes a two-step methodology for improving the discriminatory power of linear discriminant analysis (LDA) for video-based human face recognition. Results indicate that, under real-world video capture conditions, face images extracted from a video sequence have enough 3D rotations, illumination changes and background variations to reduce the discriminatory power of an LDA classifier. The proposed method involves deriving an LDA subspace from carefully selected subsets of face images that fall within a narrow range of pose angles, and then growing the classification regions in the LDA subspace using face images with a wider range of pose angle changes, illumination changes, and background variations. Polynomial support vector machines (SVM) are shown to provide better recognition rates by defining the boundaries between clusters that represent the faces of different subjects. Results show that there is an improvement in the recognition rate when the LDA subspace is derived with this methodology than when it is derived with a set of face images with a widely divergent set pose angles, illumination variations, and backgrounds.