In this paper, we present a unified framework for modeling intrinsic properties of face images for recognition. It is based on the quotient image (QI) concept, in particular on the existing works of QI [1, 2], Spherical Harmonic[13, 14, 15], [16, 17], Image Ratio [3, 5, 6, 7]and Retinex [4, 9]. Under this framework, we generalize these previous works into two new algorithms: (1) Non-Point Light Quotient Image (NPL-QI) extends QI to deal with non-point light sources by modeling non-point light directions using spherical harmonic bases; (2) Self-Quotient Image (S-QI) extends QI to perform illumination subtraction without the need for alignment and no shadow assumption. Experimental results show that our algorithms can significantly improve the performance of face recognition under varying illumination conditions.