We create a model of joint color changes in face images due to lighting variations. This is done by observing how colors of an individuals face with fixed pose and expression are mapped to new colors under different lighting conditions. One of the challenges we are dealing with in this work is that the scenes are not constant for different lighting. Hence we cannot observe the joint color changes of the scenes. However all the scenes have a human subject with approximately frontal pose, so we use the color changes observed on a human subjects face to learn the color mapping. The joint color mappings are represented in a low dimensional subspace obtained using Singular Value Decomposition (SVD). Using these maps the detected face from a new image can be transformed to appear as if taken under canonical lighting condition.