A common approach to model-based tracking is to use a model of the object to predict what will be observed, and then to compare that with real observations. For methods that make use of the object?s photometric properties (appearance) in their measurements, illumination inconsistencies between the modeled and actual scene can cause tracking problems. In this paper we address one case: model-based tracking of Lambertian objects under directional light sources. We present an iterative optimization method that uses a Kalman filter to simultaneously refine estimates of the object motion, the illumination, and the model texture. We model the illumination variance between the real and predicted observation using the intensity ratios of corresponding surface points, which we then use to make model-based image predictions consistent with the real lighting. To demonstrate the effectiveness of our method we present experimental results using both synthetic (controlled) and real image sequences.