For object recognition under varying illumination conditions, we propose a method based on photometric alignment. The photometric alignment is known as a technique that models both diffuse reflection components and attached shadows under a distant point light source by using three basis images. However, in order to reliably reproduce these components in a test image, we have to take into account outliers such as specular reflection components and shadows in the test image. Accordingly, our proposed method utilizes RANdom SAmple Consensus (RANSAC), which has been used successfully for estimating basis images. In the present study, we have conducted experiments using the Yale Face Database B and confirmed that a combination of the photometric alignment and RANSAC provides a simple but effective method for object recognition under varying illumination conditions.