Shading distortions are often perceived in geometrically distorted document images due to the change of surface normal with respect to the illumination direction. Such distortions are undesirable because they hamper OCR performance tremendously even when the geometric distortions are corrected. In this paper, we propose an effective method that removes shading distortions in images of documents with various geometric shapes based on the notion of intrinsic images. We first try to derive the shading image using an inpainting technique with an automatic mask generation routine and then apply a surface fitting procedure with radial basis functions to remove pepper noises in the inpainted image and return a smooth shading image. Once the shading image is extracted, the reflectance image can be obtained automatically. Experiments on a wide range of distorted document images demonstrate a robust performance. Moreover, we also show its potential applications to the restoration of historical handwritten documents.