This paper presents a framework for dynamic 3D shape and motion reconstruction from multi-viewpoint images using a deformable mesh model. By deforming a mesh at a frame to that at the next frame, we can obtain both 3D shape and motion of the object simultaneously. The deformation process of our mesh model is heterogeneous. Each vertex changes its deformation process according to its 1) photometric property (i.e., if it has prominent texture or not), and 2) physical property (i.e., if it is an element of rigid part of the object or not). This heterogeneous deformation model enables us to reconstruct the object which consists of different kinds of materials or parts with different motion models, e.g., rigidly acting body parts and deforming soft clothes or its skins, by a single and unified computational framework.