This paper presents a physical model-based method for recovering and tracking nonrigid motion of elastic objects. The proposed method recovers the motion in terms of actual physical parameters (Young?s modulus) that characterize the dynamics of the objects. The tracking scheme synthesizes the motion of the points inside the object from the boundary observations, constrained by the physical parameters. Experiments on three image sequences show that using the recovered physical parameters as constraints can greatly improve the tracking quality.