Motions of articulated animated agents are most easily described in the agent's motor (e.g. joint) space. Most current techniques require explicit kinematic modeling to transform between image (animation) space and motor space. We present a method where the visual-motor calibration is estimated on-line from observing the actions of a real physical agent. We show how to use the estimated models to drive animation of both (affine) geometric structure, and subspace based image intensity corrections. The estimation and animation methods are applicable where the synthesized animation is a relatively small modification of motions in the training set. To illustrate the methods a simple example online animation editing system has been implemented. Experiments animating both human and robot arms and hands are shown.