We explore a coherent framework for the simultaneous tracking and recognition of moving objects in highly cluttered environments. The procedure has three basic components: (i) A deformable template representation of the objects in a database; (ii) Dynamical equations of motion derived from Lagrangian mechanics; and (iii) an observation (or data) model designed using non-parametric image processing techniques. The combination of these components leads to a nonlinear filtering problem, which is equivalent to a Hidden Markov Model (HMM). An iterative algorithm - to be referred to as the Monte Carlo Filter - introduced in the statistics literature, and first employed in computer vision problems by Blake and Isard, solves the filtering problem. The design of the above three components is critical for real time tracking and recognition. The procedure has been successfully implemented in the tracking of fish moving in an aquarium (an environment highly degraded by clutter, occlusion, and other artifacts), and in the tracking of billiards on a pool table.