We describe an appearance-based human body model for tracking multiple people when they are interaction with others causing significant occlusion amongst them, or when they reenter the scene. The proposed model allows real time surveillance systems understand “who is who” after multiple people interactions, partial or total occlusions, or when a person “reappear” in the scene. It combines the grayscale textural appearance and expected shape information together in a 2D dynamic template. Experimental results demonstrate robustness and real time performance of the proposed model.