In this paper, we present a real time system for detecting people and interpreting some of their postures. The system runs reliably on different people and under different lighting conditions. It uses an inexpensive PC desktop camera to acquire color monocular video sequences. Two fundamental problems of such a CCD camera for background-foreground segmentation are analyzed and a solution to them has been proposed. The system locates humans by extracting their pro .le using a physics based approach, which is robust in presence of noise. Next, it uses a pattern recognition algorithm to detect and interpret the posture of a human. The system can be used for Peripheral User Interface (PUI), in pervasive computing applications, surveillance, monitoring people activities in an indoor environment, Visual Computer games, Smart screen savers etc. We have already used this system for several real-world interactive video game applications.