Conventional treatment of visual tracking has been to optimize an objective function in a probabilistic framework. In this formulation, efficient algorithms employing simple prior distributions are usually insufficient to handle clutters (e.g., Kalman filter). On the other hand, distributions that are complex enough to incorporate all a priori knowledge can make the problem computationally intractable (e.g., Particle filter (PF)). This paper proposes a new formulation of visual tracking where every piece of information, be it from a priori knowledge or observed data, is represented by a set in the solution space and the intersection of these sets, the feasibility set, represents all acceptable solutions. Based on this formulation, we propose an algorithm whose objective is to find a solution in the feasibility set. We show that this set theoretic tracking algorithm performs effective face tracking and is computationally more efficient than standard PF-based tracking.