In this paper, we present a clustering-based tracking algorithm for tracking people (e.g. hand, head, eyeball, body). A human body often appears as a concave object or an object with apertures. In this case, many background areas are mixed into the tracking target which are difficult to be removed by modifying the shape of the search area during tracking. This algorithm realizes the robust tracking for such objects by classifying the pixels in the search area into "target" and "non-target" with K-means clustering algorithm that uses both the "positive"and "negative" samples. The contributions of this research are: 1) Using a 5 - D feature vector to describe both the geometric feature "(x,y)" and color feature "(Y,U,V)" of an object (or a pixel) uniformly. This description ensures our method to follow both the position and color changes simultaneously during tracking; 2) Using a variable ellipse model (a) to describe the shape of a non-rigid object (e.g. hand) approximately, (b) to restrict the search area, and (c) to model the surrounding non-target background. This guarantees the stable tracking of objects with various geometric transformations. Through extensive experiments in various environments and conditions, the effectiveness and the efficiency of the proposed algorithm is confirmed.