In this paper, we present a clustering-based tracking algorithm for non-rigid object. Non-rigid object tracking is a challenging task because the target often appears as a concave shape or an object with apertures. In such cases, many background areas will be mixed into the tracking target, which are difficult to be removed by modifying the shape of the search area. Our algorithm realizes robust tracking for such objects by classifying the pixels in the search area into "target" and "background" with K-means clustering algorithm that uses both the "positive" and "negative" samples. The contributions of this research are: 1) Using a 5D feature vector to describe both the geometric feature "(x, y)' and color feature "(Y,U, V )" of an object (or a pixel) uniformly. This description enables the simultaneous adaptation of both the geometric and color variance during tracking; 2) Using a variable ellipse model (a) to describe the search area; (b) to model the surrounding background. This guarantees the stable tracking of objects with various geometric transformations. Through extensive experiments in various environments and conditions, the effectiveness and the efficiency of the proposed algorithm is confirmed.