Color-based tracking has proved efficient and robust recently. Trackers build the object appearance model with histogram statistics, search and evaluate hypothesis in a probabilistic framework. This method relies much on the discrimination between object and scene blobs. Color clutter in the scene, although not so many in quantity, may distract these trackers. We build explicitly object shape model and insert the head detector into the observation model to resist these clutters in the scene for improved tracker. The detector scans the image and output probability value as the possibility of current window being a candidate human head. Experiments demonstrate the method can work more accurately and robustly.