This paper proposes a method of clustering video frame pixels for a moving object extraction system. Two cascaded classifiers work cooperatively to firstly classify the pixels into background and non-background cluster and then classify the non-background cluster into four clusters. Besides the moving cluster and shadow cluster, two additional clusters, corresponding to the noisy highlighting pixels and the pixels affected by the camera auto iris function in real environment, are observed and modeled. Experiments on our people counting prototype system demonstrate that it can run smoothly with better performance of moving object extraction in long-term video surveillance of complex scenes.