We present a novel algorithm to detect and remove cast shadows in a video sequence by taking advantage of the statistical prevalence of the shadowed regions over the object regions. We model shadows using multivariate Gaussians. We apply a weak classifier as a pre-filter. We project shadow models into a quantized color space to update a shadow flow function. We use shadow flow, background models, and current frame to determine the shadow and object regions. This method has several advantages: It does not require a color space transformation. We pose the problem in the RGB color space, and we can carry out the same analysis in other Cartesian spaces as well. It is data-driven and adapts to the changing shadow conditions. In other words, accuracy of our method is not limited by the preset values. Furthermore, it does not assume any 3D models for the target objects or tracking of the cast shadows between frames. Our results show that the detection performance is superior than the benchmark method.