Moving object detection systems generally detect shadows cast by the moving object as part of the moving object. In this paper the problem of separating moving cast shadows from the moving objects in outdoor environment is addressed. Unlike other previous work, we provide a method that does not use any geometrical information. Our physics-based approach is based on a spatio-temporal albedo normalization test and a dichromatic reflection model. The physics based model is used both in the estimation and verification phases. We provide results for several different video sequences representing a variety of materials and shadows. We achieve excellent results in distinguishing moving objects from their shadows. The results indicate that our approach is robust to a variety of background and foreground materials and varying illumination conditions.