This paper presents a coarse-to-fine approach to eliminate unexpected shadows of multiple pedestrians from a static and textured background using Gaussian shadow modeling. At the coarse stage, a moment-based method is proposed to estimate the rough boundaries between shadows and moving objects. Then, at the fine stage, the rough approximation of shadow region provides a key to model shadows. The chosen shadow model is parameterized with several features including the orientation, mean, and center position of a shadow region. With these features, the chosen model can precisely eliminate the unexpected shadows from the scene background and thus improve the quality of further content analysis. Experiments demonstrate approximately 95% ratio of pedestrian-related shadows can be successfully eliminated from the scene background.