This paper presents a localized coarse-to-fine algorithm for efficient and accurate pedestrian localization and silhouette extraction for the Gait Challenge data sets. The coarse detection phase is simple and fast. It locates the target quickly based on temporal differences and some knowledge on the human target. Based on this coarse detection, the fine dectection phase applies a robust background subtraction algorithm to the coarse target regions and the detection obtained is further processed to produce the final results. This algorithm has been tested on 285 outdoor sequences from the Gait Challenge data sets, with wide variety of capture conditions. The pedestrian targets are localized very well and silhouettes extracted resemble the manually labeled silhouettes closely.