In this paper we present a video surveillance system for automated border and checkpoint analysis. The described system employs automated feature extraction and tracking to ascertain vehicle size, speed, and response to an interrogating vibration for vehicle bounce signature analysis. To increase the overall robustness of the surveillance system, we introduce a novel approach to invalid feature filtering. In particular, we use a hidden Markov model trained to simultaneously recognize specific coarse motion trajectories and tracking failures. The proposed recognition and filtering scheme effectively identifies erroneously tracked features and removes them prior to any subsequent motion analysis tasks. The result is a significant increase in classification and recognition accuracy. We demonstrate the efficacy of the suggested technique on a variety of video surveillance sequences.