To facilitate accurate and efficient detection of motion patterns in video data, it is desirable to abstract from pixel intensity values to representations that explicitly and compactly capture movement across space and time. For example, in the monitoring of surveillance video, it is useful to capture movement, as potential targets of interest traverse the scene in specific ways. Toward such ends, the "direction map" is introduced: a novel representation that captures the spatiotemporal distribution of direction of motion across regions of interest in space and time. Methods are presented for recovering direction maps from video, constructing direction map templates to define target patterns of interest and comparing predefined templates to newly acquired video for pattern detection and localization. The approach has been implemented with real-time considerations and tested on over 6300 frames across seven surveillance videos. Results show an overall recognition rate of approximately 90% hits vs. 7% false positives.