This paper presents a new conceptual model for representing visual information about moving objects in video data. Based on available automatic scene segmentation and object tracking algorithms, the proposed model calculates object motions at various levels of semantic granularity. It represents trajectory, color and dimensions of a single moving object and the directional and topological relations among multiple objects over a time interval. To facilitate query processing, there are two optimal approximate matching algorithms designed to match time-series visual features of moving objects. Experimental results indicate that the proposed algorithms outperform the conventional subsequence-matching methods substantially in the similarity between the two trajectories.