In this paper, we propose a semantic routing and filtering framework for large-scale monitoring of video streams. Our goal is to build a distributed system that at any given time is capable of simultaneously monitoring the content of multiple video streams being transmitted over the Internet (or a proprietary network). A key design requirement of such a system is the ability to handle tens of gigabytes of multimedia data per second. Traditional techniques have an important limitation. Once a bottleneck in terms of CPU or storage is reached, data is dropped indiscriminately.. In this paper, we propose distributed real-time semantic filters to route and filter video data. We propose a mechanism to alter the accuracy of classification with the complexity of execution; thus avoiding system failure during periods of overload. We propose a set of novel video features that perform better than our previous semantic classifiers. This system is capable of classifying over a hundred concepts. Experiments on 190 hours of pre-stored and live video streams validate the effectiveness of the proposed system.