We present a distributed vision-based surveillance system. The system acquires and processes grey level images through one or multiple Camera Units monitoring certain area(s) via a Local Area Network (LAN) and is capable of combining information from multiple Camera Units to obtain a consensus decision. It can be trained to detect certain type of intrusions, for example pedestrians, a group of pedestrians, vehicles, pets etc., and minimizes false alerts due to other non-interested intrusions. As a case study, we aim to detect Pedestrian/Vehicle in an observation area. Our vision-based intrusion detection approach consists of two main steps: background subtraction based Hypothesis Generation (HG) and appearance-based Hypothesis Verification (HV). HG hypothesizes possible threats (intrusions), and HV verifies those hypotheses using Gabor filter for feature extraction and Support Vector Machines (SVMs) for classification. The system has been tested under unconstrained outdoor environment, illustrating good performance.