This paper presents a modular real-time feature-based visual tracking architecture where each feature of an object is tracked by one module. A data fusion stage collects the information from various modules exploiting the relationship among features to achieve robust detection and visual tracking. This architecture takes advantage of the temporal and spatial information available in a video stream. Its effectiveness is demonstrated in a face tracking system that uses eyes and lips as features. In the architecture implementation, each module has a pre-processing stage that reduces the number of image regions that are candidates for eyes and lips. Support Vector Machines are then used in the classification process, whereas a combination of Kalman filters and template matching is used for tracking. The geometric relation between features is used in the data fusion stage to combine the information from different modules to improve tracking.