In this paper, we present an approach for video surveillance involving (a) moving object detection, (b) tracking and (c) normal/abnormal event recognition. The detection step uses an adaptive background subtraction technique with a shadow elimination model based on the color constancy principle. The target tracking involves a direct and inverse matrix matching process. The novelty of the paper lies mainly in the recognition stage, where we consider local motion properties (flow vector), and more global ones expressed by elliptic Fourier descriptors. From these temporal trajectory characterizations, two Kohonen maps allow to distinguish normal behavior from abnormal or suspicious ones. The classification results show a 94.6 % correct recognition rate with video sequences taken by a low cost webcam. Finally, this algorithm can be fully implemented in real-time.