This paper proposes an algorithm for real-time video tracking and discusses its application to automatic vehicle watching system. Recent research posits two popular methods for video tracking: background statistical modeling and image difference. The proposed algorithm called Adaptive Hybrid Difference (AHD) algorithm is designed as an improvement of the methods. AHD algorithm uses a hybrid difference strategy to optimize the image building and runs within an adaptive threshold setting based on Gaussian distribution. The experimental results indicate that AHD algorithm performs more effective and applicable than the two popular methods in the video tracking of fast-moving rigid objects.