This paper presents a novel approach to estimate the changing internal and external parameters of the camera in real time using a few 3D-2D point correspondences. We approach the problem by constructing two filters which can individually track motion and zoom respectively in a robust manner. Then, the Viterbi algorithm is used to select a filter depending on whether the camera is undergoing motion or zoom. We assume that the camera cannot move and zoom simultaneously, though our method can tolerate some hand held movement during zoom. The application of our approach is demonstrated on synthetic and real data.