This paper presents a two level automatic feature points management method for constructing a seamless entire panorama from video sequence. In the first level, through fusing the number of tracked feature points and the estimated ratio of lost information of the mosaicing image, a feature point quantity management module is developed to select the key frames. In the second level, a feature points quality management technique is used to choose the key points for mosaicing. This module includes a coarse-to-fine method with two steps: (1) Feature points quality based key point subset creation, and (2) Multi-resolution based key point selection. The main contribution of the algorithm is that it is able to achieve robust and fast mosaicing result while maintain the most valuable information of the scene. Experiments are performed using video sequences under different conditions. The results show that the proposed algorithm could achieve robust and efficient video mosaic image.