Global Motion Estimation (GME) is an important part in the object-based applications. In this paper, a fast progressive model refinement (FPMR) GME algorithm is proposed. It can select the appropriate motion model according to the complexity of the camera motion. Two techniques are used to accelerate the procedure of FPMR. The first is an outlier prediction based feature point selection method. It can predict outliers from that of the last frame and therefore can effectively remove the influence of outliers on parameter calculation. The second is an intermediate-level model prediction method, which is used to fast the model selection and the parameter calculation procedure. Experiments show that the proposed algorithm is above two times faster than that of the Feature-based Fast and Robust GME technique.