This paper proposes a template update criterion used for feature tracking of image sequences using template matching. Template update is determined by evaluating the error which will occur in the next image using the criterion. This method requires no parameter to be ad-justed and the intervals of template update become variable and adaptive to deformation and noise level. The criterion is defined based on the covariance matrix of a template matching error. Some experimental results using synthesized and natural image sequences are presented to show the validity of the criterion.