Enlarging or reducing the template size by adding new parts or removing parts of the template according to their suitability for tracking requires the ability to deal with the variation of the template size. For instance, real-time template tracking using linear predictors, although fast and reliable, requires using templates of a fixed size and does not allow online modification of the predictor. To solve this problem, we propose the Adaptive Linear Predictors (ALPs), which enable fast online modifications of prelearned linear predictors. Instead of applying a full matrix inversion for every modification of the template shape, as standard approaches to learning linear predictors do, we just perform a fast update of this inverse. This allows us to learn the ALPs in a much shorter time than standard learning approaches while performing equally well. Additionally, we propose a multilayer approach to detect occlusions and use ALPs to effectively handle them. This allows us to track large templates and modify them according to the present occlusions. We performed an exhaustive evaluation of our approach and compared it to standard linear predictors and other state-of-the-art approaches.