Visual tracking is a key component in many computer vision applications. Linear subspace techniques (e.g. eigen-tracking) are one of the most popular approaches to align templates with appearance variations (e.g. illumination, iconic changes). A number of well known tracking algorithms have been proposed in the last years to accurately fit these models to images. Computational efficiency is an important limitation in object tracking algorithms and different efficient techniques, such as the ldquoprojected-outrdquo optimization, have been proposed. They reduce the computational cost using an efficient formulation in which many of the involved operations can be precomputed. On the other hand, alternative ldquosimultaneousrdquo algorithms jointly optimize pose and appearance parameters, providing better performance but increasing the computational cost. In this paper, we propose an algorithm for efficient linear appearance model fitting based on the inverse compositional simultaneous optimization of pose and appearance. We introduce a novel formulation which reduces the required computational time while maintaining similar convergence properties of previous ldquosimultaneousrdquo approaches. Experimental results illustrate the capabilities of this algorithm in face tracking.