This paper proposes a computationally economical method of recovering the projective motion of head mounted cameras of EyeTap devices, for use in wearable computer mediated reality. The tracking system combines featureless vision and inertial tracking in a closed loop system to achieve accurate robust head tracking using inexpensive uncalibrated sensors. The combination of inertial and vision techniques provides the high accuracy visual registration needed for fitting computer graphics onto real images and robustness to large interframe camera motion due to fast head rotations. Operating on a 1.2 Ghz Pentium III wearable computer, the system is able to register live video images with less than 2 pixels of error (0.3 degrees) at 12 frames per second.