While unmanned aerial vehicles are gaining wider use in intelligence, surveillance, and reconnaissance applications, the utility of airborne video has been hampered due to inaccurate telemetry data. Too much manual control is required for exploitation, and it is difficult to relate video to other data due to its small field of view and limited context. In this paper, we present our approach for autonomous, near real time registration of video imagery to a reference image with high geodetic accuracy. We describe a rigorous sensor model that fully captures the dynamics of an airborne video camera, the autonomous generation of correspondence points between the video and the reference image, and the adjustment of the sensor model parameters using a Kalman filter. We also outline our approach to obtaining improved imaging parameters for video frames that were not actually registered by the system. Finally, we present experimental results that compare the Kalman filtered registration accuracy with that obtained using triangulation of the individual frames within a video clip. Empirical results from the prototype system show geolocation improvements of an order of magnitude and indicate that our technique works well in practice.