This paper describes a visual markerless real-time tracking system for Augmented Reality applications. The system uses a firewire camera with a fisheye lens mounted at 10 fps. Visual tracking of 3D scene points is performed simultaneously with 3D camera pose estimation without any prior scene knowledge. All visual-geometric data is acquired using a structure-from-motion approach. The lens selection was driven by research results that show the superiority of a fisheye lens to a standard perspective lens for this approach. 2D features in the hemispherical image are tracked using a 2D point tracker. Based on the feature tracks, 3D camera ego-motion and 3D features are estimated.