Hand gesture-based interface is one of the most promising modes of natural and fluid human-computer interaction that substitutes for the mouse and keyboard in wearable computing systems. In wearable computing scenarios, hand positioning and tracking is particularly difficult due to complex background, lighting variation and image dithering caused by head movement. This paper proposes a robust hand tracking method for gesture-based interaction of a wearable computer with a visual helmet. The method is an extension of the basic CONDENSATION algorithm, which is able to track hand in dynamic and complex background. Furthermore, the algorithm can recognize the current hand gesture and automatically switch between multiple well-defined gesture templates in the tracking loop. The experimental results show that the proposed algorithm worked well in dynamic and complex background in real time.