Head gestures such as nodding and shaking are often used as one of human body languages for communication with each other, and their recognition plays an important role in the advancement of human-computer interaction. As head gesture is the continuous motion on the sequential time series, the key problems of recognition are to track multi-view head and understand the head pose switch. This paper presents a novel approach to recognize the nod and shake in the interactive computer environment. First, head poses are detected by multi-view model(MVM) and then Hidden Markov Model(HMM) are used as head gesture statistic inference model for gesture recognition. Experimental results show that the approach is effective and real time.