An approach to recognise and segment 9 elementary gestures from a video input is proposed and it can be applied to continuous sign recognition. An isolated gesture is recognised by first converting a portion of video into a motion gradient orientation image and then classifying it into one of the 9 gestures by a sparse Bayesian classifier. The portion of video used is decided by using a sampling technique based on CONDENSATION framework. By doing so, gestures can be segmented from the video in a probabilistic manner. Experiments show that the proposed method can achieve accuracy around 90% in both isolated and continuous gesture recognition without using special equipment such as glove devices and the system can run in real-time.