This paper describes a modular approach to gesture recognition. The complex task of gesture recognition from image sequences was decomposed by first identifying the hand pose in individual frames. The pose information was then incorporated with hand motion to recognize gestures. Independent recognition modules were devised for different sub-tasks. A radial basis function (RBF) neural network was trained to recognize static hand poses. Inputs to the RBF network were feature vectors extracted from segmented 2D binary images of the hand. The pose recognition results of using Zernike moments and Fourier descriptors as the feature vectors were compared and it were found that Fourier descriptors were superior in terms of computational speed. Combined outputs from a set of recurrent neural networks (RNN) and hidden Markov model (HMM) chains were used to recognize gestures from the temporal sequence of pose classifier outputs. The combined classifier achieved a recognition rate of 86.8%. In addition, we illustrate that the inclusion of an intermediate pose classification stage is advantageous for recognition and training speed.