An adaptive, invariant to user performance fluctuation or noisy input signal, gesture recognition scheme is presented based on Self Organizing Maps, Markov Models and Levenshtein sequence distance. Multiple modalities, all based on the hand position during gesturing, train different classifiers which are then fused in a weak classifier boosting-like setup by weight assignment to each stream. The adaptability of the proposed approach consists of the incorporation of Self Organizing Maps during training, the exploitation of neighboring relations between states of the Markov models and the modified Levenshtein distance algorithm. The main focus of current work is to tackle intra and inter user variability during gesture performance by adding flexibility to the decoding procedure and allowing the algorithm to perform an optimal trajectory search while the processing speed of both the feature extraction and the recognition process indicate that the proposed architecture is appropriate for real time and large scale lexicon applications.