As a part of natural language understanding, sign language recognition is considered an important area of research. The applications of such a system range from human-computer interaction in virtual reality systems to auxiliary tools for deaf-mute to communicate with ordinary people through computer. A great deal of research is done so far but fewer researchers have extended it to Arabic Sign Language Recognition. In this paper we have presented a system that performs Vision Based Isolated Arabic Sign Language Recognition using Hidden Markov Models together with EM algorithm for parameters estimation. An approach to track hands in subsequent frames is proposed using a fuzzy object similarity measure based on a number of geometrical features of hands. Moreover, we have used the centroid of the signer's face to centralize the body coordinates instead of fixing the signer's position or using position tracker device. The overall accuracy of the recognition task is 98% over a dataset of 50 signs including single hand and two-handed signs.