This paper discusses an approach towards automatic recognition of emotion in speech which is adopted into a system named Voice Driven Emotion Recognizer Mobile Phone (VDERM). First, a design for the emotion recognizer is proposed. LPC analysis algorithm has been used for the speech emotion feature extraction. A total of 18 speech features have been selected to represent each emotion. A database consisting of emotional Malay and English, male and female voice samples have been developed for training and recognition purposes. Two recognition methods namely neural network and fuzzy model have been experimented and compared. The results show that both methods have their own advantage and disadvantage in application to emotion recognition. A recognition rate of up 60% is achievable by using these computer methods which is sufficient based on the recognition rate achieved by human.