We propose a method of expression learning by imitating the process of a baby?s learning process. A baby cannot know what an expression means but he/she can be affected by the action that people do to him or her, and then he/she remembers this facial expression. In our system, a robot starts learning facial expression by recognizing human actions. Our system detects the face regions and extracts the facial features using four direction features: horizontal, vertical and diagonally in both directions. Previously we used skin-color information for face detection, but this is weak in some environments. So we detect facial regions using template matching this time. We experimented facial expressions recognition compared both methods of face detection. Consequently, the rate of recognition of expression rose in proportion to the amount of data used for learning in both methods.