This paper proposes a real-time person independent facial expression recognition in two folds. One is using the stereo Active Appearance Model (STAAM) fitting algorithm that uses a geometric relationship between two tightly coupled views to increases the accuracy and speed of fitting. Also, a layered generalized discriminant analysis (GDA) classifier combines 3D shape and appearance to improve the recognition performance of person independent facial expressions. Experimental results show that the STAAM have a better fitting stability than the multi-view AAM (MVAAM), and the combination of the shape and appearance features using a layered GDA classifier improves the recognition performance of facial expressions greatly.