This paper presents a speech emotion recognition system on nonlinear manifold. Instead of straight-line distance, geodesic distance was adopted to preserve the intrinsic geometry of speech corpus. Based on geodesic distance estimation, we developed an enhanced Lipschitz embedding to embed the 64-dimensional acoustic features into a sixdimensional space. In this space, speech data with the same emotional state were located close to one plane, which was beneficial to emotion classification. The compressed testing data were classified into six archetypal emotional states (neutral, anger, fear, happiness, sadness and surprise) by a trained linear Support Vector Machine (SVM) system. Experimental results demonstrate that compared with traditional methods of feature extraction on linear manifold and feature selection, the proposed system makes 9%-26% relative improvement in speaker-independent emotion recognition and 5%-20% improvement in speaker-dependent.