This paper presents a Bhattacharyya distance based feature selection method, which utilizes a recursive algorithm to obtain the optimal dimension reduction matrix in terms of the minimum upper bound of classification error under normal distribution for multi-class classification problem. In our scheme, PCA is incorporated as a pre-processing to reduce the intractably heavy computation burden of the recursive algorithm. The superior experimental results on the handwritten-digit recognition with the MNIST database and the steganalysis applications have demonstrated the effectiveness of our proposed method.