In this paper, a new adaptive algorithm for Linear Discriminant Analysis (LDA) is presented. The major advantage of the algorithm is the fast convergence rate, which distinguishes it from the existing on-line methods. Current adaptive methods based on the gradient descent optimization technique use a fixed or a monotonically decreasing step size in each iteration. In this work, we use the steepest descent optimization method to optimally determine the step size in each iteration. It is shown that an optimally variable step size, significantly improves the convergence rate of the algorithm, compared to the conventional methods. The new algorithm has been implemented using a self-organized neural network and its advantages in an on-line pattern recognition applications are demonstrated.