In this paper, we propose a new learning algorithm for constructing kernel least squares classifier. The new algorithm adopts a recursive learning way and a novel two-step sparsification procedure is incorporated into learning phase. These two most important features not only provide a feasible approach for large-scale problems as it is not necessary to store the entire kernel matrix, but also produce a very sparse model with fast training and testing time. Experimental results on a number of data classification problems are presented to demonstrate the competitiveness of new proposed algorithm.