This paper addresses the problem of how to construct a robust character classifier when statistical pattern recognition techniques fail because of a limited quantity of data. The key ideas are two ways. One is to adopt a distortion-tolerant shape matching technique. Here, we use an affine-invariant matching technique of global affine transformation (GAT) correlation to absorb linear distortion between grayscale images. The other is to prepare multiple templates for dealing with nonlinear distortion or topologically different shapes. For this purpose K-means clustering is applied to a given limited data in a simple gradient feature space. Recognition experiments using the handwritten numeral database IPTP CDROM1B show that the proposed method achieves a much higher recognition rate of 97.2% as compared to that of 85.8% obtained by the conventional, simple correlation matching with a single template per category.