Image classification is an interesting topic in multimedia processing. Recently, there were many researchers proposed radial basis function-based (RBF) methods to deal with image classification. However, the traditional RBF neural networks were sensitive to center initialization. To obtain appropriate centers, it needs to find the significant features for further RBF clustering. In addition, the training procedure of the traditional RBF is time-consuming. In order to cope with these problems, a self-organizing map (SOM) neural network is proposed to select more appropriate centers for RBF network, and a modular RBF (MRBF) neural network is proposed to improve the classification rate and speed up the training time. The experimental results show that the proposed MRBF network has better performance than DWTbased method, traditional RBF neural network and the Tree Structured Wavelet (TWS) in image classification. The experimental results also show that the training time of proposed MRBF neural network is much faster than the traditional RBF neural network.