When a algorithms extracting and recognizing characters from text image is devised, the time loss and recognizing quality are two of the most important properties taken into account. To achieve high recognition performance in both of them, we provide a simple but rapid and efficient method to extract character features, and then a structure called Hierarchical Radial Basis Function (RBF) Neural Networks is proposed to recognize particular characters contained in text images. The networks involve several sub-RBF neural networks connected like a tree, which could classify characters correctly and rapidly. The hierarchical constitution is well designed to relieve the time expense while making a correct decision, which is a vital attribute to evaluate an algorithm and is critical when the text image analyzed consists tremendous characters. Particle Swarm Optimization, a newly developed global search evolutionary algorithm, is enrolled to train this structure represented in this paper for a swiftly convergence.