Image segmentation is critical to image processing and pattern recognition, An image segmentation system is proposed for the segmentation of color image based on neural networks. First, we introduce BP Neural network, it has the capacity of parallel computing, distributed saving, self-studying, fault-to-learnt and nonlinear function approximating. So it widely used in image segmentation, but it also has some unavoidable defects. Based on this, a new method of image segmentation based on both Wavelet Decomposition and self-organizing map neural network (short for SOM-NN) is proposed. It has a greater ability on resisting noise, improving the convergence and so on. Color prototypes provide a good estimate for object colors. The image pixels are classified by the matching of color prototypes. The experimental results show that the system has the desired ability for the segmentation of color image in a variety of vision tasks.