In this work, we present a robust color quantization method based on the matrix of local pallets. The proposed method extracts a set of one-dimensional colors resulting from image partitioning. Image windowing depends upon the image variance which gives information on color dispersion. The color sets are then used to form the rows of the local pallet matrix that will be used as a smaller image. The selection of the principal pallet used to quantize the color image is accomplished on the local pallet matrix by computing the histogram, dividing the color interval adaptively and choosing the most frequent color with respect to interval total frequency. The experimental results give a good visual appearance and show that the method is very fast.