Modelling mixtures of multivariate t-distributions are usually used instead of Gaussian mixture models(GMM) as a robust approach, when one fits a set of continuous multivariate data which have wider tail than Gaussian?s or atypical observations, but it is unable to perform model selection automatically through the traditional EM (Expectation Maximization) algorithm. To solve this problem, a new algorithm, which is called Rival Penalized Expectation-Maximization (RPEM) algorithm, is proposed to t-mixture model (TMM). It can automatically select an appropriate number of densities in t-density mixture model. Experimental results on unsupervised color image segmentation demonstrate the affectivity of the proposed algorithm.