In this paper, we present a learning and inference framework for 3D human pose recovery using silhouettes represented by Gaussian mixtures. A Bayesian mixture of experts is learnt to conduct multimodal pose regression. The major contribution of this paper is the use of Gaussian mixtures as silhouette shape descriptor and Kullback-Leibler divergence (KLD) for silhouette distance and kernel computation. Using Gaussian mixtures and KLD makes the learning and inference robust to errors in silhouettes extraction. It also allows likelihood evaluation of different pose estimates. This is done by computing the similarity of the observed silhouette and the predicted silhouettes by a generic body model onto the image plane. The system was trained with silhouettes rendered using animation software driven by motion capture data. Experimental results using both synthetic and real image silhouettes illustrate the usefulness of the proposed framework.