We present an approach to estimating high dimensional discrete probability distributions with decomposable graphical models. Starting with the independence assumption we add edges and thus gradually increase the complexity of our model. Bounded by the Minimum Description Length principle we are able to produce highly accurate models without overfitting. We discuss the properties and benefits of this approach in an experimental evaluation and compare it to the well studied Chow-Liu algorithm.