In this paper, we present a novel algorithm, called MP-Bayes, which induces Bayesian network structures from data based on entropy measures. One of the main features of this method is its parsimonious nature: it tends to represent the joint probability distribution underlying the data with the least number of arcs. While other methods that build Bayesian networks tend to overfit the data, MP-Bayes creates models that seem to have an adequate trade-off between accuracy and complexity. To support such a claim, we compare the performance of MP-Bayes, in terms of classification, against those of four different Bayesian network classifiers. The results show that our procedure generalises well in a wide range of situations.