Most merging and splitting segmentation methods aim to construct a hierarchical structure from an image by minimizing or maximizing a homogeneity measure. This latter generally includes radiometrical information, but rarely includes geometrical information and ignore the high level information on the image content. Moreover, the hierarchies issued from these approaches may suffer from a structural instability and deficiency in the "semantic" of the regions related to the image content and to the energy or the criterion which does not contain any high level prior knowledge. In this paper, we propose to improve the semantic content of the hierarchy by adding a new term called "contextual cost". This term integrates the prior knowledge on the image, derived from a classifier after a supervised learning on the semantic classes composing the image. Its purpose is to better drive the merging process in the construction of meaningful regions by penalizing spurious fusions.