We propose a unified graphical model that can represent both the causal and non-causal relationships among random variables and apply it to image segmentation problem. Specifically, we first propose to employ Conditional Random Field (CRF) to model the spatial relationships among image superpixel regions and their measurements. We then introduce a multi-layer Bayesian Network (BN) to model the causal dependencies that naturally exist among different image entities including image regions, edges, and vertices. The CRF model and the BN model are then systematically and seamlessly combined through the theories of Factor Graph to form a unified probabilistic graphical model that captures the complex relationships among different image entities. Using the unified graphical model, image segmentation can be performed through a principled probabilistic inference. Experimental results on the Weizmann horse dataset, on the VOC2006 cow dataset, and on the MSRC2 multiclass dataset demonstrate that our approach achieves favorable results compared to the state-of-the-art approaches as well as those that use either BN model or CRF model alone.