Abdominal visceral fat accumulation is one of the most important cardiovascular risk factors. Currently, Computed Tomography and Magnetic Resonance images are manually segmented to quantify abdominal fat distribution. The manual delineation of subcutaneous and visceral fat is labor intensive, time consuming, and subject to inter- and intra-observer variability. An automatic segmentation method would eliminate intra- and inter-observer variability and provide more consistent results. In this paper, we present a hierarchical, multi-class, multi-feature, fuzzy affinity-based computational framework for tissue segmentation in medical images. We have applied this framework for automatic segmentation of abdominal fat. An evaluation of the accuracy of our method indicates bias and limits of agreement comparable to the inter-observer variability inherent in manual segmentation.