This paper proposes a novel framework that integrates bottom-up information and top-down information for scene understanding. Bottom-up information is derived from local features of texture and color. Top-down information is generated from a holistic image context. The information is integrated effectively by extension of the Ising model, which is a simple model of ferromagnetism. Locally and globally consistent image recognition is achieved through an iterative process. The proposed method showed 91.8% accuracy in road-image labeling, which is superior to results obtained using only bottom-up information (81.9%) and the best accuracy obtained using the other method (90.7%).