Semantic region labeling in outdoor scenes, e.g., identifying sky, grass, foliage, water, and snow, facilitates content-based image retrieval, organization, and enhancement. A major limitation of current object detectors is the significant number of misclassifications due to the similarities in color and texture characteristics of various object types and lack of context information. Building on previous work of spatial context-aware object detection, we have developed a further improved system by modeling and enforcing spatial context constraints specific to individual scene type. In particular, the scene context, in the form of factor graphs, is obtained by learning and subsequently used via MAP estimation to reduce misclassification by constraining the object detection beliefs to conform to the spatial context models. Experimental results show that the richer spatial context models improve the accuracy of object detection over the individual object detectors and the general outdoor scene model.