In this paper we present an image segmentation framework based on patch segmentation fusion. An image is first split into small patches. Segmentation is then performed on each patch using the algorithms of standard normalized cut [9], mean shift clustering [3], or K-means clustering. Each region in a patch segmentation is assigned a label so as to represent different parts. After that, a connectedness value is calculated between any two overlapping patch segmentations with certain kinds of labeling. A weight called border strength is calculated for a segmentation with a certain labeling. We optimize a global criterion function that quantifies the consistency and quality of patch segmentations by a simulated annealing algorithm [5] in order to find the optimal patch segmentations and labeling. Finally, global segmentation is reconstructed by fusing patch segmentations by multiple techniques. Experimental results on natural images are reported. Precision and recall rates are also calculated to evaluate the performance quantitively.