In this paper we introduce the Color-based Catchment Basin Merging Algorithm, an area-based segmentation approach that is specifically designed to segment low-altitude aerial images, as a preprocessing step to 3D reconstruction. This approach extends the watershed-based Catchment Basin Merging Algorithm, which is purely geometric, by introducing color similarity as an additional criterion to decide on the merging of evolving regions. Experiments performed with real aerial images of varied nature demonstrate that this modification eliminates over-segmentation problems of the existing algorithm, allowing large-scale urban scenes to be segmented in an accurate, reliable and fully automatic way.