This paper presents a method for detecting building outlines using multiple aerial images. Since data-driven techniques may not be able to account for variability of building geometry and appearances, a key insight explored in this paper is a combination of model-based data driven front end with data driven learning in the back end for increased detection accuracy. Three main components of the detection algorithm are (i) Initialization. Image intensity and depth information are integrally used to efficiently detect buildings, and a robust rectilinear path finding algorithm is adopted to obtain good initial outlines. The initialization process involves the following steps: detecting location of buildings, determining the dominant orientations and knot points in the building outline and using these to fit the initial outline. (ii) Learning. A compact set of building features are defined and learned from the well-delineated buildings, and a tree-based classifier is applied to the whole region to detect any missing buildings and obtain their rough outlines. (iii) Verification and Refinement. Learned features are used to remove falsely detected buildings, and all outlines are refined by the deformation of rectilinear templates. The experiments, with improved detection rate and precise outlines, demonstrate the applicability of our algorithm.