This paper presents SiteCity, a semi-automated building extraction system integrating photogrammetry, geometric constraints and image understanding algorithms. Existing automated building extraction systems produce mixed results and it is clear that human intervention is required to correct mistakes from fully automated systems. SiteCity gives human operators the ability to construct and manipulate three dimensional building objects using multiple images. Image understanding algorithms are integrated into SiteCity to assist users. The automated processes in SiteCity use user-delineated roof boundaries as cues, and attempt to locate the floor of a building and match the building object in other images. In addition, photogrammetric cues are used to assist automated processes. These automated processes are described and their performance is evaluated, illustrating that automated processes in SiteCity produce comparable performance to that of human subjects.