We study the satellite image analysis problem with focus on extracting the man-made buildings. Instead of assuming simple rectangular building shape as in the most of previous work, we apply probabilistic learning method to statistical modeling the building structures. The model can achieve high robustness to large shape variation. We also propose a novel energy function to incorporate the statistical model into a graph optimization framework. Once the graph is constructed on image edges, the buildings can be extracted as closed cycles on graph efficiently and accurately. Experiments on real images demonstrate the effectiveness and robustness of the approach.