In this paper, we propose a building recognition algorithm for the next generation navigation system which is to use captured video from the camera equipped in moving vehicle. After masking out the road area and vehicles on the road, we used edge segments of small size blocks within an image frame to remove background objects, trees, etc. Search range determination and edge tracing technique were also developed to determine the building area as the final step. The experiments show promising results such as 88.9 % detection rate on the average and 95.9% for navigation video without trees and signs. The proposed algorithm can be also applied to point-of-interest (POI) detection for various telematics applications.