Man-made environments possess many regularities which can be efficiently exploited for image based rendering as well as robotic visual navigation and localization tasks. In this paper we present an approach for automatic extraction of dominant rectangular structures from a single view and show how they facilitate the recovery of camera pose, planar structure and matching across widely separated views. Since in the presented approach the rectangular hypothesis formation is based on a higher level information encoded by the presence of orthogonal vanishing directions, the dominant rectangular structures can be detected and matched despite the presence of multiple repetitive structures often encountered in a variety of buildings. The different stages of the approach are demonstrated on various examples of images of indoors and outdoors structured environments.