This paper addresses the problem of robust registering of images among perspective and omnidirectional cameras in a hybrid vision system (HVS). Nonlinearity in an HVS introduced by omnidirectional cameras poses challenges for computing pixel correspondences among images. In previous HVSs, cameras must be calibrated by performing registration. In this paper, we propose a non-linear approach for registering images in an HVS without requiring calibration of cameras. We first discuss the homographies between omnidirectional and perspective images under a local planar assumption. We then propose a robust patch level registration algorithm by exploiting a constraint on large 3D spatial planes. The proposed approach enables an HVS for applications that require quick deployment or active cameras. Experimental results have demonstrated feasibility of the proposed approach.