We present a practical framework for registering a Mixed Reality(MR) environment of an arbitrary number of agents. Each agent consist of a head mounted display (HMD), which consists of a pair of stereo cameras. Each agent is assumed to be moving freely in 3D space and multiple HMDs need not have a common Field of View (FoV). We show that the plane at infinity and a common vertical vanishing point can be use to determine the exact orientation of all HMDs with respect to each other, and establish a common reference frame. Our method generalizes previous work which considers restricted camera motions. Using minimal assumptions, we are able to successfully demonstrate promising results on real data.