We consider a scene containing two independently and generally moving objects, viewed by two general perspective views. Using matching points arising from both objects simultaneously we derive a geometrical constraint, applicable to points from both objects, we call the segmentation matrix. We then use this constraint in order to recover the fundamental matrices associated with each object, or simply to segment the scene into the two objects. Moreover, when the two bodies move in pure translation relative to each other we can both segment the scene and recover the affine calibration (homography at infinity) of the camera geometry. Unlike algorithms suggested in the past we need only two images, we work with general projective cameras (rather than affine or orthographic) and with general body motion, and no prior information beyond point matches is required.