We develop theory and algorithms to incorporate image manifold constraints in a level set segmentation algorithm. This provides a framework to simultaneously segment every image of data sets that vary due to two degrees of freedom - such as cardiopulmonary MR images which deform due to patient breathing and heartbeats. We derive two formulations: a 4D level set which loosely couples the level set function between neighbors in the 2D image manifold and a multilayer level set function which uses different levels of the level set function to represent shapes that shrink or grow. We characterize the set of shape manifolds that the multilayer level set function can represent, and derive the evolution equations for both frameworks. We offer results of segmenting the left ventricle in cardiopulmonary MRI; by automatically discovering the 2D manifold structure of the image set then simultaneously segmenting every frame. Both extensions improve on frame-by-frame approaches, and a comparison of the results offers insight into their strengths and weaknesses.