This paper addresses the fundamental problem of automatically discovering an unknown moving deformable object in a monocular video sequence. No prior model of the object is used; it is only assumed that the object is composed of a set of apparently rigid parts that are not necessarily visible simultaneously, making it possible to circumvent the typical constraint of model initialization. A set of rigid parts describing the object is incrementally extracted in a modeling-tracking loop with reinforced memory. In this framework, low-level segmentation is considered as a necessary but non reliable process that helps initiating hypotheses. Motion-based layer segmentation from feature points and edges is applied only when and where no modeled parts can be tracked. Using the quantity of motion measure, it is further shown how to deal with temporal scale. The interest for this approach in applications such as human tracking is demonstrated for a set of various sequences including a rapidly evolving shape.