Aiming at manipulating clothes, a model-driven method of estimating the state of hanging clothes is proposed. We suppose a system consisting of two manipulators and a camera. The task considered in this paper is to hold a pullover at its two shoulders by two manipulators respectively, as a first step for folding it. The proposed method estimates the state of the clothes held by one manipulator in a model-driven way and indicates the position to be held next by the other manipulator. First, the possible appearances of the pullover when it is held at one point are roughly predicted. Using discriminative features of the predicted appearances, the possible states for the observed appearance are selected. Each appearance of the possible state is partially deformed so as to get close to the observed appearance. The state whose appearance succesfully approaches closest to the observed appearance is selected as the final decision. The point to be held next is determined according to the state. The results of preliminary experiments using actual images have shown the good potential of the proposed method.