Current cloth simulation systems have become powerful enough to be used interactively, and applications relevant to textile industries like virtually tailoring real garments seem to be possible in the near future. In this paper, we show that interactive cloth simulation can be improved by Virtual Reality techniques. We present the Virtual Dress-maker, an application for interactive assembly and physically based interactive simulation of cloth. In particular, the system allows to select and drag parts of the clothes during the simulation. The proposed application consists of a VR-interface client that provides nearly live-size stereo projection combined with 6DOF interaction and a server for physically based cloth simulation. In order to verify the usability of our approach, we elaborate on usability studies, which show that our application allows fast, easy, and precise interaction compared to interaction with a traditional 3D-desktop application based on 2D input devices.