As whiteboards have become a popular tool in meeting rooms, there has been a growing interest in making use of the whiteboard as a user interface for human computer interaction. Therefore, systems based on electronic white-boards have been developed in order to serve as meeting assistants for e.g. collaborative working. However, as special pens and erasers are required, the natural interaction is restricted. In order to render this communication method more natural it was proposed to retain ordinary whiteboard and pens and to visually observe the writing process using a video camera [11, 9]. In this paper a prototype system for automatic video-based whiteboard reading is presented. The system is designed for recognizing unconstrained handwritten text and is further characterized by an incremental processing strategy in order to facilitate recognizing portions of text as soon as they have been written on the board. We will present the methods employed for extracting text regions, pre-processing, feature extraction, and statistical modeling and recognition. Evaluation results on a writer independent unconstrained handwriting recognition task demonstrate the feasibility of the proposed approach.