In this paper, we describe a new system for analyzing the motions involved in handling documents on a desk. A camera mounted above the desk captures video images and the motion, which occurs when someone handles the document, is classified as movement of the document, turning of the pages, or finger pointing. Furthermore, the system can detect the appearance of fresh regions because of movement of the document and the turning of its pages and it can detect the positions of fingers in the image to realize a natural pointing function. We evaluated implementations of some components of the system and found that they provide sufficient performance to act as parts of a user interface for an interactive document management system.