In this paper we present a framework for the classification and segmentation of motion data. First, a representation of different two-dimensional motion categories is proposed. Secondly, a system to categorize and segment motion is presented based on hidden Markov models, commonly used in speech recognition. Input to the system consists of online pen stroke data which includes the x, y position and time of each point along the line. Using derived speed and direction information the system classifies and segments the input into particular categories of motion. The resulting categorical information may be then used to describe the scene, extrapolate events, or as a part of a gesture recognition system. Applications beyond pen-based input are discussed. This paper contributes to pen-based motion recognition research in two ways. First, a classification is performed based on a continuous sequence of observations, rather then feature extraction. Secondly, pen motion is transformed into a translation and rotation invariant representation prior to classification.