Motion trajectories are powerful cues for event detection and recognition. In this paper we present a system for manual task analysis that distinguishes between skin and object motion and learns activity patterns through analysing object trajectories. It is particularly suited to the recognition of common object handling tasks. Our vision system performs hand skin detection and object segmentation for each frame in a sequence. The object trajectories are then modelled as motion time series. We have compared the performance of several different time series indexing schemes: symbolic, polynomial and orthonormal basis functions used for trajectory similarity retrieval and classification. We then attempt to cluster objectcentred motion patterns in the coefficient feature space. The proposed technique is validated on two different datasets, Australian Sign Language and object handling data obtained in the laboratory. Applications to task recognition and motion data mining in industrial surveillance applications are envisaged.