Duo is a wearable system designed to learn about everyday actions and the objects to which they are applied. Duo uses first person video and kinematic sensing of the head, torso, and dominant arm. Duo fundamentally relies on general methods for segmenting both kinematic and visual data into meaningful units. We describe methods for segmenting the wearer's arm, the objects with which the wearer interacts, and the actions the wearer applies to these objects. Both real-time interactive methods and off-line annotation tools are used to help Duo create meaningful segmentations from which to learn.