Many health monitoring applications require that the patient keep a log of activities so that when the physiological data is analyzed off-line, the data can be correlated to what the patient was doing before or during health events of interest. However, patients often do a poor job of self-reporting, so it would be desirable to use context-awareness to automatically annotate the physiological data with the user?s activity. This paper presents some of the issues that arise in the design and use of a health monitoring and activity annotation system, including results from an informal survey about users? concerns about the control of physiological and activity data.