We propose DiaWear, a novel assistive mobile phone-based calorie monitoring system to improve the quality of life of diabetes patients and individuals with unique nutrition management needs. Our goal is to achieve improved daily semi-automatic food recognition using a mobile wearable cell phone. DiaWear currently uses a neural network classification scheme to identify food items from a captured image. It is difficult to account for the varying and implicit nature of certain foods using traditional image recognition techniques. To overcome these limitations, we introduce the role of the mobile phone as a platform to gather contextual information from the user and system in obtaining better food recognition.