Visualizations leverage innate human capabilities for recognizing interesting aspects of data. Even if users might agree on what is interesting about a visualization, the steps that they use in the knowledge discovery process may be significantly different. This results in an inability to effectively recreate the exact conditions of the discovery process, which we call the knowledge rediscovery problem. Because we cannot expect a user to fully document each of their interactions, there is a need for visualization systems to maintain user trace data in a way that enhances a user's ability to communicate what they found to be interesting, as well as how they found it. We present a model for representing user interactions that articulates with a corresponding set of annotations, or observations that are made during the exploration. Such an ability is critical to addressing the knowledge rediscovery problem, and is a fundamental component for systems that must provide information provenance.