To be most useful, evaluation requires detailed observation and effective analysis of a full spectrum of system use. We have developed an approach and architecture for in-depth data collection and analysis of all use of a visualization system. User interface components in a large visualization and analysis platform automatically record user actions, and can restore previous system states on demand. Audio and text annotations are collected and indexed to states, allowing users to find a comment and restore the system state in which they made it; then explore actions before and after. History is visible as data; so a variety of visual displays and analysis techniques may be used to develop insights about the user?s experience. States of any part of the interface may be analyzed separately. Actions are categorized in a taxonomy as the user interface is built, allowing comparison of similar patterns in all tools. History data can co-exist with other data during data exploration, supporting further individual or group data exploration.