There is widespread interest today in developing tools that can diagnose the??cause of a system failure accurately and efficiently based on monitoring data collected from the system. Over time, the system monitoring data will contain two types of failure data: (i) annotated failure data L, which is monitoring data collected from failure states of the system, where the cause of failure has been diagnosed and attached as annotations with the data; and (ii) unannotated failure data U. Previous work on wholly- or partially-automated diagnosis focused on L or U in isolation. In this paper, we argue that it is important to consider both L and U together to improve the overall accuracy??of diagnosis; and in particular, to proactively move instances from U to??L.??However, such movement requires manual diagnosis effort from system??administrators. Since manual diagnosis is expensive and time-consuming, we propose an algorithm to make the best use of manual effort while maximizing??the benefit gained from newly diagnosed instances. We report an experimental evaluation of our algorithm using data from a variety??of failures---both single failures and multiple correlated failures---injected in a testbed, as well as with synthetic data.