Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007)
Toward Realistic and Artifact-Free Insider-Threat Data
Miami Beach, Florida, USA
December 10-December 14
ISBN: 0-7695-3060-5
Progress in insider-threat detection is currently limited by a lack of realistic, publicly available, real-world data. For reasons of privacy and confidentiality, no one wants to expose their sensitive data to the research community. Data can be sanitized to mitigate privacy and confidentiality con- cerns, but the mere act of sanitizing the data may introduce artifacts that compromise its utility for research purposes. If sanitization artifacts change the results of insider-threat experiments, then those results could lead to conclusions which are not true in the real world. The goal of this work is to investigate the consequences of sanitization artifacts on insider-threat detection experi- ments. We assemble a suite of tools and present a method- ology for collecting and sanitizing data. We use these tools and methods in an experimental evaluation of an insider- threat detection system. We compare the results of the eval- uation using raw data to the results using each of three types of sanitized data, and we measure the effect of each saniti- zation strategy. We establish that two of the three sanitization strategies actually alter the results of the experiment. Since these two sanitization strategies are commonly used in practice, we must be concerned about the consequences of sanitization artifacts on insider-threat research. On the other hand, we demonstrate that the third sanitization strategy addresses these concerns, indicating that realistic, artifact-free data sets can be created with appropriate tools and methods.