The Community for Technology Leaders
2015 IEEE 31st International Conference on Data Engineering (ICDE) (2015)
Seoul, South Korea
April 13, 2015 to April 17, 2015
ISBN: 978-1-4799-7964-6
pp: 1316-1327
Douglas L. Schales , IBM Research, Switzerland
Xin Hu , IBM Research, Switzerland
Jiyong Jang , IBM Research, Switzerland
Reiner Sailer , Brown Brothers Harriman, USA
Marc Ph. Stoecklin , IBM Research, Switzerland
Ting Wang , IBM Research, Switzerland
ABSTRACT
In this paper, we present the design, architecture, and implementation of a novel analysis engine, called Feature Collection and Correlation Engine (FCCE), that finds correlations across a diverse set of data types spanning over large time windows with very small latency and with minimal access to raw data. FCCE scales well to collecting, extracting, and querying features from geographically distributed large data sets. FCCE has been deployed in a large production network with over 450,000 workstations for 3 years, ingesting more than 2 billion events per day and providing low latency query responses for various analytics. We explore two security analytics use cases to demonstrate how we utilize the deployment of FCCE on large diverse data sets in the cyber security domain: 1) detecting fluxing domain names of potential botnet activity and identifying all the devices in the production network querying these names, and 2) detecting advanced persistent threat infection. Both evaluation results and our experience with real-world applications show that FCCE yields superior performance over existing approaches, and excels in the challenging cyber security domain by correlating multiple features and deriving security intelligence.
INDEX TERMS
Feature extraction, Data mining, IP networks, Correlation, Computer security, Distributed databases, Real-time systems
CITATION
Douglas L. Schales, Xin Hu, Jiyong Jang, Reiner Sailer, Marc Ph. Stoecklin, Ting Wang, "FCCE: Highly scalable distributed Feature Collection and Correlation Engine for low latency big data analytics", 2015 IEEE 31st International Conference on Data Engineering (ICDE), vol. 00, no. , pp. 1316-1327, 2015, doi:10.1109/ICDE.2015.7113379
81 ms
(Ver 3.3 (11022016))