loading...
 This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
2009 Ninth IEEE International Conference on Data Mining
Online System Problem Detection by Mining Patterns of Console Logs
Miami, Florida
December 06-December 09
ISBN: 978-0-7695-3895-2
We describe a novel application of using data mining and statistical learning methods to automatically monitor and detect abnormal execution traces from console logs in an online setting. Different from existing solutions, we use a two stage detection system. The first stage uses frequent pattern mining and distribution estimation techniques to capture the dominant patterns (both frequent sequences and time duration). The second stage use principal component analysis based anomaly detection technique to identify actual problems. Using real system data from a 203-node Hadoop [1] cluster, we show that we can not only achieve highly accurate and fast problem detection, but also help operators better understand execution patterns in their system.
Index Terms:
console logs, system management, monitoring, problem detection, logs, pattern mining
Citation:
Wei Xu, Ling Huang, Armando Fox, David Patterson, Michael Jordan, "Online System Problem Detection by Mining Patterns of Console Logs," icdm, pp.588-597, 2009 Ninth IEEE International Conference on Data Mining, 2009
Usage of this product signifies your acceptance of the Terms of Use.