This Article 
 Bibliographic References 
 Add to: 
Efficient and Scalable Algorithms for Inferring Likely Invariants in Distributed Systems
November 2007 (vol. 19 no. 11)
pp. 1508-1523
Distributed systems generate large amount of monitoring data such as log files to track their operational status. However, it is hard to correlate such monitoring data effectively across distributed systems and along observation time for system management. In previous work, we proposed a concept named flow intensity to measure the intensity with which internal monitoring data reacts to the volume of user requests. We calculated flow intensity measurements from monitoring data and proposed an algorithm to automatically search constant relationships between flow intensities measured at various points across distributed systems. If such relationships hold all the time, we regard them as invariants of the underlying systems. Invariants can be used to characterize complex systems and support various system management tasks. However, the computational complexity of previous invariant search algorithm is high so that it may not scale well in large systems with thousands of measurements. In this paper, we propose two efficient but approximate algorithms for inferring invariants in large-scale systems. The computational complexity of new randomized algorithms is significantly reduced and experimental results from a real system are also included to demonstrate the accuracy and efficiency of our new algorithms.

[1] G. Adomavicius and A. Tuzhilin, “Using Data Mining Methods to Build Customer Profiles,” Computer, vol. 34, no. 2, pp. 74-82, 2001.
[2] M.F. Arlitt and C.L. Williamson, “Web Server Workload Characterization: The Search for Invariants,” ACM SIGMETRICS Performance Evaluation Rev., vol. 24, no. 1, pp. 126-137, 1996.
[3] R. Baeza-Yates and B. Ribeiro-Neto, Modern Information Retrieval, first ed. Addison-Wesley, 1999.
[4] W. Brogan, Modern Control Theory, third ed. Prentice Hall, 1990.
[5] I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox, “Capturing, Indexing, Clustering, and Retrieving System History,” SIGOPS Operating Systems Rev., vol. 39, no. 5, pp. 105-118, 2005.
[6] T. Cormen, C. Leiserson, and R. Rivest, Introduction to Algorithms, first ed. MIT Press and McGraw-Hill, 1990.
[7] M. DeGroot and M. Schervish, Probability and Statistics, third ed. Addison-Wesley, 2001.
[8] M. Ernst, J. Cockrell, W. Griswold, and D. Notkin, “Dynamically Discovering Likely Program Invariants to Support Program Evolution,” IEEE Trans. Software Eng., vol. 27, no. 2, pp. 99-123, Feb. 2001.
[9] J. Gertler, Fault Detection and Diagnosis in Engineering Systems. Marcel Dekker, 1998.
[10] S. Hangal and M. Lam, “Tracking Down Software Bugs Using Automatic Anomaly Detection,” Proc. 24th Int'l Conf. Software Eng. (ICSE '02), pp. 291-301, 2002.
[11] J. Hoxmeier and C. DiCesare, “System Response Time and User Satisfaction: An Experimental Study of Browser-Based Applications,” Proc. Sixth Americas Conf. Information Systems (AMCIS '00), pp. 140-145, 2000.
[12] R. Isermann and P. Balle, “Trends in the Application of Model-Based Fault Detection and Diagnosis of Industrial Process,” Control Eng. Practice, vol. 5, no. 5, pp. 709-719, 1997.
[13] G. Jiang, H. Chen, and K. Yoshihira, “Discovering Likely Invariants of Distributed Transaction Systems for Autonomic System Management,” Proc. Third Int'l Conf. Autonomic Computing (ICAC '06), pp. 199-208, June 2006.
[14] G. Jiang, H. Chen, and K. Yoshihira, “Modeling and Tracking of Transaction Flow Dynamics for Fault Detection in Complex Systems,” IEEE Trans. Dependable and Secure Computing, vol. 3, no. 4, pp. 312-326, Oct.-Dec. 2006.
[15] N. Jiang, R. Villafane, K. Hua, A. Sawant, and K. Prabkakara, “ADMiRe: An Algebraic Data Mining Approach to System Performance Analysis,” IEEE Trans. Knowledge and Data Eng., vol. 17, no. 7, pp. 888-901, Aug. 2005.
[16] L. Ljung, System Identification—Theory for The User, second ed. Prentice Hall, 1998.
[17] D. Menasce, V. Almeida, R. Riedi, F. Ribeiro, R. Fonseca, and W. Meira, “In Search of Invariants for E-Business Workloads,” Proc. Second ACM Conf. Electronic Commerce (EC '00), pp. 56-65, 2000.
[18] R. Motwani and P. Raghavan, Randomized Algorithms. Cambridge Univ. Press, 1995.
[19] J. O'Madadhain, D. Fisher, S. White, and Y. Boey, “The Jung (Java Universal Network/Graph) Framework,” Technical Report UCI-ICS 03-17, UC Irvine, Dept. of Information and Computer Science,, 2003.
[20] D. Oppenheimer, A. Ganapathi, and D. Patterson, “Why Do Internet Services Fail, and What Can Be Done about It,” Proc. Fourth Usenix Symp. Internet Technologies and Systems (USITS '03), pp. 1-16, 2003.
[21] D. Patterson and A. Brown et al., “, Recovery-Oriented Computing (ROC): Motivation, Definition, Techniques, and Case Studies,” Technical Report UCB//CSD-02-1175, UC Berkeley, Dept. of Computer Science,, 2002.
[22] J. Perkins and M. Ernst, “Efficient Incremental Algorithms for Dynamic Detection of Likely Invariants,” Proc. ACM 12th Symp. Foundations of Software Eng. (FSE '04), pp. 23-32, Nov. 2004.
[23] /, 2006.
[24] J.O. Smith, Math. of the Discrete Fourier Transform (DFT). W3K Publishing, 2003.
[25] M. Spiliopoulou, C. Pohle, and L. Faulstich, “Improving the Effectiveness of a Web Site with Web Usage Mining,” Proc. Int'l Workshop Web Usage Analysis and User Profiling (WEBKDD '99), pp.142-162, 2000.
[26] J. Srivastava, R. Cooley, M. Deshpande, and P. Tan, “Web Usage Mining: Discovery and Applications of Usage Patterns from Web Data,” ACM SIGKDD Explorations Newsletter, vol. 1, no. 2, pp. 12-23, 2000.
[27] L. Wolsey and G. Nemhauser, Integer and Combinatorial Optimization. Wiley-Interscience, 1999.
[28] Q. Yang, H. Zhang, and T. Li, “Mining Web Logs for Prediction Models in WWW Caching and Prefetching,” Proc. Seventh ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '01), pp. 473-478, 2001.
[29] O. Zaiane, M. Xin, and J. Han, “Discovering Web Access Patterns and Trends by Applying Olap and Data Mining Technology on Web Logs,” Proc. IEEE Forum on Research and Technology Advances in Digital Libraries (ADL '98), pp. 19-29, Apr. 1998.
[30] G. Zhen, G. Jiang, H. Chen, and K. Yoshihira, “Tracking Probabilistic Correlation of Monitoring Data for Fault Detection in Complex Systems,” Proc. Int'l Conf. Dependable Systems and Networks (DSN '06), pp. 259-268, June 2006.

Index Terms:
Distributed Systems, System Management, Data mining, Time series analysis, Algorithms for data and knowledge management, Analysis of Algorithms and Problem Complexity
Guofei Jiang, Haifeng Chen, Kenji Yoshihira, "Efficient and Scalable Algorithms for Inferring Likely Invariants in Distributed Systems," IEEE Transactions on Knowledge and Data Engineering, vol. 19, no. 11, pp. 1508-1523, Nov. 2007, doi:10.1109/TKDE.2007.190648
Usage of this product signifies your acceptance of the Terms of Use.