This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Detecting Intrusions through System Call Sequence and Argument Analysis
October-December 2010 (vol. 7 no. 4)
pp. 381-395
Federico Maggi, Politecnico di Milano, Milano
Matteo Matteucci, Politecnico di Milano, Milano
Stefano Zanero, Politecnico di Milano, Milano
We describe an unsupervised host-based intrusion detection system based on system call arguments and sequences. We define a set of anomaly detection models for the individual parameters of the call. We then describe a clustering process that helps to better fit models to system call arguments and creates interrelations among different arguments of a system call. Finally, we add a behavioral Markov model in order to capture time correlations and abnormal behaviors. The whole system needs no prior knowledge input; it has a good signal-to-noise ratio, and it is also able to correctly contextualize alarms, giving the user more information to understand whether a true or false positive happened, and to detect global variations over the entire execution flow, as opposed to punctual ones over individual instances.

[1] J.P. Anderson, "Computer Security Threat Monitoring and Surveillance," technical report, J.P. Anderson, Apr. 1980.
[2] S. Forrest, S.A. Hofmeyr, A. Somayaji, and T.A. Longstaff, "A Sense of Self for Unix Processes," Proc. IEEE Symp. Security and Privacy (S&P), 1996.
[3] J.B.D. Cabrera, L. Lewis, and R. Mehara, "Detection and Classification of Intrusion and Faults Using Sequences of System Calls," ACM SIGMOD Record, vol. 30, no. 4, 2001.
[4] S. Hofmeyr, S. Forrest, and A. Somayaji, "Intrusion Detection Using Sequences of System Calls," J. Computer Security, vol. 6, pp. 151-180, 1998.
[5] A. Somayaji and S. Forrest, "Automated Response Using System-Call Delays," Proc. Ninth USENIX Security Symp., Aug. 2000.
[6] W.W. Cohen, "Fast Effective Rule Induction," Proc. 12th Int'l Conf. Machine Learning (ICML '95), A. Prieditis and S. Russell, eds., pp. 115-123, July 1995.
[7] W. Lee and S. Stolfo, "Data Mining Approaches for Intrusion Detection," Proc. Seventh USENIX Security Symp., 1998.
[8] D. Ourston, S. Matzner, W. Stump, and B. Hopkins, "Applications of Hidden Markov Models to Detecting Multi-Stage Network Attacks," Proc. 36th Ann. Hawaii Int'l Conf. System Sciences (HICSS-36 '03), p. 334, 2003.
[9] S. Jha, K. Tan, and R.A. Maxion, "Markov Chains, Classifiers, and Intrusion Detection," Proc. 14th IEEE Workshop Computer Security Foundations (CSFW '01), p. 206, 2001.
[10] C.C. Michael and A. Ghosh, "Simple, State-Based Approaches to Program-Based Anomaly Detection," ACM Trans. Information and System Security, vol. 5, no. 3, pp. 203-237, 2002.
[11] R. Sekar, M. Bendre, D. Dhurjati, and P. Bollineni, "A Fast Automaton-Based Method for Detecting Anomalous Program Behaviors," Proc. IEEE Symp. Security and Privacy (S&P '01), pp. 144-155, 2001.
[12] D. Wagner and D. Dean, "Intrusion Detection via Static Analysis," Proc. IEEE Symp. Security and Privacy (S&P '01), p. 156, 2001.
[13] J.T. Giffin, D. Dagon, S. Jha, W. Lee, and B.P. Miller, "Environment-Sensitive Intrusion Detection," Proc. Eighth Int'l Symp. Recent Advances in Intrusion Detection (RAID '05), pp. 185-206, 2005.
[14] D.-Y. Yeung and Y. Ding, "Host-Based Intrusion Detection Using Dynamic and Static Behavioral Models," Pattern Recognition, vol. 36, pp. 229-243, Jan. 2003.
[15] C. Kruegel, D. Mutz, F. Valeur, and G. Vigna, "On the Detection of Anomalous System Call Arguments," Proc. European Symp. Research in Computer Security (ESORICS '03), Oct. 2003.
[16] G. Tandon and P. Chan, "Learning Rules from System Call Arguments and Sequences for Anomaly Detection," Proc. ICDM Workshop Data Mining for Computer Security (DMSEC '03), pp. 20-29, 2003.
[17] S. Bhatkar, A. Chaturvedi, and R. Sekar, "Dataflow Anomaly Detection," Proc. IEEE Symp. Security and Privacy (S&P '06), May 2006.
[18] R.G. Bace, Intrusion Detection. Macmillan, 2000.
[19] D.E. Denning, "An Intrusion-Detection Model," IEEE Trans. Software Eng., vol. 13, no. 2, pp. 222-232, Feb. 1987.
[20] M. Burgess, H. Haugerud, S. Straumsnes, and T. Reitan, "Measuring System Normality," ACM Trans. Computer Systems, vol. 20, no. 2, pp. 125-160, 2002.
[21] N. Ye and Q. Chen, "An Anomaly Detection Technique Based on a Chi-Square Statistic for Detecting Intrusions into Information Systems," Quality and Reliability Eng. Int'l, vol. 17, no. 2, pp. 105-112, 2001.
[22] H. Debar, M. Becker, and D. Siboni, "A Neural Network Component for an Intrusion Detection System," Proc. IEEE Symp. Research in Computer Security and Privacy, 1992.
[23] M. Theus and M. Schonlau, "Intrusion Detection Based on Structural Zeroes," Statistical Computing and Graphics Newsletter, vol. 9, pp. 12-17, 1998.
[24] A.K. Gosh, J. Wanken, and F. Charron, "Detecting Anomalous and Unknown Intrusions against Programs," Proc. 14th Ann. Computer Security Applications Conf. (ACSAC '98), p. 259, 1998.
[25] J.C. Galeano, A. Veloza-Suan, and F.A. Gonz?lez, "A Comparative Analysis of Artificial Immune Network Models," Proc. 2005 Conf. Genetic and Evolutionary Computation (GECCO '05), pp. 361-368, 2005.
[26] S. Forrest, S.A. Hofmeyr, and A. Somayaji, "Computer Immunology," Comm. ACM, vol. 40, no. 10, pp. 88-96, 1997.
[27] C. Ko, G. Fink, and K. Levitt, "Automated Detection of Vulnerabilities in Privileged Programs by Execution Monitoring," Proc. 10th Ann. Computer Security Applications Conf. (ACSAC '94), pp. 134-144, 1994.
[28] W. Lee, S.J. Stolfo, and P.K. Chan, "Learning Patterns from Unix Process Execution Traces for Intrusion Detection," Proc. AAAI97 Workshop AI Approaches to Fraud Detection and Risk Management, pp. 50-56, http://citeseer.ist.psu.edulee97learning.html , 1997.
[29] W. Lee and W. Fan, "Mining System Audit Data: Opportunities and Challenges," ACM SIGMOD Record, vol. 30, no. 4, pp. 35-44, 2001.
[30] C. Warrender, S. Forrest, and B.A. Pearlmutter, "Detecting Intrusions Using System Calls: Alternative Data Models," Proc. IEEE Symp. Security and Privacy (S&P '99), pp. 133-145, 1999.
[31] S. Forrest, A.S. Perelson, L. Allen, and R. Cherukuri, "Self-Nonself Discrimination in a Computer," Proc. IEEE Symp. Security and Privacy (S&P '94), p. 202, 1994.
[32] S. Zanero, "Behavioral Intrusion Detection," Proc. 19th Int'l Symp. Computer and Information Sciences (ISCIS '04), pp. 657-666, Oct. 2004.
[33] D. Wagner and P. Soto, "Mimicry Attacks on Host-Based Intrusion Detection Systems," Proc. Ninth ACM Conf. Computer and Comm. Security (CCS '02), pp. 255-264, 2002.
[34] D. Mutz, F. Valeur, G. Vigna, and C. Kruegel, "Anomalous System Call Detection," ACM Trans. Information and System Security, vol. 9, no. 1, pp. 61-93, 2006.
[35] A. Stolcke and S. Omohundro, "Hidden Markov Model Induction by Bayesian Model Merging," Advances in Neural Information Processing Systems. Morgan Kaufmann, vol. 5, pp. 11-18, 1993.
[36] A. Stolcke and S.M. Omohundro, "Inducing Probabilistic Grammars by Bayesian Model Merging," Proc. Second Int'l Colloquium on Grammatical Inference and Applications (ICGI '94), pp. 106-118, 1994.
[37] S.Y. Lee, W.L. Low, and P.Y. Wong, "Learning Fingerprints for a Database Intrusion Detection System," Proc. Seventh European Symp. Research in Computer Security (ESORICS '02), pp. 264-280, 2002.
[38] LibAnomaly, http://www.cs.ucsb.edu/~rsglibAnomaly, 2008.
[39] S. Zanero, "Unsupervised Learning Algorithms for Intrusion Detection," PhD dissertation, Politecnico di Milano T.U., May 2006.
[40] G.H. Golub and C.F.V. Loan, Matrix Computations, third ed. Johns Hopkins Univ. Press, 1996.
[41] L.R. Rabiner, "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition," Proc. IEEE, vol. 77, pp. 257-286, 1989.
[42] W.R. Pestman, Mathematical Statistics: An Introduction. Walter de Gruyter, 1998.
[43] R. Lippmann, J.W. Haines, D.J. Fried, J. Korba, and K. Das, "The 1999 DARPA Off-Line Intrusion Detection Evaluation," Computer Networks, vol. 34, no. 4, pp. 579-595, 2000.
[44] J. McHugh, "Testing Intrusion Detection Systems: A Critique of the 1998 and 1999 DARPA Intrusion Detection System Evaluations as Performed by Lincoln Laboratory," ACM Trans. Information and System Security, vol. 3, no. 4, pp. 262-294, 2000.
[45] M.V. Mahoney and P.K. Chan, "An analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection," Proc. Sixth Int'l Symp. Recent Advances in Intrusion Detection (RAID '03), pp. 220-237, Sept. 2003.
[46] Shmoo Group, Capture the CTF, http:/cctf.shmoo.com 2008.
[47] R.N.M. Watson and W. Salamon, "The FreeBSD Audit System," Proc. UKUUG Ann. Large Installation Systems Administration Conf. (LISA '06), Mar. 2006.

Index Terms:
Intrusion detection, anomaly detection, behavior detection, Markov models.
Citation:
Federico Maggi, Matteo Matteucci, Stefano Zanero, "Detecting Intrusions through System Call Sequence and Argument Analysis," IEEE Transactions on Dependable and Secure Computing, vol. 7, no. 4, pp. 381-395, Oct.-Dec. 2010, doi:10.1109/TDSC.2008.69
Usage of this product signifies your acceptance of the Terms of Use.