This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Beyond Output Voting: Detecting Compromised Replicas Using HMM-Based Behavioral Distance
April-June 2009 (vol. 6 no. 2)
pp. 96-110
Debin Gao, Singapore Management University, Singapore
Michael K. Reiter, University of North Carolina at Chapel Hill, Chapel Hill
Dawn Song, University of California, Berkeley, Berkeley
Many host-based anomaly detection techniques have been proposed to detect code-injection attacks on servers. The vast majority, however, are susceptible to "mimicry” attacks in which the injected code masquerades as the original server software, including returning the correct service responses, while conducting its attack. "Behavioral distance,” by which two diverse replicas processing the same inputs are continually monitored to detect divergence in their low-level (system-call) behaviors and hence potentially the compromise of one of them, has been proposed for detecting mimicry attacks. In this paper, we present a novel approach to behavioral distance measurement using a new type of Hidden Markov Model, and present an architecture realizing this new approach. We evaluate the detection capability of this approach using synthetic workloads and recorded workloads of production web and game servers, and show that it detects intrusions with substantially greater accuracy than a prior proposal on measuring behavioral distance. We also detail the design and implementation of a new architecture, which takes advantage of virtualization to measure behavioral distance. We apply our architecture to implement intrusion-tolerant web and game servers, and through trace-driven simulations demonstrate that it experiences moderate performance costs even when thresholds are set to detect stealthy mimicry attacks.

[1] S. Forrest, S.A. Hofmeyr, A. Somayaji, and T.A. Longstaff, “A Sense of Self for Unix Processes,” Proc. IEEE Symp. Security and Privacy (S&P), 1996.
[2] A. Wespi, M. Dacier, and H. Debar, “Intrusion Detection Using Variable-Length Audit Trail Patterns,” Proc. Third Int'l Symp. Recent Advances in Intrusion Detection (RAID), 2000.
[3] D. Wagner and D. Dean, “Intrusion Detection via Static Analysis,” Proc. IEEE Symp. Security and Privacy (S&P), 2001.
[4] R. Sekar, M. Bendre, D. Dhurjati, and P. Bollineni, “A Fast Automaton-Based Method for Detecting Anomalous Program Behaviors,” Proc. IEEE Symp. Security and Privacy (S&P), 2001.
[5] J. Giffin, S. Jha, and B. Miller, “Detecting Manipulated Remote Call Streams,” Proc. 11th USENIX Security Symp., 2002.
[6] H. Feng, O. Kolesnikov, P. Fogla, W. Lee, and W. Gong, “Anomaly Detection Using Call Stack Information,” Proc. IEEE Symp. Security and Privacy (S&P), 2003.
[7] D. Gao, M.K. Reiter, and D. Song, “On Gray-Box Program Tracking for Anomaly Detection,” Proc. 13th USENIX Security Symp., 2004.
[8] H. Feng, J. Giffin, Y. Huang, S. Jha, W. Lee, and B.P. Miller, “Formalizing Sensitivity in Static Analysis for Intrusion Detection,” Proc. IEEE Symp. Security and Privacy (S&P), 2004.
[9] J. Giffin, S. Jha, and B. Miller, “Efficient Context-Sensitive Intrusion Detection,” Proc. Symp. Network and Distributed System Security (NDSS), 2004.
[10] D. Gao, M.K. Reiter, and D. Song, “Gray-Box Extraction of Execution Graph for Anomaly Detection,” Proc. 11th ACM Conf. Computer and Comm. Security (CCS), 2004.
[11] K. Tan, J. McHugh, and K. Killourhy, “Hiding Intrusions: From the Abnormal to the Normal and Beyond,” Proc. Fifth Int'l Workshop Information Hiding, Oct. 2002.
[12] D. Wagner and P. Soto, “Mimicry Attacks on Host-Based Intrusion Detection Systems,” Proc. Ninth ACM Conf. Computer and Comm. Security (CCS), 2002.
[13] C. Kruegel, E. Kirda, D. Mutz, W. Robertson, and G. Vigna, “Automating Mimicry Attacks Using Static Binary Analysis,” Proc. 14th USENIX Security Symp., Aug. 2005.
[14] J. Giffin, S. Jha, and B. Miller, “Automated Discovery of Mimicry Attacks,” Proc. Ninth Int'l Symp. Recent Advances in Intrusion Detection (RAID), 2006.
[15] K. Shin and P. Ramanathan, “Diagnosis of Processors with Byzantine Faults in a Distributed Computing System,” Proc. 17th Int'l Symp. Fault-Tolerant Computing (FTC), 1987.
[16] R.W. Buskens and R.P. Bianchini Jr., “Distributed On-Line Diagnosis in the Presence of Arbitrary Faults,” Proc. 23rd Int'l Symp. Fault-Tolerant Computing (FTC '93), June 1993.
[17] L. Alvisi, D. Malkhi, E. Pierce, and M.K. Reiter, “Fault Detection for Byzantine Quorum Systems,” IEEE Trans. Parallel Distributed Systems, vol. 12, no. 9, Sept. 2001.
[18] L. Lamport, “The Implementation of Reliable Distributed Multiprocess Systems,” Computer Networks, vol. 2, 1978.
[19] F.B. Schneider, “Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial,” ACM Computing Surveys, vol. 22, no. 4, Dec. 1990.
[20] M.K. Reiter, “Secure Agreement Protocols: Reliable and Atomic Group Multicast in Rampart,” Proc. Second ACM Conf. Computer and Comm. Security (CCS '94), Nov. 1994.
[21] M. Castro and B. Liskov, “Practical Byzantine Fault Tolerance and Proactive Recovery,” ACM Trans. Computer Systems, vol. 20, no. 4, Nov. 2002.
[22] C. Cachin and J.A. Poritz, “Secure Intrusion-Tolerant Replication on the Internet,” Proc. Int'l Conf. Dependable Systems and Networks (DSN), 2002.
[23] J. Yin, J. Martin, A. Venkataramani, L. Alvisi, and M. Dahlin, “Separating Agreement from Execution for Byzantine Fault Tolerant Services,” Proc. 19th ACM Symp. Operating System Principles (SOSP '03), Oct. 2003.
[24] M. Abd-El-Malek, G.R. Ganger, G.R. Goodson, M.K. Reiter, and J.J. Wylie, “Fault-Scalable Byzantine Fault-Tolerant Services,” Proc. 20th ACM Symp. Operating System Principles (SOSP'05), Oct. 2005.
[25] D. Gao, M.K. Reiter, and D. Song, “Behavioral Distance for Intrusion Detection,” Proc. Eighth Int'l Symp. Recent Advances in Intrusion Detection (RAID), 2005.
[26] D. Gao, M.K. Reiter, and D. Song, “Behavioral Distance Measurement Using Hidden Markov Models,” Proc. Ninth Int'l Symp. Recent Advances in Intrusion Detection (RAID), 2006.
[27] P.H. Sellers, “On the Theory and Computation of Evolutionary Distances,” SIAM J. Applied Math., vol. 26, 1974.
[28] C. Warrender, S. Forrest, and B. Pearlmutter, “Detecting Intrusions Using System Calls: Alternative Data Models,” Proc. IEEE Symp. Security and Privacy (S&P), 1999.
[29] S. Cho and S. Han, “Two Sophisticated Techniques to Improve HMM-Based Intrusion Detection Systems,” Proc. Sixth Int'l Symp. Recent Advances in Intrusion Detection (RAID), 2003.
[30] B. Cox, D. Evans, A. Filipi, J. Rowanhill, W. Hu, J. Davidson, J. Knight, A. Nguyen-Tuong, and J. Hiser, “N-Variant Systems—A Secretless Framework for Security through Diversity,” Proc. 15th USENIX Security Symp., Aug. 2006.
[31] L. Cavallaro, “Comprehensive Memory Error Protection via Diversity and Taint-Tracking,” PhD dissertation, Universita' Degli Studi Di Milano, 2007.
[32] L.R. Rabiner, “A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition,” Proc. IEEE, Feb. 1989.
[33] X.D. Hoang, J. Hu, and P. Bertok, “A Multi-Layer Model for Anomaly Intrusion Detection Using Program Sequences of System Calls,” Proc. 11th IEEE Int'l Conf. Networks (ICON), 2003.
[34] I.M. Meyer and R. Durbin, Comparative ab initio Prediction of Gene Structures Using Pair HMMs. Oxford Univ. Press, 2002.
[35] L. Pachter, M. Alexandersson, and S. Cawley, “Applications of Generalized Pair Hidden Markov Models to Alignment and Gene Finding Problems,” Computational Biology, vol. 9, no. 2, 2002.
[36] J. Just, J. Reynolds, L. Clough, M. Danforth, K. Levitt, R. Maglich, and J. Rowe, “Learning Unknown Attacks—A Start,” Proc. Fifth Int'l Symp. Recent Advances in Intrusion Detection (RAID), 2002.
[37] J. Reynolds, J. Just, E. Lawson, L. Clough, and R. Maglich, “The Design and Implementation of an Intrusion Tolerant System,” Proc. Int'l Conf. Dependable Systems and Networks (DSN), 2002.
[38] E. Totel, F. Majorczyk, and L. Me, “COTS Diversity Based Intrusion Detection and Application to Web Servers,” Proc. Eighth Int'l Symp. Recent Advances in Intrusion Detection (RAID), 2005.
[39] C. Kruegel, D. Mutz, F. Valeur, and G. Vigna, “On the Detection of Anomalous System Call Arguments,” Proc. Eighth European Symp. Research in Computer Security (ESORICS), 2003.
[40] Sufatrio and R.H.C. Yap, “Improving Host-Based IDS with Argument Abstraction to Prevent Mimicry Attacks,” Proc. Eighth Int'l Symp. Recent Advances in Intrusion Detection (RAID), 2005.
[41] G. Tandon and P. Chan, “Learning Rules from System Calls Arguments and Sequences for Anomaly Detection,” Proc. ICDM Workshop Data Mining for Computer Security (DMSEC), 2003.
[42] G. Tandon and P. Chan, “Learning Useful System Call Attributes for Anomaly Detection,” Proc. 18th Int'l Florida Artificial Intelligence Research Symp. (FLAIRS), 2005.
[43] S. Bhatkar, A. Chaturvedi, and R. Sekar, “Dataflow Anomaly Detection,” Proc. IEEE Symp. Security and Privacy (S&P), 2006.
[44] C. Parampalli, R. Sekar, and R. Johnson, “A Practical Mimicry Attack against Powerful System-Call Monitors,” Proc. ACM Symp. Information, Computer and Comm. Security (ASIACCS '08), Mar. 2008.
[45] S. Chen, J. Xu, E.C. Sezer, P. Gauriar, and R.K. Iyer, “Non-Control-Data Attacks are Realistic Threats,” Proc. 14th USENIX Security Symp., Aug. 2005.
[46] L.E. Baum and T. Petrie, “Statistical Inference for Probabilistic Functions of Finite State Markov Chains,” Ann. Math. Statistics, vol. 37, 1966.
[47] R.I.A. Davis, B.C. Lovell, and T. Caelli, “Improved Estimation of Hidden Markov Model Parameters from Multiple Observation Sequences,” Proc. 16th Int'l Conf. Pattern Recognition (ICPR), 2002.
[48] L. von Ahn, R. Liu, and M. Blum, “Peekaboom: A Game for Locating Objects in Images,” Proc. Conf. Human Factors in Computing Systems (CHI), 2006.
[49] P. Narasimhan, L.E. Moser, and P.M. Melliar-Smith, “Enforcing Determinism for the Consistent Replication of Multithreaded CORBA Applications,” Proc. 18th IEEE Symp. Reliable Distributed Systems (SRDS '99), pp. 263-273, Oct. 1999.

Index Terms:
Intrusion detection, replicated system, output voting, system call, behavioral distance.
Citation:
Debin Gao, Michael K. Reiter, Dawn Song, "Beyond Output Voting: Detecting Compromised Replicas Using HMM-Based Behavioral Distance," IEEE Transactions on Dependable and Secure Computing, vol. 6, no. 2, pp. 96-110, April-June 2009, doi:10.1109/TDSC.2008.39
Usage of this product signifies your acceptance of the Terms of Use.