The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.05 - September/October (2011 vol.8)
pp: 670-684
Arati Baliga , Rutgers University, Piscataway
Vinod Ganapathy , Rutgers University, Piscataway
Liviu Iftode , Rutgers University, Piscataway
ABSTRACT
Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that maliciously modify noncontrol data. Most prior techniques for rootkit detection have focused solely on detecting control data modifications and, therefore, fail to detect such rootkits. This paper presents a novel technique to detect rootkits that modify both control and noncontrol data. The main idea is to externally observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A rootkit detection phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show that Gibraltar can effectively detect previously known rootkits, including those that modify noncontrol data structures.
INDEX TERMS
Kernel-level rootkits, noncontrol data attacks, invariant inference, static and dynamic program analysis.
CITATION
Arati Baliga, Vinod Ganapathy, Liviu Iftode, "Detecting Kernel-Level Rootkits Using Data Structure Invariants", IEEE Transactions on Dependable and Secure Computing, vol.8, no. 5, pp. 670-684, September/October 2011, doi:10.1109/TDSC.2010.38
REFERENCES
[1] Chkrootkit: Locally checks for rootkits, http:/www.chkrootkit. org, 2010.
[2] F-Secure Rootkit Information Pages: Fu Rootkit, http://www. f-secure.com/v-descsfu.shtml , 2010.
[3] Myricom: Pioneering High Performance Computing, http:/www.myri.com, 2010.
[4] Packet storm, http://packetstormsecurity.org/UNIX/ penetration rootkits, 2010.
[5] "Rootkits, Part 1 of 3: A Growing Threat," white paper, MacAfee AVERT Labs, Apr. 2006.
[6] "2010 Threat Predictions," white paper, MacAfee AVERT Labs, Dec. 2009.
[7] A. Baliga, X. Chen, and L. Iftode, "Automated Containment of Rootkit Attacks," Computers and Security J., vol. 27, pp. 323-334, 2008.
[8] A. Baliga, V. Ganapathy, and L. Iftode, "Automatic Inference and Enforcement of Kernel Data Structure Invariants," Proc. Ann. Computer Security Applications Conf., 2008.
[9] A. Baliga, P. Kamat, and L. Iftode, "Lurking in the Shadows: Identifying Systemic Threats to Kernel Data," Proc. IEEE Symp. Security and Privacy, 2007.
[10] D. Beck, B. Vo, and C. Verbowski, "Detecting Stealth Software with Strider GhostBuster," Proc. Int'l Conf. Dependable Systems and Networks, 2005.
[11] K. Butler, S. McLaughlin, and P. McDaniel, "Rootkit-Resistant Disks," Proc. ACM Conf. Computer and Comm. Security, 2008.
[12] M. Carbone, W. Cui, L. Lu, W. Lee, M. Peinado, and X. Jiang, "Mapping Kernel Objects to Enable Systematic Integrity Checking," Proc. ACM Conf. Computer and Comm. Security, 2009.
[13] T. Chilimbi and V. Ganapathy, "HeapMD: Identifying Heap-Based Bugs Using Anomaly Detection," Proc. Int'l Conf. Architectural Support for Programming Languages and Operating Systems, 2006.
[14] B. Cogswell and M. Russinovich, "RootkitRevealer v1.71," Rootkit detection tool by Microsoft, 2006.
[15] A. Cozzie, F. Stratton, H. Xue, and S. King, "Digging for Data Structures," Proc. ACM/USENIX Symp. Operating Systems Design and Implementation, 2008.
[16] C. Csallner and Y. Smaragdakis, "DSD-Crasher: A Hybrid Analysis Tool for Bug Finding," Proc. Int'l Symp. Software Testing and Analysis, 2006.
[17] B. Demsky, M. Ernst, P. Guo, S. McCamant, J. Perkins, and M. Rinard, "Inference and Enforcement of Data Structure Consistency Specifications," Proc. Int'l Symp. Software Testing and Analysis, 2006.
[18] M.D. Ernst, W.G. Griswold, Y. Kataoka, and D. Notkin, "Dynamically Discovering Program Invariants Involving Collections," Technical Report UW-CSE-99-11-02, Univ. of Washington, 2000.
[19] M.D. Ernst, J.H. Perkins, P.J. Guo, S. McCamant, C. Pacheco, M.S. Tschantz, and C. Xiao, "The Daikon System for Dynamic Detection of Likely Invariants," Science of Computer Programming, vol. 69, pp. 35-45, 2007.
[20] T. Fraser, M.R. Evenson, and W.A. Arbaugh, "VICI: Virtual Machine Introspection for Cognitive Immunity," Proc. Ann. Computer Security Applications Conf., 2008.
[21] T. Garfinkel, B. Pfaff, J. Chow, M. Rosenblum, and D. Boneh, "Terra: A Virtual Machine-Based Platform for Trusted Computing," Proc. ACM Symp. Operating System Principles, 2003.
[22] T. Garfinkel and M. Rosenblum, "A Virtual Machine Introspection Based Architecture for Intrusion Detection," Proc. Network and Distributed Systems Security Symp., 2003.
[23] N. Gupta, "Generating Test Data for Dynamically Discovering Likely Program Invariants," Proc. Int'l Workshop Dynamic Analysis, 2003.
[24] S. Hangal and M.S. Lam, "Tracking Down Software Bugs Using Automatic Anomaly Detection," Proc. Int'l Conf. Software Eng., 2002.
[25] G.H. Kim and E.H. Spafford, "The Design and Implementation of Tripwire: A File System Integrity Checker," Proc. ACM Conf. Computer and Comm. Security, 1994.
[26] C. Kruegel, W. Robertson, and G. Vigna, "Detecting Kernel-Level Rootkits through Binary Analysis," Proc. Ann. Computer Security Applications Conf., 2004.
[27] L. Litty and D. Lie, "Manitou: A Layer-Below Approach to Fighting Malware," Proc. Architectural and System Support for Improving Software Dependability, 2006.
[28] J.D. McCalpin, "Memory Bandwidth and Machine Balance in Current High Performance Computers," Proc. IEEE Technical Committee on Computer Architecture, 1995.
[29] L. McVoy and C. Staelin, "LMbench: Portable Tools for Performance Analysis," Proc. USENIX Ann. Technical Conf., 1996.
[30] G.C. Necula, S. McPeak, S. Rahul, and W. Weimer, "CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs," Proc. Int' Conf. Compiler Construction, 2002.
[31] W. Norcott, "IOzone Filesystem Benchmark," http:/www. iozone.org, 2001.
[32] J. Perkins, S. Kim, S. Larsen, S. Amarasinghe, J. Bachrach, M. Carbin, C. Pacheco, F. Sherwood, S. Sidiroglou, G. Sullivan, W. Wong, Y. Zibin, M.D. Ernst, and M. Rinard, "Automatically Patching Errors in Deployed Software," Proc. ACM Symp. Operating System Principles, 2009.
[33] N.L. Petroni, T. Fraser, J. Molina, and W.A. Arbaugh, "Copilot: A Coprocessor-Based Kernel Runtime Integrity Monitor," Proc. USENIX Security Symp., 2004.
[34] N.L. Petroni, T. Fraser, A. Walters, and W.A. Arbaugh, "An Architecture for Specification-Based Detection of Semantic Integrity Violations in Kernel Dynamic Data," Proc. USENIX Security Symp., 2006.
[35] N.L. Petroni and M. Hicks, "Automated Detection of Persistent Kernel Control-Flow Attacks," Proc. ACM Conf. Computer and Comm. Security, 2007.
[36] R. Riley, X. Jiang, and D. Xu, "Guest-Transparent Prevention of Kernel Rootkits with VMM-Based Memory Shadowing," Proc. Int'l Symp. Recent Advances in Intrusion Detection, 2008.
[37] J. Rutkowska, "Beyond the CPU: Defeating Hardware Based RAM Acquisition, Part I: AMD Case," Proc. Blackhat Conf., 2007.
[38] R. Sailer, T. Jaeger, X. Zhang, and L. van Doorn, "Attestation-Based Policy Enforcement for Remote Access," Proc. ACM Conf. Computer and Comm. Security, 2004.
[39] R. Sailer, X. Zhang, T. Jaeger, and L. van Doorn, "Design and Implementation of a TCG-Based Integrity Measurement Architecture," Proc. USENIX Security Symp., 2004.
[40] S.D. and Devik, "Linux On-the-Fly Kernel Patching without LKM: SucKIT Rootkit," Phrack Magazine, vol. 11, no. 58, Dec. 2001.
[41] E. Shi, A. Perrig, and L. van Doorn, "Bind: A Fine-Grained Attestation Service for Secure Distributed Systems," Proc. IEEE Symp. Security and Privacy, 2005.
[42] S. Sidiroglou, O. Laadan, N. Viennot, C. Perez, A. Keromytis, and J. Neih, "ASSURE: Automatic Software Self-Healing Using Rescue Points," Proc. Int'l Conf. Architectural Support for Programming Languages and Operating Systems, 2009.
[43] S. Sparks and J. Butler, "Shadow Walker," Phrack Magazine, no. 63, Jan. 2005.
[44] Shellcode Security Research Team, "Registration Weakness in Linux Kernel's Binary Formats: Polluting sys_execve in Kernel Space without Depending on the sys_call_table," http:// goodfellas.shellcode.com.ar/ ownbinfmt-en.pdf, 2006.
[45] Y. Wang, R. Roussev, C. Verbowski, A. Johnson, M. Wu, Y. Huang, and S. Kuo, "Gatekeeper: Monitoring Auto-Start Extensibility Points (ASEPs) for Spyware Management," Proc. USENIX Conf. System Administration, 2004.
[46] Z. Wang, X. Jiang, W. Cui, and P. Ning, "Countering Kernel Rootkits with Lightweight Hook Protection," Proc. ACM Conf. Computer and Comm. Security, 2009.
[47] Z. Wang, X. Jiang, W. Cui, and X. Wang, "Countering Persistent Kernel Rootkits through Systematic Hook Discovery," Proc. Int'l Symp. Recent Advances in Intrusion Detection, 2008.
[48] J. Wei, B. Payne, J. Giffin, and C. Pu, "Soft-Timer Driven Transient Kernel Control Flow Attacks and Defense," Proc. Ann. Computer Security Applications Conf., 2008.
[49] J. Wilhelm and T. Chiueh, "A Forced Sampled Execution Approach to Kernel Rootkit Identification," Proc. Int'l Symp. Recent Advances in Intrusion Detection, 2007.
[50] H. Yin, Z. Liang, and D. Song, "HookFinder: Identifying and Understanding Malware Hooking Behaviors," Proc. Network and Distributed System Security Symp., 2008.
[51] X. Zhang, L. van Doorn, T. Jaeger, R. Perez, and R. Sailer, "Secure Coprocessor-Based Intrusion Detection," Proc. 10th ACM SIGOPS European Workshop: Beyond the PC, 2002.
5 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool