This Article 
 Bibliographic References 
 Add to: 
A Native APIs Protection Mechanism in the Kernel Mode against Malicious Code
June 2011 (vol. 60 no. 6)
pp. 813-823
Hung-Min Sun, National Tsing Hua University, Hsinchu
Hsun Wang, National Tsing Hua University, Hsinchu
King-Hang Wang, National Tsing Hua University, Hsinchu
Chien-Ming Chen, National Tsing Hua University, Hsinchu
As new vulnerabilities on Windows systems are reported endlessly, it is more practical to stop polymorphic malicious code from exploiting these vulnerabilities by building an behavior-based monitor, rather than adopting a signature-based detection system or fixing these vulnerabilities. Many behavior-based monitors have been proposed for Windows systems to serve this purpose. Some of them hook high-level system APIs to detect the suspicious behaviors of code. However, they cannot detect malicious code that directly invokes Native APIs. In this paper, we present a novel security scheme that hooks Native APIs in the kernel mode. This method effectively prevents malicious code calling Native APIs directly. It introduces an average eight percent computation overhead into the system. Analyses and a series of experiments are given in the paper to support our claims.

[1] C. Cowan, C. Pu, D. Maier, H. Hintony, J. Walpole, P. Bakke, S. Beattie, A. Grier, P. Wagle, and Q. Zhang, "StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks," Proc. Seventh Conf. USENIX Security Symp. (SSYM '98), p. 5, 1998.
[2] P. Akritidis, E. Markatos, M. Polychronakis, and K. Anagnostakis, "STRIDE: Polymorphic Sled Detection through Instruction Sequence Analysis," Proc. 20th IFIP Int'l Information Security Conf. Security and Privacy in the Age of Ubiquitous Computing, 2005.
[3] T. Toth and C. Kruegel, "Accurate Buffer Overflow Detection via Abstract Payload Execution," Proc. Fifth Int'l Symp. Recent Advances in Intrusion Detection (RAID '02), 2002.
[4] U. Payer, P. Teufl, and M. Lamberger, "Hybrid Engine for Polymorphic Shellcode Detection," Proc. Second Int'l Conf. Intrusion and Malware Detection and Vulnerability Assessment (DIMVA '05), 2005.
[5] H. Sun, Y. Lin, and M. Wu, "API Monitoring System for Defeating Worms and Exploits in MS-Windows System," Proc. 11th Australasian Conf. Information Security and Privacy, pp. 159-170, 2006.
[6] J. Rabek, R. Khazan, S. Lewandowski, and R. Cunningham, "Detection of Injected, Dynamically Generated, and Obfuscated Malicious Code," Proc. ACM Workshop Rapid Malcode (WORM '03), pp. 76-82, 2003.
[7] Y. Ye, D. Wang, T. Li, and D. Ye, "IMDS: Intelligent Malware Detection System," Proc. 13th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '07), pp. 1043-1047, 2007.
[8] D. Wagner and P. Soto, "Mimicry Attacks on Host-Based Intrusion Detection Systems," Proc. Ninth ACM Conf. Computer and Comm. Security, pp. 255-264, 2002.
[9] M. Rajagopalan, M. Hiltunen, T. Jim, and R. Schlichting, "Authenticated System Calls," Proc. Int'l Conf. Dependable Systems and Networks (DSN '05), 2005.
[10] C.M. Linn, M. Rajagopalan, S. Baker, C. Collberg, S.K. Debray, and J.H. Hartman, "Protecting against Unexpected System Calls," Proc. 14th Conf. USENIX Security Symp. (SSYM '05), p. 16, 2005.
[11] M. Rajagopalan, S. Baker, C. Linn, S. Debray, R. Schlichting, and J. Hartman, "Signed System Calls and Hidden Fingerprints," Technical Report TR04-15, Dept. of Computer Science, The Univ. of Arizona, May 2004.
[12] M. Wang, C. Zhang, and J. Yu, "Native API Based Windows Anomaly Intrusion Detection Method Using SVM," Proc. IEEE Int'l Conf. Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC '06), pp. 514-519, June 2006.
[13] R. Battistoni, E. Gabrielli, and L.V. Mancini, "A Host Intrusion Prevention System for Windows Operating Systems," Proc. Ninth European Symp. Research in Computer Security (ESORICS '04), pp. 352-368, Sept. 2004.
[14] L. Nguyen, T. Demir, J. Rowe, F. Hsu, and K. Levitt, "A Framework for Diversifying Windows Native APIs to Tolerate Code Injection Attacks," Proc. Second ACM Symp. Information, Computer and Comm. Security (ASIACCS '07), pp. 392-394, 2007.
[15] Vendicator, "Stack Shield,", Jan. 2000.
[16] C. Cowan, S. Beattie, J. Johansen, and P. Wagle, "PointGuard: Protecting Pointers from Buffer Overflow Vulnerabilities," Proc. 12th Conf. USENIX Security Symp. (SSYM '03), p. 7, 2003.
[17] C. Cowan, M. Barringer, S. Beattie, G. Kroah-Hartman, M. Frantzen, and J. Lokier, "FormatGuard: Automatic Protection from Printf Format String Vulnerabilities," Proc. 10th Conf. USENIX Security Symp., p. 15, 2001.
[18] T. Chiueh and F. Hsu, "RAD: A Compile-Time Solution to Buffer Overflow Attacks," Proc. 21st Int'l Conf. Distributed Computing Systems, pp. 409-417, 2001.
[19] D. Larochelle and D. Evans, "Statically Detecting Likely Buffer Overflow Vulnerabilities," Proc. 10th Conf. USENIX Security Symp., 2001.
[20] G. Hunt and D. Brubacher, "Detours: Binary Interception of Win32 Functions," Proc. Third Conf. USENIX Windows NT Symp., July 1999.
[21] "Windows Driver Kit," devtools WDK, 2011.
[22] "OSR Driver Loader," http:/, 2011.
[23] H. Shacham, M. Page, B. Pfaff, E.-J. Goh, N. Modadugu, and D. Boneh, "On the Effectiveness of Address-Space Randomization," Proc. ACM Conf. Computer and Comm. Security, http://dblp. uni- , pp. 298-307, 2004.

Index Terms:
API hooking, Windows API, code injection.
Hung-Min Sun, Hsun Wang, King-Hang Wang, Chien-Ming Chen, "A Native APIs Protection Mechanism in the Kernel Mode against Malicious Code," IEEE Transactions on Computers, vol. 60, no. 6, pp. 813-823, June 2011, doi:10.1109/TC.2011.46
Usage of this product signifies your acceptance of the Terms of Use.