This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
A Method for Detecting Obfuscated Calls in Malicious Binaries
November 2005 (vol. 31 no. 11)
pp. 955-968
ASCII Text
x 
 
Arun Lakhotia, Eric Uday Kumar, Michael Venable, "A Method for Detecting Obfuscated Calls in Malicious Binaries," IEEE Transactions on Software Engineering, vol. 31, no. 11, pp. 955-968, November, 2005.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2005.120,
author = {Arun Lakhotia and Eric Uday Kumar and Michael Venable},
title = {A Method for Detecting Obfuscated Calls in Malicious Binaries},
journal ={IEEE Transactions on Software Engineering},
volume = {31},
number = {11},
issn = {0098-5589},
year = {2005},
pages = {955-968},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2005.120},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - A Method for Detecting Obfuscated Calls in Malicious Binaries
IS - 11
SN - 0098-5589
SP955
EP968
EPD - 955-968
A1 - Arun Lakhotia,
A1 - Eric Uday Kumar,
A1 - Michael Venable,
PY - 2005
KW - Index Terms- Invasive software (viruses
KW - worms)
KW - program analysis
KW - validation
KW - obfuscation
KW - abstract stack.
VL - 31
JA - IEEE Transactions on Software Engineering
ER -
 
Information about calls to the operating system (or kernel libraries) made by a binary executable may be used to determine whether the binary is malicious. Being aware of this approach, malicious programmers hide this information by making such calls without using the call instruction. For instance, the call addr instruction may be replaced by two push instructions and a ret instruction, the first push pushes the address of instruction after the ret instruction, and the second push pushes the address addr. The code may be further obfuscated by spreading the three instructions and by splitting each instruction into multiple instructions. This work presents a method to statically detect obfuscated calls in binary code. The idea is to use abstract interpretation to detect where the normal call-ret convention is violated. These violations can be detected by what is called an abstract stack graph. An abstract stack graph is a concise representation of all potential abstract stacks at every point in a program. An abstract stack is used to associate each element in the stack to the instruction that pushes the element. An algorithm for constructing the abstract stack graph is also presented. Methods for using the abstract stack graph are shown to detect eight different obfuscations. The technique is demonstrated by implementing a prototype tool called DOC (Detector for Obfuscated Calls).

[1] “TESO, Burneye Elf Encryption Program,” https:/teso.scene.at, Nov. 2004.
[2] “z0mbie,” http:/z0mbie.host.sk, Nov. 2004.
[3] G. Balakrishnan and T. Reps, “Analyzing Memory Accesses in X86 Executables,” Proc. Int'l Conf. Compiler Construction (CC), 2004.
[4] B. Barak, O. Goldreich, R. Impagliazzo, S. Rudich, A. Sahai, S. Vadhan, and K. Yang, “On the (Im)Possibility of Obfuscating Programs,” Proc. Conf. Advances in Cryptology (CRYPTO '01), 2001.
[5] D. Chess and S. White, “An Undetectable Computer Virus,” Proc. Virus Bulletin Conf., 2000.
[6] S. Cho, “Win32 Disassembler,” http://www.geocities.com/~sangchodisasm.html , Nov. 2004.
[7] M. Christodorescu, S. Jha, S.A. Seshia, D. Song, and R.E. Bryant, “Semantics-Aware Malware Detection,” Proc. IEEE Symp. Security and Privacy, 2005.
[8] M. Christodrescu and S. Jha, “Static Analysis of Executables to Detect Malicious Patterns,” Proc. 12th USENIX Security Symp. (Security '03), 2003.
[9] C. Cifuentes and K.J. Gough, “Decompilation of Binary Programs,” Software Practice and Experience, vol. 25, pp. 811-829, 1995.
[10] F. Cohen, “Computational Aspects of Computer Viruses,” Computers and Security, vol. 8, pp. 325-344, 1989.
[11] C. Collberg and C. Thomborson, “Watermarking, Tamper-Proofing, and Obfuscation— Tools for Software Protection,” IEEE Trans. Software Eng., vol. 28, pp. 735-746, 2002.
[12] C. Collberg, C. Thomborson, and D. Low, “A Taxonomy of Obfuscating Transformations,” technical report, Dept. of Computer Science, The University of Auckland 148, July 1997.
[13] P. Cousot and R. Cousot, “Static Determination of Dynamic Properties of Programs,” Proc. Second Int'l Symp. Programming, 1976.
[14] N.D. Jones and F. Nielson, “Abstract Interpretation: A Semantics-Based Tool for Program Analysis,” Handbook of Logic in Computer Science: Semantic Modelling, vol. 4, pp. 527-636, 1995.
[15] C. Kruegel, W. Robertson, F. Valeur, and G. Vigna, “Static Disassembly of Obfuscated Binaries,” Proc. USENIX Security Conf., 2004.
[16] A. Lakhotia, M.E. Karim, A. Walenstein, and L. Parida, “Phylogeny Using Maximal pi-Patterns,” Proc. 14th EICAR Conf., 2005.
[17] A. Lakhotia and M. Mohammed, “Imposing Order on Program Statements and Its Implication to AV Scanners,” Proc. 11th IEEE Working Conf. Reverse Eng., 2004.
[18] A. Lakhotia and P.K. Singh, “Challenges in Getting Formal with Viruses,” Virus Bull., pp. 14-18, 2003.
[19] C. Linn and S. Debray, “Obfuscation of Executable Code to Improve Resistance to Static Disassembly,” Proc. 10th ACM Conf. Computer and Comm. Security, 2003.
[20] C. Nachenberg, “Computer Virus-Antivirus Coevolution,” Comm. ACM, vol. 40, pp. 46-51, 1997.
[21] D. Schmidt, “Abstract Interpretation and Static Analysis,” http://www.cis.ksu.edu/santos/schmidtEscuela03 /, Feb. 2005.
[22] B. Schwarz, S. Debray, and G. Andrews, “Disassembly of Executable Code Revisited,” Proc. Ninth Working Conf. Reverse Eng. (WCRE '02), 2002.
[23] Symantec, “Understanding Heuristics: Symantec's Bloodhound Technology,” http://www.symantec.com/avcenter/reference heuristc.pdf, July 2004.
[24] P. Szor and P. Ferrie, “Hunting for Metamorphic,” Proc. Virus Bull. Conf., 2001.
[25] M. Venable, M. Chouchane, M.E. Karim, and A. Lakhotia, “Analyzing Memory Accesses in Obfuscated x86 Executables,” Proc. Conf. Detection of Intrusions and Malware and Vulnerability Assessment, 2005.
[26] L. Vinciguerra, L. Wills, N. Kejriwal, P. Martino, and R. Vinciguerra, “An Experimentation Framework for Evaluating Disassembly and Decompilation Tools for C++ and Java,” Proc. 10th Working Conf. Reverse Eng., 2003.
[27] G. Wroblewski, “General Method of Program Code Obfuscation,” technical report, Inst. of Eng. Cybernetics, Wroclaw Univ. of Tech nology, 2002.

Index Terms:
Index Terms- Invasive software (viruses, worms), program analysis, validation, obfuscation, abstract stack.
Citation:
Arun Lakhotia, Eric Uday Kumar, Michael Venable, "A Method for Detecting Obfuscated Calls in Malicious Binaries," IEEE Transactions on Software Engineering, vol. 31, no. 11, pp. 955-968, Nov. 2005, doi:10.1109/TSE.2005.120
Usage of this product signifies your acceptance of the Terms of Use.