This Article 
 Bibliographic References 
 Add to: 
SmashGuard: A Hardware Solution to Prevent Security Attacks on the Function Return Address
October 2006 (vol. 55 no. 10)
pp. 1271-1285
A buffer overflow attack is perhaps the most common attack used to compromise the security of a host. This attack can be used to change the function return address and redirect execution to the attacker's code. We present a hardware-based solution, called SmashGuard, to protect against all known forms of attack on the function return addresses stored on the program stack. With each function call instruction, the current return address is pushed onto a hardware stack. A return instruction compares its address to the return address from the top of the hardware stack. An exception is raised to signal the mismatch. Because the stack operations and checks are done in hardware in parallel with the usual execution of instructions, our best-performing implementation scheme has virtually no performance overhead (because we are modifying hardware, it is impossible to guarantee zero overhead without an actual hardware implementation). While previous software-based approaches' average performance degradation for the SPEC2000 benchmarks is only 2.8 percent, their worst-case degradation is up to 8.3 percent. Apart from the lack of robustness in performance, the software approaches' key disadvantages are less security coverage and the need for recompilation of applications. SmashGuard, on the other hand, is secure and does not require recompilation of applications.

[1] AMD, “AMD Chips Include New Buffer Overflow Protection,” http://www.computerweekly.comArticle127571.htm , 2004.
[2] Intel, “Execute Disable (XD) Bit,” /bss/infrastructure/ securityxdbit.htm, 2001.
[3] T. Corporation, “AntiVirusNX Technology,” http://www.trans , 2004.
[4] Microsoft, “Microsoft Windows XP SP2 Data Execution Prevention,” winxp pro/maintainsp2mempr.mspx, 2004.
[5] Aleph1, “Smashing the Stack for Fun and Profit,” Phrack Magazine, vol. 7, no. 49, Nov. 1996, http://www.phrack.orgshow. php?p=49&a=14 .
[6] CERT Coordination Center, “CERT Incident Note IN-2001-08 Code Red Worm Exploiting Buffer Overflow in IIS Indexing Service DLL,” html , June 2001.
[7] CERT Coordination Center, “CERT Incident Note IN-2001-09 Code Red II: Another Worm Exploiting Buffer Overflow In IIS Indexing Service DLL,” IN-2001-09.html , Aug. 2001.
[8] CERT Coordination Center, “CERT Advisory CA-2003-20 W32/Blaster Worm,” , Aug. 2003.
[9] Sophos Virus Analysis, “W32/Nachi-A,” http://www.sophos. com/virusinfo/analyses w32nachia.html, Aug. 2003.
[10] Sophos Virus Analysis, “W32/Sasser,” AD20040501.html, May 2004.
[11] CERT Coordination Center, “CERT Advisory CA-2001-13 Buffer Overflow in IIS Indexing Service DLL,” , June 2001.
[12] CERT Coordination Center, “CERT Vulnerability Note VU 568148 Microsoft Windows RPC Vulnerable to Buffer Overflow,”, July 2003.
[13] CERT Coordination Center, “CERT Coordination Center Advisories for 2002,”, 2002.
[14] SANS Institute, “SANS/FBI Top 20 List, the Twenty Most Critical Internet Security Vulnerabilities,”, 2002.
[15] CERT Coordination Center, “CERT Coordination Center Advisories for 2003,”, 2003.
[16] SANS Institute, “SANS Top 20 List, The Twenty Most Critical Internet Security Vulnerabilities,” http://www.sans.orgtop20/, 2003.
[17] Scut, “Format String Vulnerabilities,” formatstring , Sept. 2001.
[18] T. Newsham, “Format String Attacks,” newshamformat-string-attacks.pdf , Sept. 2000.
[19] Blexim, “Basic Integer Overflows,” Phrack Magazine, vol. 11, no. 60, Dec. 2002, http://www.phrack.orgshow.php? p=60&a=10 .
[20] S. Designer, “Linux Kernel Patch from the Openwall Project: Non-Executable User Stack,”, Jan. 2001.
[21] The SmashGuard Group, SmashGuard Web Site, http:/, 2003.
[22] J. Wilander and M. Kamkar, “A Comparison of Publicly Available Tools for Static Intrusion Prevention,” Proc. Seventh Nordic Workshop Secure IT Systems, pp. 68-84, Nov. 2002.
[23] D. Wagner, J.S. Foster, E.A. Brewer, and A. Aiken, “A First Step Towards Automated Detection of Buffer Overrun Vulnerabilities,” Proc. Network and Distributed System Security Symp., pp. 3-7, Feb. 2000.
[24] N. Dor, M. Rodeh, and M. Sagiv, “CSSV: Towards a Realistic Tool for Statically Detecting All Buffer Overflows in C,” Proc. ACM SIGPLAN 2003 Conf. Programming Language Design and Implementation, pp. 155-167, June 2003.
[25] D. Larochelle and D. Evans, “Statically Detecting Likely Buffer Overflow Vulnerabilities,” Proc. 10th USENIX Security Symp., pp. 177-190, Aug. 2001.
[26] E. Haugh and M. Bishop, “Testing C Programs for Buffer Overflow Vulnerabilities,” Proc. Network and Distributed System Security Symp., 2003, .
[27] S.H. Yong and S. Horwitz, “Protecting C Programs from Attacks via Invalid Pointer Dereferences,” Proc. Ninth European Software Eng. Conf. held Jointly with 10th ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 307-316, Sept. 2003.
[28] T. Toth and C. Kruegel, “Accurate Buffer Overflow Detection via Abstract Payload Execution,” Proc. Fifth Int'l Symp. Recent Advances in Intrusion Detection, 2002, http://www.infosys.
[29] S. Bhatkar, D.C. DuVarney, and R. Sekar, “Address Obfuscation: An Efficient Approach to Combat a Broad Range of Memory Error Exploits,” Proc. 12th USENIX Security Symp., pp. 105-120, Aug. 2003.
[30] The PaX Team, PaX, http:/, 2001.
[31] M. Prasad and T. Chiueh, “A Binary Rewriting Defense against Stack Based Buffer Overflow Attacks,” Proc. Usenix Ann. Technical Conf., General Track, pp. 211-224, June 2003.
[32] Microsoft, “Visual C++ Option to Tighten Security,” 2001/mh0301mh0301-1.asp, 2001.
[33] C. Cowan, C. Pu, D. Maier, H. Hinton, P. Bakke, S. Beattie, A. Grier, P. Wagle, and Q. Zhang, “StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks,” Proc. Seventh USENIX Security Conf., pp. 63-78, Jan. 1998.
[34] C. Cowan, S. Beattie, R.F. Day, C. Pu, P. Wagle, and E. Walthinsen, “Protecting Systems from Stack Smashing Attacks with StackGuard,” Proc. Fifth Linux Expo, May 1999, http://www.cse.ogi. edu/DISC/projects/immunix
[35] Bulba and Kil3r, “Bypassing StackGuard and StackShield,” Phrack Magazine, vol. 10, no. 56, May 2000, http://www.phrack.orgshow.php?p=56&a=5 .
[36] Vendicator, “StackShield: A `Stack Smashing' Technique Protection Tool for Linux,” , Jan. 2001.
[37] T. Chiueh and F. Hsu, “RAD: A Compile-Time Solution to Buffer Overflow Attacks,” Proc. 21st Int'l Conf. Distributed Computing Systems (ICDCS '01), pp. 409-417, Apr. 2001.
[38] H. Etoh, “GCC Extension for Protecting Applications from Stack-Smashing Attacks,” IBM Research, ssp/, Apr. 2003.
[39] The OpenBSD Project, http:/, Apr. 2003.
[40] C. Cowan, S. Beattie, J. Johansen, and P. Wagle, “Pointguard: Protecting Pointers from Buffer Overflow Vulnerabilities,” Proc. 12th USENIX Security Symp., pp. 91-104, Aug. 2003.
[41] Various, “OpenSSL,” http:/, 2004.
[42] O. Ruwase and M.S. Lam, “A Practical Dynamic Buffer Overflow Detector,” Proc. 11th Ann. Network and Distributed System Security Symp. (NDSS '04), pp. 159-169, Feb. 2004.
[43] A. Snarskii, “FreeBSD Stack Integrity Patch,” , 1997.
[44] A. Snarskii, “Libparanoia,” noia/, Apr. 2000.
[45] A. Baratloo, T.K. Tsai, and N. Singh, “Libsafe: Protecting Critical Elements of Stacks,” technical report, Bell Labs, Lucent Technologies, Murray Hill, N.J., Dec. 1999, .
[46] A. Baratloo, N. Singh, and T. Tsai, “Transparent Run-Time Defense against Stack Smashing Attacks,” Proc. USENIX Ann. Technical Conf., pp. 251-262, June 2000.
[47] T. Tsai and N. Singh, “Libsafe 2.0: Detection of Format String Vulnerability Exploits,” Technical Report ALR-2001-019, Avaya Labs, Avaya Inc., Basking Ridge, N.J., Aug. 2001, http://www. ALR-2001-019-paper.pdf.
[48] C. Cowan, M. Barringer, S. Beattie, G. Kroah-Hartman, M. Frantzen, and J. Lokier, “FormatGuard: Automatic Protection from Print Format String Vulnerabilities,” Proc 2001 USENIX Security Conf., pp. 191-200, Aug. 2001.
[49] M. Frantzen and M. Shuey, “StackGhost: Hardware Facilitated Stack Protection,” Proc. 10th USENIX Security Symp., pp. 55-66, Aug. 2001.
[50] L. Torvalds, “Reply to Non-Executable Stack Patch,” http://old. , Aug. 1998.
[51] GNU Compiler Collection Internals, docs/gccintTrampolines.html , 2004.
[52] The OpenBSD 3.3, http://www.openbsd.org33.html, Apr. 2003.
[53] J. Xu, Z. Kalbarczyk, S. Patel, and R.K. Iyer, “Architecture Support for Defending against Buffer Overflow Attacks,” Proc. Workshop Evaluating and Architecting System Dependability (EASY-2002), Oct. 2002.
[54] R.B. Lee, D.K. Karig, J.P. McGregor, and Z. Shi, “Enlisting Hardware Architecture to Thwart Malicious Code Injecttion,” Proc. Int'l Conf. Security in Pervasive Computing (SPC-2003), Mar. 2003.
[55] T. Jim, G. Morrisett, D. Grossman, M. Hicks, J. Cheney, and Y. Wang, “Cyclone: A Safe Dialect of C,” Proc. 2002 USENIX Ann. Technical Conf., pp. 275-288, June 2002.
[56] T. Austin, S. Breach, and G. Sohi, “Safe C Compiler (SCC),”, June 1994.
[57] G.C. Necula, S. McPeak, and W. Weimer, “CCured: Type-Safe Retrofitting of Legacy Code,” Proc. ACM Symp. Principles of Programming Languages, pp. 128-139, Jan. 2002.
[58] D.M. Tullsen, S.J. Eggers, and H.M. Levy, “Simultaneous Multithreading: Maximizing On-Chip Parallelism,” Proc. 22nd Ann. Int'l Symp. Computer Architecture, pp. 392-403, June 1995.
[59] T. Austin, “SimpleScalar LLC,” http:/, 2001.
[60] CERT Coordination Center, CERT Coordination Center Statistics 1988-2002, , 2004.
[61] CERT Coordination Center, CERT Coordination Center Incident and Vulnerability Trends, /, 2003.

Index Terms:
Buffer overflow, function return address, hardware stack.
Hilmi ?zdoganoglu, T.N. Vijaykumar, Carla E. Brodley, Benjamin A. Kuperman, Ankit Jalote, "SmashGuard: A Hardware Solution to Prevent Security Attacks on the Function Return Address," IEEE Transactions on Computers, vol. 55, no. 10, pp. 1271-1285, Oct. 2006, doi:10.1109/TC.2006.166
Usage of this product signifies your acceptance of the Terms of Use.