This Article 
 Bibliographic References 
 Add to: 
On the General Applicability of Instruction-Set Randomization
July-September 2010 (vol. 7 no. 3)
pp. 255-270
Stephen W. Boyd, SAS Institute Inc., Pittsburgh
Gaurav S. Kc, Google Inc., New York
Michael E. Locasto, George Mason University, Fairfax
Angelos D. Keromytis, Columbia University, New York
Vassilis Prevelakis, AEGIS Research Center in Information Security, Athens
We describe Instruction-Set Randomization (ISR), a general approach for safeguarding systems against any type of code-injection attack. We apply Kerckhoffs' principle to create OS process-specific randomized instruction sets (e.g., machine instructions) of the system executing potentially vulnerable software. An attacker who does not know the key to the randomization algorithm will inject code that is invalid for that (randomized) environment, causing a runtime exception. Our approach is applicable to machine-language programs and scripting and interpreted languages. We discuss three approaches (protection for Intel x86 executables, Perl scripts, and SQL queries), one from each of the above categories. Our goal is to demonstrate the generality and applicability of ISR as a protection mechanism. Our emulator-based prototype demonstrates the feasibility ISR for x86 executables and should be directly usable on a suitably modified processor. We demonstrate how to mitigate the significant performance impact of emulation-based ISR by using several heuristics to limit the scope of randomized (and interpreted) execution to sections of code that may be more susceptible to exploitation. The SQL prototype consists of an SQL query-randomizing proxy that protects against SQL injection attacks with no changes to database servers, minor changes to CGI scripts, and with negligible performance overhead. Similarly, the performance penalty of a randomized Perl interpreter is minimal. Where the performance impact of our proposed approach is acceptable (i.e., in an already-emulated environment, in the presence of programmable or specialized hardware, or in interpreted languages), it can serve as a broad protection mechanism and complement other security mechanisms.

[1] M. Abadi, M. Budiu, U. Erlingsson, and J. Ligatti, "Control-Flow Integrity: Principles, Implementations, and Applications," Proc. 12th ACM Conf. Computer and Comm. Security (CCS '05), Nov. 2005.
[2] Aleph One, "Smashing the Stack for Fun and Profit," Phrack, vol. 7, no. 49, 1996.
[3] C. Anley, Advanced SQL Injection in SQL Server Applications, 2008.
[4] E.G. Barrantes, D.H. Ackley, S. Forrest, T.S. Palmer, D. Stefanovic, and D.D. Zovi, "Randomized Instruction Set Emulation to Disrupt Binary Code Injection Attacks," Proc. 10th ACM Conf. Computer and Comm. Security (CCS '03), pp. 281-289, Oct. 2003.
[5] E.G. Barrantes, D.H. Ackley, S. Forrest, and D. Stefanovic, "Randomized Instruction Set Emulation," ACM Trans. Information and System Security, vol. 8, no. 1, pp. 3-40, Feb. 2005.
[6] S. Bhatkar, D.C. DuVarney, and R. Sekar, "Address Obfuscation: An Efficient Approach to Combat a Broad Range of Memory Error Exploits," Proc. 12th USENIX Security Symp., pp. 105-120, Aug. 2003.
[7] S. Bhatkar, R. Sekar, and D.C. DuVarney, "Efficient Techniques for Comprehensive Protection from Memory Error Exploits," Proc. 14th USENIX Security Symp., pp. 255-270, Aug. 2005.
[8] Bochs Emulator Web Page, http:/, 2008.
[9] D. Bruening, T. Garnett, and S. Amarasinghe, "An Infrastructure for Adaptive Dynamic Optimization," Proc. Symp. Code Generation and Optimization (CGO '03), pp. 265-275, 2003.
[10] CERT Vulnerability Note VU#496064,, Apr. 2002.
[11] CERT Vulnerability Note VU#282403,, Sept. 2002.
[12] S. Chen, J. Xu, N. Nakka, Z. Kalbarczyk, and C. Verbowski, "Defeating Memory Corruption Attacks via Pointer Taintedness Detection," Proc. Int'l Conf. Dependable Systems and Networks (DSN '05), pp. 378-387, June 2005.
[13] S. Chen, J. Xu, E.C. Sezer, P. Gauriar, and R.K. Iyer, "Non-Control-Data Attacks Are Realistic Threats," Proc. 14th USENIX Security Symp., pp. 177-191, Aug. 2005.
[14] M. Costa, J. Crowcroft, M. Castro, and A. Rowstron, "Vigilante: End-to-End Containment of Internet Worms," Proc. 20th Symp. Systems and Operating Systems Principles (SOSP), 2005.
[15] C. Cowan, S. Beattie, J. Johansen, and P. Wagle, "PointGuard: Protecting Pointers from Buffer Overflow Vulnerabilities," Proc. 12th USENIX Security Symp., pp. 91-104, Aug. 2003.
[16] B. Cox, D. Evans, A. Filipi, J. Rowanhill, W. Hu, J. Davidson, J. Knight, A. Nguyen-Tuong, and J. Hiser, "$N$ -Variant Systems: A Secretless Framework for Security through Diversity," Proc. 15th USENIX Security Symp., pp. 105-120, July/Aug. 2005.
[17] G.W. Dunlap, S.T. King, S. Cinar, M.A. Basrai, and P.M. Chen, "ReVirt: Enabling Intrusion Analysis through Virtual-Machine Logging and Replay," Proc. Fifth Symp. Operating Systems Design and Implementation (OSDI '02), Dec. 2002.
[18] D. Evans and D. Larochelle, "Improving Security Using Extensible Lightweight Static Analysis," IEEE Software, vol. 19, no. 1, Jan./Feb. 2002.
[19] L. Garber, "New Chips Stop Buffer Overflow Attacks," Computer, vol. 37, no. 10, p. 28, Oct. 2004.
[20] W.G.J. Halfond and A. Orso, "SQL Command-Form Coverage for Testing Database Applications," Proc. 20th Int'l Conf. Automated Software Eng. (ASE '05), Sept. 2005.
[21] G.S. Kc, A.D. Keromytis, and V. Prevelakis, "Countering Code-Injection Attacks with Instruction-Set Randomization," Proc. 10th ACM Conf. Computer and Comm. Security (CCS '03), Oct. 2003.
[22] L. Lam and T. Chiueh, "Checking Array Bound Violation Using Segmentation Hardware," Proc. Int'l Conf. Dependable Systems and Networks (DSN '05), pp. 388-397, June 2005.
[23] Z. Liang and R. Sekar, "Fast and Automated Generation of Attack Signatures: A Basis for Building Self-Protecting Servers," Proc. 12th ACM Conf. Computer and Comm. Security (CCS '05), pp. 213-222, Nov. 2005.
[24] C. Linn and S. Debray, "Obfuscation of Executable Code to Improve Resistance to Static Disassembly," Proc. 10th ACM Conf. Computer and Comm. Security (CCS '03), pp. 290-299, Oct. 2003.
[25] D. Litchfield, Web Application Disassembly with ODBC Error Messages, , 2008.
[26] M. Locasto, K. Wang, A. Keromytis, and S. Stolfo, "FLIPS: Hybrid Adaptive Intrusion Prevention," Proc. Eighth Symp. Recent Advances in Intrusion Detection (RAID '05), pp. 82-101, Sept. 2005.
[27] M. Conover and w00w00 Security Team, w00w00 on Heap Overflows, , 2008.
[28] J. Newsome and D. Song, "Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software," Proc. 12th Ann. Symp. Network and Distributed System Security (SNDSS '05), Feb. 2005.
[29] PaX Home Page, http:/, 2008.
[30] Perltidy Home Page, http:/, 2008.
[31] T. Pietraszek and C.V. Berghe, "Defending against Injection Attacks through Context-Sensitive String Evaluation," Proc. Eighth Int'l Symp. Recent Advances in Intrusion Detection (RAID '05), Sept. 2005.
[32] J. Pincus and B. Baker, "Beyond Stack Smashing: Recent Advances in Exploiting Buffer Overflows," IEEE Security and Privacy Magazine, vol. 2, no. 4, pp. 20-27, July/Aug. 2004.
[33] V. Prevelakis and A.D. Keromytis, "Drop-in Security for Distributed and Portable Computing Elements," Internet Research: Electronic Networking, Applications and Policy, vol. 13, no. 2, 2003.
[34] B. Rogers, Y. Solihin, and M. Prvulovic, "Memory Predecryption: Hiding the Latency Overhead of Memory Encryption," Proc. Workshop Architectural Support for Security and Anti-Virus (WASSA '04), pp. 22-28, Oct. 2004.
[35] H. Shacham, M. Page, B. Pfaff, E. Goh, N. Modadugu, and D. Boneh, "On the Effectiveness of Address-Space Randomization," Proc. 11th ACM Conf. Computer and Comm. Security (CCS '04), pp. 298-307, Oct. 2004.
[36] S. Sidiroglou and A.D. Keromytis, "A Network Worm Vaccine Architecture," Proc. IEEE Int'l Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE '03), Workshop Enterprise Security, pp. 220-225, June 2003.
[37] A. Smirnov and T. Chiueh, "DIRA: Automatic Detection, Identification, and Repair of Control-Hijacking Attacks," Proc. ISOC Symp. Network and Distributed System Security (SNDSS '05), Feb. 2005.
[38] A.N. Sovarel, D. Evans, and N. Paul, "Where's the FEEB? The Effectiveness of Instruction Set Randomization," Proc. 14th USENIX Security Symp., pp. 145-160, Aug. 2005.
[39] G.E. Suh, J.W. Lee, D. Zhang, and S. Devadas, "Secure Program Execution via Dynamic Information Flow Tracking," SIGOPS Operating Systems Rev., vol. 38, no. 5, pp. 85-96, 2004.
[40] N. Tuck, B. Calder, and G. Varghese, "Hardware and Binary Modification Support for Code Pointer Protection from Buffer Overflow," Proc. 37th Int'l Symp. Microarchitecture (MICRO '04), pp. 209-220, Dec. 2004.
[41] D. Wagner, J.S. Foster, E.A. Brewer, and A. Aiken, "A First Step towards Automated Detection of Buffer Overrun Vulnerabilities," Proc. ISOC Symp. Network and Distributed System Security (SNDSS '00), pp. 3-17, Feb. 2000.
[42] A. Whitaker, M. Shaw, and S.D. Gribble, "Scale and Performance in the Denali Isolation Kernel," Proc. Fifth Symp. Operating Systems Design and Implementation (OSDI '02), Dec. 2002.
[43] J. Xu, "Intrusion Prevention Using Control Data Randomization," Proc. IEEE Int'l Conf. Dependable Systems and Networks (DSN '03), June 2003.
[44] J. Xu, Z. Kalbarczyk, and R.K. Iyer, "Transparent Runtime Randomization for Security," Proc. 22nd Int'l Symp. Reliable Distributed Systems (SRDS '03), pp. 260-273, Oct. 2003.
[45] J. Xu, P. Ning, C. Kil, Y. Zhai, and C. Bookholt, "Automatic Diagnosis and Response to Memory Corruption Vulnerabilities," Proc. 12th ACM Conf. Computer and Comm. Security (CCS '05), pp. 222-234, Nov. 2005.
[46] W. Xu, S. Bhatkar, and R. Sekar, "Taint-Enhanced Policy Enforcement: A Practical Approach to Defeat a Wide Range of Attacks," Proc. USENIX Security Symp., pp. 121-136, July/Aug. 2006.
[47] D. Ye and D. Kaeli, "A Reliable Return Address Stack: Microarchitectural Features to Defeat Stack Smashing," Proc. Workshop Architectural Support for Security and Anti-Virus (WASSA '04), pp. 69-76, Oct. 2004.

Index Terms:
Interpreters, emulators, buffer overflows, SQL injection, randomization, security, performance.
Stephen W. Boyd, Gaurav S. Kc, Michael E. Locasto, Angelos D. Keromytis, Vassilis Prevelakis, "On the General Applicability of Instruction-Set Randomization," IEEE Transactions on Dependable and Secure Computing, vol. 7, no. 3, pp. 255-270, July-Sept. 2010, doi:10.1109/TDSC.2008.58
Usage of this product signifies your acceptance of the Terms of Use.