| | This Article | |
| |
| |
| | Share | |
| |
| |
| | Bibliographic References | |
| |
| |
| | Add to: | |
| |
 Digg
 Furl
 Spurl
 Blink
 Simpy
 Google
 Del.icio.us
 Y!MyWeb
| |
| | Search | |
| |
| |
| | |
System Call Monitoring Using Authenticated System Calls
July-September 2006 (vol. 3 no. 3)
pp. 216-229
System call monitoring is a technique for detecting and controlling compromised applications by checking at runtime that each system call conforms to a policy that specifies the program's normal behavior. Here, we introduce a new approach to implementing system call monitoring based on authenticated system calls. An authenticated system call is a system call augmented with extra arguments that specify the policy for that call, and a cryptographic message authentication code that guarantees the integrity of the policy and the system call arguments. This extra information is used by the kernel to verify the system call. The version of the application in which regular system calls have been replaced by authenticated calls is generated automatically by an installer program that reads the application binary, uses static analysis to generate policies, and then rewrites the binary with the authenticated calls. This paper presents the approach, describes a prototype implementation based on Linux and the Plto binary rewriting system, and gives experimental results suggesting that the approach is effective in protecting against compromised applications at modest cost.
[1] 216 A. Aho, R. Sethi, and J. Ullman, Compilers: Principles, Techniques, and Tools. Addison-Wesley Publishing Company, 1986.[2] J. Anderson, “Computer Security Technology Planning Study,” Technical Report ESD-TR-73-51, vol. II, US Air Force, Command and Management Systems, Bedford, Mass., Oct. 1972.[3] M. Bernaschi, E. Gabrielli, and L. Mancini, “Operating System Enhancements to Prevent the Misuse of System Calls,” Proc. ACM Conf. Computer and Comm. Security, pp. 174-183, 2000.[4] M. Blum, W. Evans, P. Gemmell, S. Kannan, and M. Naor, “Checking the Correctness of Memories,” Algorithmica, vol. 12, nos. 2-3, pp. 225-244, Aug. 1994.[5] M. Blum and S. Kannan, “Designing Programs that Check Their Work,” J. ACM, vol. 42, no. 1, pp. 269-291, Jan. 1995.[6] S. Chen, J. Xu, E.C. Sezer, P. Gauriar, and R. Iyer, “Non-Control-Data Attacks Are Realistic Threats,” Proc. USENIX Security Symp., Aug. 2005.[7] U. Elingsson and F. Schneider, “SASI Enforcement of Security Policies: A Retrospective,” Proc. New Security Paradigms Workshop, pp. 87-95, Sept. 1999.[8] T. Garfinkel, “Traps and Pitfalls: Practical Problems in System Call Interposition Based Security Tools,” Proc. Network and Distributed Systems Security Symp., Feb. 2003.[9] T. Garfinkel, B. Pfaff, and M. Rosenblum, “Ostia: A Delegating Architecture for Secure System Call Interposition,” Proc. Network and Distributed Systems Security Symp., Feb. 2004.[10] J. Geovedi, J. Nazario, N. Provos, and D. Song, “Project Hairy Eyeball,” http://blafasel.org/~flohhe/, 2005.[11] J.T. Giffin, S. Jha, and B.P. Miller, “Detecting Manipulated Remote Call Streams,” Proc. 11th USENIX Security Symp., Aug. 2002.[12] J.T. Giffin, S. Jha, and B.P. Miller, “Efficient Context-Sensitive Intrusion Detection,” Proc. Network and Distributed System Security Symp., Feb. 2004.[13] B. Gladman AES Combined Encryption/Authentication Library, http://fp.gladman.plus.com/AESindex.htm, 2006.[14] I. Goldberg, D. Wagner, R. Thomas, and E. Brewer, “A Secure Environment for Untrusted Helper Applications,” Proc. Sixth Usenix Security Symp., 1996.[15] S. Hofmeyr, S. Forrest, and A. Somayaji, “Intrusion Detection Using Sequences of System Calls,” J. Computer Security, vol. 6, no. 3, pp. 151-180, 1998.[16] T. Iwata and K. Kurosawa OMAC: One-Key CBC MAC, 2002.[17] K. Jain, and R. Sekar, “User-Level Infrastructure for System Call Interposition: A Platform for Intrusion Detection and Confinement,” Proc. Network and Distributed Systems Security Symp., pp. 19-34, Feb. 2000.[18] M. Jones, “Interposition Agents: Transparently Interposing User Code at the System Interface,” Proc. 14th ACM Symp. Operating Systems Principles (SOSP), pp. 80-93, Dec. 1993.[19] E. Krell and B. Krishnamurthy, “COLA: Customized Overlaying,” Proc. Winter 1992 Usenix Conf., pp. 3-7, Jan. 1992.[20] C. Kruegel, D. Mutz, F. Valeur, and G. Vigna, “On the Detection of Anomalous System Call Arguments,” Proc. Eighth European Symp. Research in Computer Security (ESORICS '03), pp. 326-343, 2003.[21] C.M. Linn, M. Rajagopalan, S. Baker, C. Collberg, and J.H. Hartman, “Protecting against Unexpected System Calls,” Proc. Usenix Security Symp., pp. 239-254, Aug. 2005.[22] M. Naor and K. Nissim, “Certificate Revocation and Certificate Update,” Proc. Seventh USENIX Security Symp., pp. 217-228, Jan. 1998.[23] G. Necula, J. Condit, M. Harren, S. McPeak, and W. Weimer, “CCured: Type-Safe Retrofitting of Legacy Software,” ACM Trans. Programming Languages and Systems, vol. 27, no. 3, pp. 477-526, May 2005.[24] G. Necula and P. Lee, “Safe Kernel Extensions without Run-Time Checking,” Proc. Operating System Design and Implementation (OSDI), pp. 229-243, Oct. 1996.[25] N. Provos, “Improving Host Security with System Call Policies,” Proc. 12th USENIX Security Symp., pp. 257-272, Aug. 2003.[26] R. Schlichting and F. Schneider, “Fail-Stop Processors: An Approach to Designing Fault Tolerant Computing Systems,” ACM Trans. Computer Systems, vol. 1, no. 3, pp. 222-238, Aug. 1983.[27] B. Schwarz, S. Debray, and G. Andrews, “Plto: A Link-Time Optimizer for the Intel IA-32 Architecture,” Proc. 2001 Workshop Binary Translation (WBT '01), 2001.[28] R. Sekar, M. Bendre, D. Dhurjati, and P. Bollineni, “A Fast Automaton-Based Method for Detecting Anomalous Program Behaviors,” Proc. IEEE Symp. Security and Privacy, pp. 144-155, 2001.[29] R. Sekar and P. Uppuluri, “Synthesizing Fast Intrusion Prevention/Detection Systems from High-Level Specifications,” Proc. Eighth USENIX Security Symp., pp. 63-78, 1999.[30] R. Sekar, V. Venkatakrishnan, S. Basu, S. Bhatkar, and D. DuVarney, “Model-Carrying Code: A Practical Approach for Safe Execution of Untrusted Applications,” Operating Systems Rev., vol. 37, no. 5, pp. 15-28, Dec. 2003.[31] K.M.C. Tan and R.A. Maxion, ““Why 6? Defining the Operational Limits of Stide, an Anomaly-Based Intrusion Detector,” Proc. 2002 IEEE Symp. Security and Privacy, p. 188, 2002.[32] V. Venkatakrishnan, R. Peri, and R. Sekar, “Empowering Mobile Code Using Expressive Security Policies,” Proc. 2002 Workshop New Security Paradigms, pp. 61-68, 2002.[33] D. Wagner and D. Dean, “Intrusion Detection via Static Analysis,” Proc. IEEE Symp. Security and Privacy, pp. 156-169, 2001.[34] C. Warrender, S. Forrest, and B. Pearlmutter, “Detecting Intrusions Using System Calls: Alternative Data Models,” Proc. IEEE Symp. Security and Privacy, pp. 133-145, 1999.[35] A. Wespi, M. Dacier, and H. Debar, “Intrusion Detection Using Variable-Length Audit Trail Patterns,” RAID '00: Proc. Third Int'l Workshop Recent Advances in Intrusion Detection, pp. 110-129, 2000.
Index Terms:
Intrusion tolerance, operating systems, security policy, sandboxing, compiler techniques.
Citation:
Mohan Rajagopalan, Matti A. Hiltunen, Trevor Jim, Richard D. Schlichting, "System Call Monitoring Using Authenticated System Calls," IEEE Transactions on Dependable and Secure Computing, vol. 3, no. 3, pp. 216-229, July-Sept. 2006, doi:10.1109/TDSC.2006.41
