This Article 
 Bibliographic References 
 Add to: 
A Methodology for Testing Intrusion Detection Systems
October 1996 (vol. 22 no. 10)
pp. 719-729

Abstract—Intrusion Detection Systems (IDSs) attempt to identify unauthorized use, misuse, and abuse of computer systems. In response to the growth in the use and development of IDSs, we have developed a methodology for testing IDSs. The methodology consists of techniques from the field of software testing which we have adapted for the specific purpose of testing IDSs. In this paper, we identify a set of general IDS performance objectives which is the basis for the methodology. We present the details of the methodology, including strategies for test-case selection and specific testing procedures. We include quantitative results from testing experiments on the Network Security Monitor (NSM), an IDS developed at UC Davis. We present an overview of the software platform that we have used to create user-simulation scripts for testing experiments. The platform consists of the UNIX tool expect and enhancements that we have developed, including mechanisms for concurrent scripts and a record-and-replay feature. We also provide background information on intrusions and IDSs to motivate our work.

[1] D. Anderson et al., "Next Generation Intrusion Detection Expert System (NIDES)," Software Design, Product Specification, and Version Description Document, Project 3131, SRI Int'l, July11, 1994.
[2] R.G. Bace Division of Infosec Computer Science, Research, and Technology, Nat'l Security Agency, private communication, May 1995.
[3] S.M. Bellovin, "There Be Dragons," Proc. Third USENIX UNIX Security Symp., pp. 1-16,Baltimore, Sept. 1992.
[4] S.M. Bellovin, "Security Problems in the TCP/IP Protocol Suite," Computer Comm. Reviews, May 1989.
[5] M. Bishop, "A Taxonomy of UNIX System and Network Vulnerabilities," Technical Report CSE-95-10, Univ. of California at Davis, Sept. 1995.
[6] P. Brinch Hansen, "Reproducible Testing of Monitors," Software-Practice and Experience, vol. 8, pp. 721-729, 1978.
[7] M. Chung, N. Puketza, R.A. Olsson, and B. Mukherjee, "Simulating Concurrent Intrusions for Testing Intrusion Detection Systems: Parallelizing Intrusions," Proc., 18th Nat'l Information Systems Security Conf., pp. 173-183,Baltimore, Oct. 1995.
[8] D.E. Denning, “An Intrusion-Detection Model,” IEEE Trans. Software Eng., vol. 13, pp. 222–232, Feb. 1987.
[9] C. Dowell and P. Ramstedt, "The COMPUTERWATCH Data Reduction Tool," Proc. 13th Nat'l Computer Security Conf., pp. 99-108, Washington, D.C., Oct. 1990.
[10] D. Farmer and W. Venema, "Improving the Security of Your Site by Breaking into It," USENET posting, Dec. 1993.
[11] D. Farmer and E.H. Spafford, "The COPS Security Checker System," Proc. Summer USENIX Conf., pp. 165-170, June 1990.
[12] L.D. Gary talk presented in "Crime on the Internet" session, 17th Nat'l Computer Security Conf., Baltimore, Oct.12, 1994.
[13] L.T. Heberlein et al., "A Network Security Monitor," Proc. IEEE Symp. Research Security and Privacy, IEEE Computer Soc. Press, Los Alamitos, Calif., 1990, pp. 296-304.
[14] J. Hochberg et al., "NADIR: An Automated System for Detecting Network Intrusion and Misuse," Computers and Security, vol. 12, no. 3, pp. 235-248, May 1993.
[15] K. Ilgun, "USTAT: A real-time intrusion detection system for UNIX," Proc. 1993 IEEE Symp. Research in Security and Privacy, pp. 16-28, May 1993.
[16] K. Ilgun, R.A. Kemmerer, and P.A. Porras, “State Transition Analysis: A Rule-Based Intrusion Detection Approach,” IEEE Trans. Software Eng., vol. 21, no. 3, pp. 181–199, 1995.
[17] H.S. Javitz and A. Valdes, “The Sri Ides Statistical Anomaly Detector,” Proc. IEEE Computer Society Symp. Security and Privacy, May 1991.
[18] S. Kumar and E.H. Spafford, "A Software Architecture to Support Misuse Intrusion Detection," Technical Report CSD-TR-95-009, Purdue Univ., Mar.17, 1995.
[19] S. Kumar and E.H. Spafford, "An Application of Pattern Matching in Intrusion Detection," Technical Report CSD-TR-94-013, Purdue Univ., June17, 1994.
[20] S. Kumar and E.H. Spafford, "A Pattern Matching Model for Misuse Intrusion Detection," Proc. 17th Nat'l Computer Security Conf., pp. 11-21,Baltimore, Oct. 1994.
[21] C. Landwehr et al., "A Taxonomy of Computer Program Security Flaws," Computing Surveys, Vol. 26, No. 3, Sept. 1994, pp. 211-255.
[22] T.J. LeBlanc and J.M. Mellor-Crummey, "Debugging Parallel Programs with Instant Replay," IEEE Trans. Computers, vol. 36, no. 4, pp. 471-482, Apr. 1987.
[23] D. Libes, Exploring Expect: A Tcl-Based Toolkit for Automating Interactive Programs. O'Reilly&Associates, 1994.
[24] R.H. Wilson, “On Geometric Assembly Planning,” PhD thesis, Dept. of Computer Science, Stanford Univ., 1992.
[25] T.F. Lunt et al., "A Real-Time Intrusion Detection Expert System(IDES)," Interim Progress Report, Project 6784, SRI Int'l, May 1990.
[26] G.J. Myers,The Art of Software Testing.New York: Wiley, 1979.
[27] B. Mukherjee, L.T. Heberlein, and K.N. Levitt, “Network Intrusion Detection,” IEEE Network, pp. 26–41, June 1994.
[28] P.G. Neumann and D.B. Parker, "A Summary of Computer Misuse Techniques," Proc. 12th Nat'l Computer Security Conf., pp. 396-407,Baltimore, Oct. 1989.
[29] J. Ousterhout, Tcl and the Tk Toolkit, Addison Wesley Longman, Reading, Mass., 1994.
[30] D.R. Safford, D.L. Schales, and D.K. Hess, "The TAMU Security Package: An Ongoing Response to Internet Intruders in an Academic Environment," Proc. Fourth USENIX UNIX Security Symp., pp. 91-118,Santa Clara, Calif., Oct. 1993.
[31] M.M. Sebring, E. Shellhouse, M.E. Hanna, and R.A. Whitehurst, "Expert Systems in Intrusion Detection: A Case Study," Proc. 11th Nat'l Computer Security Conf., pp. 74-81,Baltimore, Oct. 1988.
[32] S.E. Smaha, "Haystack: An Intrusion Detection System," Proc. Fourth Aerospace Computer Security Application Conf.,Orlando, Fla., pp. 37-44, Dec. 1988.
[33] S. Snapp, J. Brentano, G. Dias, T. Goan, L. Heberlein, C. Ho, K. Levitt, B. Mukherjee, S. Smaha, T. Grance, D. Teal, and D. Mansur, "DIDS (Distributed Intrusion Detection System)—Motivation, Architecture, and An Early Prototype," Proc. 14th Nat'l Computer Security Conf., pp. 167-176,Washington, D.C., Oct. 1991.
[34] E.J. Weyuker and B. Jeng,“Analyzing partition testing strategies,” IEEE Trans. Software Engineering, vol. 17, pp. 703-711, 1991.
[35] K. Zhang, "A Methodology for Testing Intrusion Detection Systems," MS thesis, Univ. of California at Davis, May 1993.

Index Terms:
Intrusion detection, software testing, computer security, computer user simulation
Nicholas J. Puketza, Kui Zhang, Mandy Chung, Biswanath Mukherjee, Ronald A. Olsson, "A Methodology for Testing Intrusion Detection Systems," IEEE Transactions on Software Engineering, vol. 22, no. 10, pp. 719-729, Oct. 1996, doi:10.1109/32.544350
Usage of this product signifies your acceptance of the Terms of Use.