Mimicry attacks on host-based intrusion detection systems

ABSTRACT

We examine several host-based anomaly detection systems and study their security against evasion attacks. First, we introduce the notion of a mimicry attack, which allows a sophisticated attacker to cloak their intrusion to avoid detection by the IDS. Then, we develop a theoretical framework for evaluating the security of an IDS against mimicry attacks. We show how to break the security of one published IDS with these methods, and we experimentally confirm the power of mimicry attacks by giving a worked example of an attack on a concrete IDS implementation. We conclude with a call for further research on intrusion detection from both attacker's and defender's viewpoints.

REFERENCES

• 1: M. Chung, N. Puketza, R.A. Olsson, B. Mukherjee, "Simulating Concurrent Intrusions for Testing Intrusion Detection Systems: Parallelizing Intrusions," National Information Systems Security Conference, pp.173--183, 1995.
• 2: Stephanie Forrest , Steven A. Hofmeyr , Anil Somayaji , Thomas A. Longstaff, A Sense of Self for Unix Processes, Proceedings of the 1996 IEEE Symposium on Security and Privacy, p.120, May 06-08, 1996
• 3: Stephanie Forrest , Alan S. Perelson , Lawrence Allen , Rajesh Cherukuri, Self-Nonself Discrimination in a Computer, Proceedings of the 1994 IEEE Symposium on Security and Privacy, p.202, May 16-18, 1994
• 4: Anup K. Ghosh , Aaron Schwartzbard , Michael Schatz, Learning Program Behavior Profiles for Intrusion Detection, Proceedings of the Workshop on Intrusion Detection and Network Monitoring, p.51-62, April 09-12, 1999
• 5: Anup K. Ghosh , Aaron Schwartzbard , Michael Schatz, Learning Program Behavior Profiles for Intrusion Detection, Proceedings of the Workshop on Intrusion Detection and Network Monitoring, p.51-62, April 09-12, 1999
• 6: Jonathon T. Giffin , Somesh Jha , Barton P. Miller, Detecting Manipulated Remote Call Streams, Proceedings of the 11th USENIX Security Symposium, p.61-79, August 05-09, 2002
• 7: M. Handley, C. Kreibich, V. Paxson, "Network Intrusion Detection: Evasion, Traffic Normalization, and End-to-End Protocol Semantics," 10th USENIX Security Symposium, 2001.
• 8: S. Hofmeyr, S. Forrest, A. Somayaji, "Intrusion Detection Using Sequences of System Calls," Journal of Computer Security, vol. 6, pp. 151--180, 1998.
• 9: Gerard J. Holzmann, Design and validation of computer protocols, Prentice-Hall, Inc., Upper Saddle River, NJ, 1990
• 10: Gerard J. Holzmann, The Model Checker SPIN, IEEE Transactions on Software Engineering, v.23 n.5, p.279-295, May 1997 [doi>10.1109/32.588521]
• 11: John E. Hopcroft , Jeffrey D. Ullman, Introduction To Automata Theory, Languages, And Computation, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1990
• 12: T. Lane, C.E. Brodley, "Sequence Matching and Learning in Anomaly Detection for Computer Security," AAAI Workshop: AI Approaches to Fraud Detection and Risk Management, pp.49--49, 1997.
• 13: Terran Lane , Carla E. Brodley, Temporal sequence learning and data reduction for anomaly detection, ACM Transactions on Information and System Security (TISSEC), v.2 n.3, p.295-331, Aug. 1999 [doi>10.1145/322510.322526]
• 14: W. Lee, S.J. Stolfo, "Data Mining Approaches for Intrusion Detection," 7th USENIX Security Symposium, 1998.
• 15: W. Lee, S.J. Stolfo, K. Mok, "A Data Mining Framework for Building Intrusion Detection Models," IEEE Symposium on Security & Privacy, 1999.
• 16: Kenneth L. McMillan, Symbolic Model Checking, Kluwer Academic Publishers, Norwell, MA, 1993
• 17: Christoph Michael , Anup K. Ghosh, Using Finite Automata to Mine Execution Data for Intrusion Detection: A Preliminary Report, Proceedings of the Third International Workshop on Recent Advances in Intrusion Detection, p.66-79, October 02-04, 2000
• 18: Vern Paxson, Bro: a system for detecting network intruders in real-time, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.23-24, p.2435-2463, Dec. 1999 [doi>10.1016/S1389-1286(99)00112-7]
• 19: T.H. Ptacek, T.N. Newsham, "Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection," Secure Networks, Jan. 1998.
• 20: Fred B. Schneider, Enforceable security policies, ACM Transactions on Information and System Security (TISSEC), v.3 n.1, p.30-50, Feb. 2000 [doi>10.1145/353323.353382]
• 21: A. Somayaji, S. Forrest, "Automated Response Using System-Call Delays," 9th Usenix Security Symposium, pp.185--197, 2000.
• 22: Anil Buntwal Somayaji , Stephanie Forrest, Operating system stability and security through process homeostasis, 2002
• 23: K.M.C. Tan, K.S. Killourhy, R.A. Maxion, "Undermining an Anomaly-Based Intrusion Detection System Using Common Exploits," to appear at RAID 2002, 16--18 Oct. 2002.
• 24: Kymie M. C. Tan , John McHugh , Kevin S. Killourhy, Hiding Intrusions: From the Abnormal to the Normal and Beyond, Revised Papers from the 5th International Workshop on Information Hiding, p.1-17, October 07-09, 2002
• 25: David Wagner , Drew Dean, Intrusion Detection via Static Analysis, Proceedings of the 2001 IEEE Symposium on Security and Privacy, p.156, May 14-16, 2001
• 26: C. Warrender, S. Forrest, B. Pearlmutter, "Detecting intrusions using system calls: Alternative data models," 1999 IEEE Symposium on Security & Privacy.
• 27: Andreas Wespi , Marc Dacier , Hervé Debar, Intrusion Detection Using Variable-Length Audit Trail Patterns, Proceedings of the Third International Workshop on Recent Advances in Intrusion Detection, p.110-129, October 02-04, 2000

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.6 Security and Protection

General Terms:
Security

Keywords:
anomaly detection, evasion attacks, host-based intrusion detection