Applying data mining to intrusion detection: the quest for automation, efficiency, and credibility

ABSTRACT

Intrusion detection is an essential component of the layered computer security mechanisms. It requires accurate and efficient models for analyzing a large amount of system and network audit data. This paper is an overview of our research in applying data mining techniques to build intrusion detection models. We describe a framework for mining patterns from system and network audit data, and constructing features according to analysis of intrusion patterns. We discuss approaches for improving the run-time efficiency as well as the credibility of detection models. We report the ideas, algorithms, and prototype systems we have developed, and discuss open research problems.

REFERENCES

• 1: Rakesh Agrawal , Tomasz Imieliński , Arun Swami, Mining association rules between sets of items in large databases, Proceedings of the 1993 ACM SIGMOD international conference on Management of data, p.207-216, May 25-28, 1993, Washington, D.C., United States
• 2: D. Anderson, T. Frivold, and A. Valdes. Next-generation intrusion detection expert system (NIDES): A summary. Technical Report SRI-CSL-95-07, Computer Science Laboratory, SRI International, Menlo Park, California, May 1995.
• 3: Stefan Axelsson, The base-rate fallacy and the difficulty of intrusion detection, ACM Transactions on Information and System Security (TISSEC), v.3 n.3, p.186-205, Aug. 2000 [doi>10.1145/357830.357849]
• 4: W. W. Cohen. Fast effective rule induction. In Machine Learning: the 12th International Conference, Lake Taho, CA, 1995. Morgan Kaufmann.
• 5: Wei Fan , Wenke Lee , Salvatore J. Stolfo , Matthew Miller, A Multiple Model Cost-Sensitive Approach for Intrusion Detection, Proceedings of the 11th European Conference on Machine Learning, p.142-153, May 31-June 02, 2000
• 6: A. K. Ghosh and A. Schwartzbard. A study in using neural networks for anomaly and misuse detection. In Proceedings of the 8th USENIX Security symposium, August 1999.
• 7: Koral Ilgun , R. A. Kemmerer , Phillip A. Porras, State Transition Analysis: A Rule-Based Intrusion Detection Approach, IEEE Transactions on Software Engineering, v.21 n.3, p.181-199, March 1995 [doi>10.1109/32.372146]
• 8: S. Kumar and E. H. Spafford. A software architecture to support misuse intrusion detection. In Proceedings of the 18th National Information security Conference, pages 194--204, 1995.
• 9: Terran Lane , Carla E. Brodley, Temporal sequence learning and data reduction for anomaly detection, ACM Transactions on Information and System Security (TISSEC), v.2 n.3, p.295-331, Aug. 1999 [doi>10.1145/322510.322526]
• 10: Wenke Lee , Salvatore J. Stolfo, A data mining framework for constructing features and models for intrusion detection systems (computer security, network security), 1999
• 11: Wenke Lee , Salvatore J. Stolfo, A framework for constructing features and models for intrusion detection systems, ACM Transactions on Information and System Security (TISSEC), v.3 n.4, p.227-261, Nov. 2000 [doi>10.1145/382912.382914]
• 12: W. Lee, S. J. Stolfo, and K. W. Mok. Mining audit data to build intrusion detection models. In Proceedings of the 4th International Conference on Knowledge Discovery and Data Mining, New York, NY, August 1998. AAAI Press.
• 13: Wenke Lee , Salvatore J. Stolfo , Kui W. Mok, Mining in a data-flow environment: experience in network intrusion detection, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.114-124, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312212]
• 14: R. Lippmann, D. Fried, I. Graf, J. Haines, K. Kendall, D. McClung, D. Weber, S. Webster, D. Wyschogrod, R. Cunninghan, and M. Zissman. Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. In Proceedings of the 2000 DARPA Information Survivability Conference and Exposition, January 2000.
• 15: H. Mannila, H. Toivonen, and A. I. Verkamo. Discovering frequent episodes in sequences. In Proceedings of the 1st International Conference on Knowledge Discovery in Databases and Data Mining, Montreal, Canada, August 1995.
• 16: S. McCanne, C. Leres, and V. Jacobson. libpcap. available via anonymous ftp to ftp.ee.lbl.gov, 1994.
• 17: Foster Provost , David Jensen , Tim Oates, Efficient progressive sampling, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.23-32, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312188]
• 18: Bruce Schneier, Secrets & Lies: Digital Security in a Networked World, John Wiley & Sons, Inc., New York, NY, 2000
• 19: Salvatore J. Stolfo , Wenke Lee , Philip K. Chan , Wei Fan , Eleazar Eskin, Data mining-based intrusion detectors: an overview of the columbia IDS project, ACM SIGMOD Record, v.30 n.4, December 2001 [doi>10.1145/604264.604267]
• 20: SunSoft. SunSHIELD Basic Security Module Guide. SunSoft, Mountain View, CA, 1995.
• 21: P. Viola and M. Jones. Robust real-time object detection. In Proceedings of the Second International Workshop on Statistical and Computational Theories of Vison - Modeling, Learning, Computing, and Sampling, May 2002.
• 22: C. Warrender, S. Forrest, and B. Pearlmutter. Detecting intrusions using system calls: Alternative data models. In Proceedings of the 1999 IEEE Symposium on Security and Privacy, May 1999.

INDEX TERMS

Keywords:
Bayesian detection rate, feature construction, intrusion detection, model efficiency