A framework for constructing features and models for intrusion detection systems

ABSTRACT

Intrusion detection (ID) is an important component of infrastructure protection mechanisms. Intrusion detection systems (IDSs) need to be accurate, adaptive, and extensible. Given these requirements and the complexities of today's network environments, we need a more systematic and automated IDS development process rather that the pure knowledge encoding and engineering approaches. This article describes a novel framework, MADAM ID, for Mining Audit Data for Automated Models for Instrusion Detection. This framework uses data mining algorithms to compute activity patterns from system audit data and extracts predictive features from the patterns. It then applies machine learning algorithms to the audit records taht are processed according to the feature definitions to generate intrusion detection rules. Results from the 1998 DARPA Intrusion Detection Evaluation showed that our ID model was one of the best performing of all the participating systems. We also briefly discuss our experience in converting the detection models produced by off-line data mining programs to real-time modules of existing IDSs.

REFERENCES

• 1: Rakesh Agrawal , Tomasz Imieliński , Arun Swami, Mining association rules between sets of items in large databases, Proceedings of the 1993 ACM SIGMOD international conference on Management of data, p.207-216, May 25-28, 1993, Washington, D.C., United States
• 2: ALLEN, J., CHRISTIE, A., FITHEN, W., MCHUGH, J., PICKEL, J., AND STONER, E. 2000. State of the practice of intrusion detection technologies. CMU/SEI-99-TR-028,CMU/SEI. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA.
• 3: ANDERSON, D., FRIVOLD, T., AND VALDES, A. 1995. Next-generation intrusion detection expert system (NIDES): A summary. SRI-CSL-95-07 (May).
• 4: CHAN,P.K.AND STOLFO, S. J. 1993. Toward parallel and distributed learning by metalearning. In Proceedings of the AAAI Workshop on Knowledge Discovery in Databases. 227-240.
• 5: COHEN, W. W. 1995. Fast effective rule induction. In Proceedings of 12th International Conference on Machine Learning (Lake Tahoe, CA). Morgan Kaufmann, San Mateo, CA.
• 6: Usama Fayyad , Gregory Piatetsky-Shapiro , Padhraic Smyth, The KDD process for extracting useful knowledge from volumes of data, Communications of the ACM, v.39 n.11, p.27-34, Nov. 1996 [doi>10.1145/240455.240464]
• 7: Stephanie Forrest , Steven A. Hofmeyr , Anil Somayaji , Thomas A. Longstaff, A Sense of Self for Unix Processes, Proceedings of the 1996 IEEE Symposium on Security and Privacy, p.120, May 06-08, 1996
• 8: GHOSH,A.K.AND SCHWARTZBARD, A. 1999. A study in using neural networks for anomaly and misuse detection. In Proceedings of the 8th Security Symposium on USENIX (USENIX, Aug.).
• 9: ILGUN, K. 1992. USTAT: A real-time intrusion detection system for Unix. Master's Thesis. University of California at Santa Barbara, Santa Barbara, CA.
• 10: Koral Ilgun , R. A. Kemmerer , Phillip A. Porras, State Transition Analysis: A Rule-Based Intrusion Detection Approach, IEEE Transactions on Software Engineering, v.21 n.3, p.181-199, March 1995 [doi>10.1109/32.372146]
• 11: JACOBSON, V., LERES, C., AND MCCANNE, S. 1989. Tcpdump. available via anonymous ftp to ftp.ee.lbl.gov.
• 12: KO, C., FINK, G., AND LEVITT, K. 1994. Automated detection of vulnerabilities in privileged programs by execution monitoring. In Proceedings of the 10th Conference on Computer Security Applications (Dec.). IEEE Computer Society Press, Los Alamitos, CA, 134-144.
• 13: KUMAR,S.AND SPAFFORD, E. H. 1995. A software architecture to support misuse intrusion detection. In Proceedings of the 18th National Conference on Information Security. 194-204.
• 14: Terran Lane , Carla E. Brodley, Temporal sequence learning and data reduction for anomaly detection, ACM Transactions on Information and System Security (TISSEC), v.2 n.3, p.295-331, Aug. 1999 [doi>10.1145/322510.322526]
• 15: Wenke Lee , Salvatore J. Stolfo, A data mining framework for constructing features and models for intrusion detection systems (computer security, network security), 1999
• 16: LEE,W.AND STOLFO, S. J. 1998. Data mining approaches for intrusion detection. In Proceedings of the 7th Symposium on USENIX Security (San Antonio, TX, Jan.).
• 17: LEE, W., STOLFO,S.J.,AND MOK, K. W. 1999. A data mining framework for building intrusion detection models. In Proceedings of the 1999 IEEE Symposium on Security and Privacy (Oakland, California, May).
• 18: Wenke Lee , Salvatore J. Stolfo , Kui W. Mok, Mining in a data-flow environment: experience in network intrusion detection, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.114-124, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312212]
• 19: LIPPMANN,R.P.,FRIED, D., GRAF, I., HAINES, J., KENDALL, K., MCCLUNG, D., WEBBER, D., WEBSTER, S., WYSCHOGRAD, D., CUNNINGHAN, R., AND ZISSMAN, M. 2000. Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. In Proceedings of the on DARPA Information Survivability Conference and Exposition (DISCEX '00, Hilton Head, South Carolina, Jan. 25-27). IEEE Computer Society Press, Los Alamitos, CA, 12-26.
• 20: LUNT, T. 1993. Detecting intruders in computer systems. In Proceedings of the 1993 Conference on Auditing and Computer Technology.
• 21: LUNT, T., TAMARU, A., GILHAM, F., JAGANNATHAN, R., NEUMANN, P., JAVITZ, H., VALDES, A., AND GARVEY, T. 1992. A real-time intrusion detection expert system (IDES) - final technical report.
• 22: MANNILA,H.AND TOIVONEN, H. 1996. Discovering generalized episodes using minimal occurrences. In Proceedings of the 2nd International Conference on Knowledge Discovery in Databases and Data Mining (Portland, OR, Aug.).
• 23: MANNILA, H., TOIVONEN, H., AND VERKAMO, A. I. 1995. Discovering frequent episodes in sequences. In Proceedings of the First International Conference on Knowledge Discovery in Databases and Data Mining (Montreal, Canada, Aug. 20-21).
• 24: Thomas M. Mitchell, Machine Learning, McGraw-Hill Higher Education, 1997
• 25: MUKHERJEE, B., HEBERLEIN,L.T.,AND LEVITT, K. N. 1994. Network intrusion detection. IEEE Network 8, 1 (Jan.).
• 26: NETWORK FLIGHT RECORDER INC. 1997. Network flight recorder. http://www.nfr.com
• 27: PAXSON, V. 1998. Bro: A system for detecting network intruders in real-time. In Proceedings of the 7th Symposium on USENIX Security (San Antonio, TX, Jan.).
• 28: PORRAS,P.AND NEUMANN, P. 1997. EMERALD: Event monitoring enabling responses to anomalous live disturbances. In Proceedings of the 20th National Conference on National Information Systems Security. Vol.1 (Baltimore, MD). National Institute of Standards and Technology, Gaithersburg, MD, 353-365.
• 29: STOLFO,S.J.,PRODROMIDIS,A.L.,TSELEPIS, S., LEE, W., FAN,D.W.,AND CHAN, P. K. 1997. JAM: Java agents for meta-learning over distributed databases. In Proceedings of the 3rd ACM SIGMOD International Workshop on Data Mining and Knowledge Discovery (SIG-MOD- 96, Newport Beach, CA, Aug.), R. Ng, Ed. ACM Press, New York, NY, 74-81.
• 30: SUNSOFT. 1995. SunSHIELD Basic Security Module Guide.
• 31: WARRENDER, C., FORREST, S., AND PERLMUTTER, B. 1999. Detecting intrusions using system calls: Alternative data models. In Proceedings of the 1999 IEEE Computer Society Symposium on Research in Security and Privacy (Berkeley, CA, May). IEEE Computer Society Press, Los Alamitos, CA, 133-145.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.0 General
          Subjects: Security and protection (e.g., firewalls)

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations
          Subjects: Network monitoring

  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Concept learning

General Terms:
Design, Experimentation, Security

Keywords:
data mining, feature construction, intrusion detection