The base-rate fallacy and the difficulty of intrusion detection

ABSTRACT

Many different demands can be made of intrusion detection systems. An important requirement is that an intrusion detection system be effective; that is, it should detect a substantial percentage of intrusions into the supervised system, while still keeping the false alarm rate at an acceptable level. This article demonstrates that, for a reasonable set of assumptions, the false alarm rate is the limiting factor for the performance of an intrusion detection system. This is due to the base-rate fallacy phenomenon, that in order to achieve substantial values of the Bayesian detection rate P(Intrusion***Alarm), we have to achieve a (perhaps in some cases unattainably) low false alarm rate. A selection of reports of intrusion detection performance are reviewed, and the conclusion is reached that there are indications that at least some types of intrusion detection have far to go before they can attain such low false alarm rates.

REFERENCES

• 1: ANDERSON, J. P. 1980. Computer security threat monitoring and surveillance. 79F26400 26 Feb revised April 15.
• 2: AXELSSON, S. 1998. Research in intrusion-detection systems: A survey. 98--17.
• 3: AXELSSON, S. 2000. Intrusion-detection systems: A taxonomy and survey. 99-15 (March).
• 4: AXELSSON, S. 2000. A preliminary attempt to apply detection and estimation theory to intrusion detection. 00--4 (March).
• 5: AXELSSON, S., LINDQVIST, U., GUSTAFSON, U., AND JONSSON, E. 1998. An approach to UNIX security logging. In Proceedings of the 21st NIST-NCSC National Conference on Informa-tion Systems Security (Crystal City, Arlington, VA, Oct. 5-8). National Institute of Standards and Technology, Gaithersburg, MD, 62-75.
• 6: DEATHERAGE, B. H. 1972. Auditory and other sensory forms of information. In Human Engineering Guide to Equipment Design: Army, Navy, Air Force, H. Van Cott and R. Kinkade, Eds.
• 7: Herve DEBAR , Monique BECKER , Didier SIBONI, A Neural Network Component for an Intrusion Detection System, Proceedings of the 1992 IEEE Symposium on Security and Privacy, p.240, May 04-06, 1992
• 8: Dorothy E. Denning, An intrusion-detection model, IEEE Transactions on Software Engineering, v.13 n.2, p.222-232, Feb. 1987 [doi>10.1109/TSE.1987.232894]
• 9: DENNING,D.E.AND NEUMANN, P. G. 1985. Requirements and model for IDES: A real-time intrusion detection system.
• 10: HALME,L.AND KAHN, B. 1988. Building a security monitor with adaptive user work profiles. In Proceedings of the 11th National Computer Security Conference (NIST-NCSC, Baltimore, Maryland, Oct.17-20). National Institute of Standards and Technology, Gaithersburg, MD, 000-000.
• 11: P. Helman , G. Liepins, Statistical Foundations of Audit Trail Analysis for the Detection of Computer Misuse, IEEE Transactions on Software Engineering, v.19 n.9, p.886-901, September 1993 [doi>10.1109/32.241771]
• 12: Terran Lane , Carla E. Brodley, Temporal sequence learning and data reduction for anomaly detection, ACM Transactions on Information and System Security (TISSEC), v.2 n.3, p.295-331, Aug. 1999 [doi>10.1145/322510.322526]
• 13: LEE, W. 1999. A data mining framework for building intrusion detection models. In Proceedings of the 1999 IEEE Computer Society Symposium on Research in Security and Privacy (Berkeley, CA, May). IEEE Computer Society Press, Los Alamitos, CA, 120-132.
• 14: LIPPMANN,R.P.,FRIED, D., GRAF, I., ET AL. 2000. Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. In Proceedings of the DARPA Informa-tion Survivability Conference and Exposition (DISCEX '00, Hilton Head, South Carolina, Jan. 25-27). IEEE Computer Society Press, Los Alamitos, CA, 12-26.
• 15: LUNT, T. F. 1988. Automated audit trail analysis and intrusion detection. In Proceedings of the on 11th National Computer Security Conference (NIST-NCSC, Baltimore, Maryland, Oct.17-20). National Institute of Standards and Technology, Gaithersburg, MD, 65-73.
• 16: MATTHEWS, R. 1996. Base-rate errors and rain forecasts. Nature 382, 6594, 766.
• 17: MATTHEWS, R. 1997. Decision-theoretic limits on earthquake prediction. Geophys. J. Int. 131, 3 (Dec.), 526-529.
• 18: MCHUGH, J. 2000. Testing intrusion detection systems: A critique of the 1998 and 1999 Lincoln Laboratory evaluations. ACM Trans. Inf. Syst. Secur. 3.
• 19: NYGREN, E. 1994. Moderna tider: teknikutveckling inom medicinsk service.
• 20: PIERCE, G. M. 1943. Destruction by demolition, incendiaries and sabotage: Field training manual, Fleet Marine Force, US Marine Corps.
• 21: Jens Rasmussen, Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering, Elsevier Science Inc., New York, NY, 1986
• 22: Stuart J. Russell , Peter Norvig, Artificial intelligence: a modern approach, Prentice-Hall, Inc., Upper Saddle River, NJ, 1995
• 23: SEBRING,M.M.,SHELLHOUSE, E., HANNA,M.E.,AND WHITEHURST, R. A. 1988. Expert systems in intrusion detection: A case study. In Proceedings of the 11th National Computer Security Conference (NIST-NCSC, Baltimore, Maryland, Oct.17-20). National Institute of Standards and Technology, Gaithersburg, MD, 74-81.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS

General Terms:
Performance, Security, Theory

Keywords:
base-rate fallacy, detection rate, false alarm rate, intrusion detection