Operational experiences with high-volume network intrusion detection

ABSTRACT

In large-scale environments, network intrusion detection systems (NIDSs) face extreme challenges with respect to traffic volume, traffic diversity, and resource management. While crucial for acceptance and operational deployment, the research literature mainly omits such practical difficulties. In this paper, we offer an evaluation based on extensive operational experience. More specifically, we identify and explore key factors with respect to resource management and efficient packet processing and highlight their impact using a set of real-world traces. On the one hand, these insights help us gauge the trade-offs of tuning a NIDS. On the other hand, they motivate us to explore several novel ways of reducing resource requirements. These enable us to improve the state management considerably as well as balance the processing load dynamically. Overall this enables us to operate a NIDS successfully in our high-volume network environments.

REFERENCES

• 1: D. Agarwal, J. M. Gonzalez, G. Jin, and B. Tierney. An infrastructure for passive network monitoring of application data streams. In Proc. Passive and Active Measurement Workshop, 2003.
• 2: S. A. Crosby and D. S. Wallach. Denial of service via algorithmic complexity attacks. In Proc. 12th USENIX Security Symposium, 2003.
• 3: L. Deri. Improving passive packet capture: Beyond device polling. Technical report, University of Pisa, 2003.
• 4: A. Feldmann , A. C. Gilbert , W. Willinger, Data networks as cascades: investigating the multifractal nature of Internet WAN traffic, Proceedings of the ACM SIGCOMM '98 conference on Applications, technologies, architectures, and protocols for computer communication, p.42-55, August 31-September 04, 1998, Vancouver, British Columbia, Canada
• 5: Sally Floyd , Vern Paxson, Difficulties in simulating the internet, IEEE/ACM Transactions on Networking (TON), v.9 n.4, p.392-403, August 2001 [doi>10.1109/90.944338]
• 6: GNU Binutils. http://www.gnu.org/software/binutils.
• 7: M. Hall and K. Wiley. Capacity verification for high speed network intrusion detection systems. In Proc. Recent Advances in Intrusion Detection, number 2516 in Lecture Notes in Computer Science. Springer-Verlag, 2002.
• 8: Christopher Kruegel , Fredrik Valeur , Giovanni Vigna , Richard Kemmerer, Stateful Intrusion Detection for High-Speed Networks, Proceedings of the 2002 IEEE Symposium on Security and Privacy, p.285, May 12-15, 2002
• 9: W. Lee, J. B. Cabrera, A. Thomas, N. Balwalli, S. Saluja, and Y. Zhang. Performance adaptation in real-time intrusion detection systems. In Proc. Recent Advances in Intrusion Detection, number 2516 in Lecture Notes in Computer Science. Springer-Verlag, 2002.
• 10: S. McCanne and V. Jacobson. The BSD packet filter: A new architecture for user-level packet capture. In Proc. Winter 1993 USENIX Conference, 1993.
• 11: David Moore , Vern Paxson , Stefan Savage , Colleen Shannon , Stuart Staniford , Nicholas Weaver, Inside the Slammer Worm, IEEE Security and Privacy, v.1 n.4, p.33-39, July 2003 [doi>10.1109/MSECP.2003.1219056]
• 12: D. Moore and C. Shannon. The spread of the Witty. http://www.caida.org/analysis/security/witty, 2004.
• 13: D. Moore, G. M. Voelker, and S. Savage. Inferring Internet denial-of-service activity. In Proc. 10th USENIX Security Symposium, 2001.
• 14: mpatrol. http://www.cbmamiga.demon.co.uk/mpatrol.
• 15: Vern Paxson, Empirically derived analytic models of wide-area TCP connections, IEEE/ACM Transactions on Networking (TON), v.2 n.4, p.316-336, Aug. 1994 [doi>10.1109/90.330413]
• 16: Vern Paxson, Bro: a system for detecting network intruders in real-time, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.23-24, p.2435-2463, Dec. 1999 [doi>10.1016/S1389-1286(99)00112-7]
• 17: T. H. Ptacek and T. N. Newsham. Insertion, evasion, and denial of service: Eluding network intrusion detection. Technical report, Secure Networks, Inc., 1998.
• 18: M. J. Ranum. Experiences benchmarking intrusion detection systems. Technical report, NFR Security, Inc., http://www.itsecurity.com/papers/nfr2.htm, 2001.
• 19: Martin Roesch, Snort - Lightweight Intrusion Detection for Networks, Proceedings of the 13th USENIX conference on System administration, November 07-12, 1999, Seattle, Washington
• 20: Configuring SPAN and RSPAN (Cisco Catalyst 6500 Series). http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/sw_7_5/conf%g_gd/span.pdf.
• 21: Snot. http://www.stolenshoes.net/sniph/index.html.
• 22: Robin Sommer , Vern Paxson, Enhancing byte-level network intrusion detection signatures with context, Proceedings of the 10th ACM conference on Computer and communications security, October 27-30, 2003, Washington D.C., USA [doi>10.1145/948109.948145]
• 23: R. Sommer and V. Paxson. Exploiting independent state for network intrusion detection. Technical report, TU München, 2004.
• 24: Stuart Staniford , Vern Paxson , Nicholas Weaver, How to Own the Internet in Your Spare Time, Proceedings of the 11th USENIX Security Symposium, p.149-167, August 05-09, 2002
• 25: Stick. http://packetstormsecurity.nl/distributed/stick.htm.
• 26: tcpdump. http://www.tcpdump.org.
• 27: Valgrind. http://developer.kde.org/sewardj/.
• 28: Walter Willinger , Murad S. Taqqu , Robert Sherman , Daniel V. Wilson, Self-similarity through high-variability: statistical analysis of Ethernet LAN traffic at the source level, IEEE/ACM Transactions on Networking (TON), v.5 n.1, p.71-86, Feb. 1997 [doi>10.1109/90.554723]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations
          Subjects: Network monitoring

General Terms:
Measurement, Performance, Security

Keywords:
bro, evaluation, network intrusion detection, security