Resource-aware multi-format network security data storage

ABSTRACT

Internet security systems like intrusion detection and intrusion prevention systems are based on a simple input-output principle: they receive a high-bandwidth stream of input data and produce summaries of suspicious events. This simple model has serious drawbacks, including the inability to attach context to security alerts, a lack of detailed historical information for anomaly detection baselines, and a lack of detailed forensics information. Together these problems highlight a need for fine-grained security data in the shortterm, and coarse-grained security data in the long-term. To address these limitations we propose resource-aware multi-format security data storage. Our approach is to develop an architecture for recording different granularities of security data simultaneously. To explore this idea we present a novel framework for analyzing security data as a spectrum of information and a set of algorithms for collecting and storing multi-format data. We construct a prototype system and deploy it on darknets at academic, Fortune 100 enterprise, and ISP networks. We demonstrate how a hybrid algorithm that provides guarantees on time and space satisfies the short and long-term goals across a four month deployment period and during a series of large-scale denial of service attacks.

REFERENCES

• 1: M. Bailey, E. Cooke, F. Jahanian, J. Nazario, and D. Watson. The Internet Motion Sensor: A distributed blackhole monitoring system. In Proceedings of Network and Distributed System Security Symposium (NDSS '05), San Diego, CA, February 2005.
• 2: E. Cooke, M. Bailey, F. Jahanian, and R. Mortier. The dark oracle: Perspective-aware unused and unreachable address discovery. In Proceedings of the 3rd USENIX Symposium on Networked Systems Design and Implementation (NSDI '06), May 2006.
• 3: Nick Duffield , Carsten Lund, Predicting resource usage and estimation accuracy in an IP flow measurement collection infrastructure, Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement, October 27-29, 2003, Miami Beach, FL, USA [doi>10.1145/948205.948228]
• 4: Nick Duffield , Carsten Lund , Mikkel Thorup, Charging from sampled network usage, Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement, November 01-02, 2001, San Francisco, California, USA [doi>10.1145/505202.505232]
• 5: Cristian Estan , Ken Keys , David Moore , George Varghese, Building a better NetFlow, Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications, August 30-September 03, 2004, Portland, Oregon, USA
• 6: C. S. Inc. Netflow services and applications. http://www.cisco.com/warp/public/cc/pd/iosw/ioft/neflct/tech/napps_wp.h%tm, 2002.
• 7: Ruoming Pang , Vinod Yegneswaran , Paul Barford , Vern Paxson , Larry Peterson, Characteristics of internet background radiation, Proceedings of the 4th ACM SIGCOMM conference on Internet measurement, October 25-27, 2004, Taormina, Sicily, Italy [doi>10.1145/1028788.1028794]
• 8: Vern Paxson, Bro: a system for detecting network intruders in real-time, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.23-24, p.2435-2463, Dec. 1999 [doi>10.1016/S1389-1286(99)00112-7]
• 9: P. Phaal , S. Panchen , N. McKee, InMon Corporation's sFlow: A Method for Monitoring Traffic in Switched and Routed Networks, RFC Editor, 2001
• 10: Martin Roesch, Snort - Lightweight Intrusion Detection for Networks, Proceedings of the 13th USENIX conference on System administration, November 07-12, 1999, Seattle, Washington

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations

General Terms:
Measurement, Security

Keywords:
anomaly classification, anomaly detection, darknet, network-wide traffic analysis