An empirical evaluation of entropy-based traffic anomaly detection

ABSTRACT

Entropy-based approaches for anomaly detection are appealing since they provide more fine-grained insights than traditional traffic volume analysis. While previous work has demonstrated the benefits of entropy-based anomaly detection, there has been little effort to comprehensively understand the detection power of using entropy-based analysis of multiple traffic distributions in conjunction with each other. We consider two classes of distributions: flow-header features (IP addresses, ports, and flow-sizes), and behavioral features (degree distributions measuring the number of distinct destination/source IPs that each host communicates with). We observe that the timeseries of entropy values of the address and port distributions are strongly correlated with each other and provide very similar anomaly detection capabilities. The behavioral and flow size distributions are less correlated and detect incidents that do not show up as anomalies in the port and address distributions. Further analysis using synthetically generated anomalies also suggests that the port and address distributions have limited utility in detecting scan and bandwidth flood anomalies. Based on our analysis, we discuss important implications for entropy-based anomaly detection.

REFERENCES

• 1: Snort. http://www.snort.org.
• 2: Argus. http://qosient.com/argus/.
• 3: Paul Barford , Jeffery Kline , David Plonka , Amos Ron, A signal analysis of network traffic anomalies, Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment, November 06-08, 2002, Marseille, France [doi>10.1145/637201.637210]
• 4: Daniela Brauckhoff , Bernhard Tellenbach , Arno Wagner , Martin May , Anukool Lakhina, Impact of packet sampling on anomaly detection metrics, Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, October 25-27, 2006, Rio de Janeriro, Brazil [doi>10.1145/1177080.1177101]
• 5: Feinstein, L., Schnackenberg, D., Balupari, R., and Kindred, D. Statistical Approaches to DDoS Attack Detection and Response. In Proc. of DARPA Information Survivability Conference and Exposition (2003).
• 6: Jung, J., Paxson, V., Berger, A. W., and Balakrishnan, H. Fast Portscan Detection Using Sequential Hypothesis Testing. In Proc. of the IEEE Symposium on Security and Privacy (2004).
• 7: Vijay Karamcheti , Davi Geiger , Zvi Kedem , S. Muthukrishnan, Detecting malicious network traffic using inverse distributions of packet contents, Proceedings of the 2005 ACM SIGCOMM workshop on Mining network data, August 26-26, 2005, Philadelphia, Pennsylvania, USA [doi>10.1145/1080173.1080176]
• 8: Kazaa. www.kazaa.com.
• 9: Abhishek Kumar , Minho Sung , Jun (Jim) Xu , Jia Wang, Data streaming algorithms for efficient and accurate estimation of flow size distribution, Proceedings of the joint international conference on Measurement and modeling of computer systems, June 10-14, 2004, New York, NY, USA
• 10: Anukool Lakhina , Mark Crovella , Christophe Diot, Mining anomalies using traffic feature distributions, Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications, August 22-26, 2005, Philadelphia, Pennsylvania, USA
• 11: Ashwin Lall , Vyas Sekar , Mitsunori Ogihara , Jun Xu , Hui Zhang, Data streaming algorithms for estimating entropy of network traffic, Proceedings of the joint international conference on Measurement and modeling of computer systems, June 26-30, 2006, Saint Malo, France
• 12: Wenke Lee , Dong Xiang, Information-Theoretic Measures for Anomaly Detection, Proceedings of the 2001 IEEE Symposium on Security and Privacy, p.130, May 14-16, 2001
• 13: Jim Morrison, Blaster Revisited, Queue, v.2 n.4, June 2004 [doi>10.1145/1016978.1016980]
• 14: Cisco Netflow. http://www.cisco.com/warp/public/732/Tech/nmp/netflow/index.shtml.
• 15: Nychis, G., Sekar, V., Andersen, D. G., Kim, H., and Zhang, H. An Empirical Evaluation of Entropy-Based Traffic Anomaly Detection. Tech. Rep. CMU-CS-08-145, Computer Science Department, Carnegie Mellon University, 2008.
• 16: P. Phaal , S. Panchen , N. McKee, InMon Corporation's sFlow: A Method for Monitoring Traffic in Switched and Routed Networks, RFC Editor, 2001
• 17: Trammell, B., and Boschi, E. Bidirectional Flow Export Using IP Flow Information Export (IPFIX). RFC 5103, 2008.
• 18: Arno Wagner , Bernhard Plattner, Entropy Based Worm and Anomaly Detection in Fast IP Networks, Proceedings of the 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise, p.172-177, June 13-15, 2005 [doi>10.1109/WETICE.2005.35]
• 19: Jun Xu , Jinliang Fan , Mostafa H. Ammar , Sue B. Moon, Prefix-Preserving IP Address Anonymization: Measurement-Based Security Evaluation and a New Cryptography-Based Scheme, Proceedings of the 10th IEEE International Conference on Network Protocols, p.280-289, November 12-15, 2002
• 20: Kuai Xu , Zhi-Li Zhang , Supratik Bhattacharyya, Profiling internet backbone traffic: behavior models and applications, Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications, August 22-26, 2005, Philadelphia, Pennsylvania, USA

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations
          Subjects: Network management

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations
          Subjects: Network monitoring

General Terms:
Management, Measurement

Keywords:
anomaly detection, entropy