A data mining approach for analysis of worm activity through automatic signature generation

ABSTRACT

This paper proposes a novel framework to automatically discover and analyze traffic generated by computer worms and other anomalous behaviors that interact with a non-solicited traffic monitoring system. Network packets are analyzed by an Intrusion Detection System (IDS), and new signatures are generated clustering those which remain unknown for the IDS. Furthermore, the framework provides a mechanism to cluster the alarms produced by the IDS producing a correlated vision of the traffic observed. Both the automatic signature generation and the alarm clusters are accomplished using data mining techniques.

REFERENCES

• 1: John Mark Agosta , Carlos Diuk-Wasser , Jaideep Chandrashekar , Carl Livadas, An adaptive anomaly detector for worm detection, Proceedings of the 2nd USENIX workshop on Tackling computer systems problems with machine learning techniques, p.1-6, April 10, 2007, Cambridge, MA
• 2: Vincent Berk , George Bakos , Robert Morris, Designing a Framework for Active Worm Detection on Global Networks, Proceedings of the First IEEE International Workshop on Information Assurance (IWIA'03), p.13, March 24-24, 2003
• 3: P. Berkhin. Survey of clustering data mining techniques. Technical report, Accrue Software, San Jose, CA, 2002.
• 4: S. Chen and S. Ranka. An internet-worm early warning system. In Proceedings of the 47th annual IEEE Global Telecommunications Conference (GLOBECOM 2004) -- Security and Network Management, November 2004.
• 5: Jeffrey Erman , Martin Arlitt , Anirban Mahanti, Traffic classification using clustering algorithms, Proceedings of the 2006 SIGCOMM workshop on Mining network data, p.281-286, September 11-15, 2006, Pisa, Italy [doi>10.1145/1162678.1162679]
• 6: Cristian Estan , Stefan Savage , George Varghese, Automatically inferring patterns of resource consumption in network traffic, Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications, August 25-29, 2003, Karlsruhe, Germany [doi>10.1145/863955.863972]
• 7: Usama Fayyad , Gregory Piatetsky-Shapiro , Padhraic Smyth, The KDD process for extracting useful knowledge from volumes of data, Communications of the ACM, v.39 n.11, p.27-34, Nov. 1996 [doi>10.1145/240455.240464]
• 8: R. S. Gray and V. H. Berk. Rapid detection of worms using ICMP-T3 analysis. In E. M. Carapezza, editor, Proceedings of SPIE Conference on Sensors, and Command, Control, Communications, and Intelligence (C3I) Technologies for Homeland Security and Homeland Defense III., volume 5403, pages 89--101, Orlando, FL, USA, September 2004.
• 9: Dan Gusfield, Algorithms on strings, trees, and sequences: computer science and computational biology, Cambridge University Press, New York, NY, 1997
• 10: O. Heinonen and H. Mannila. Attribute-oriented induction and conceptual clustering. Technical Report C-1996-2, University of Helsinki, 1996.
• 11: T. Holz, M. Dornseif, and F. Freiling. The Nepenthes Platform: An efficient approach to collect malware. In 9th International Symposium on Recent Advances in Intrusion Detection (RAID 2006), Hamburg, Germany, September 2006.
• 12: Anil K. Jain , Richard C. Dubes, Algorithms for clustering data, Prentice-Hall, Inc., Upper Saddle River, NJ, 1988
• 13: Klaus Julisch, Clustering intrusion detection alarms to support root cause analysis, ACM Transactions on Information and System Security (TISSEC), v.6 n.4, p.443-471, November 2003 [doi>10.1145/950191.950192]
• 14: K. Julisch. Using Root Cause Analysis to Handle Intrusion Detection Alarms. PhD thesis, University of Dortmund, Germany, 2003.
• 15: L. Kaufman and P. J. Rousseeuw. Finding Groups in Data: An Introduction to Cluster Analysis. Series in Applied Probability and Statistics. Wiley-Interscience, John Wiley & Sons, Inc, New York, USA, March 1990.
• 16: Hyang-Ah Kim , Brad Karp, Autograph: toward automated, distributed worm signature detection, Proceedings of the 13th conference on USENIX Security Symposium, p.19-19, August 09-13, 2004, San Diego, CA
• 17: C. Kreibich and J. Crowcroft. Honeycomb -- creating intrusion detection signatures using honeypots. In Proceedings of the Second Workshop on Hot Topics in Networks (Hotnets II), Boston, November 2003.
• 18: Justin Ma , Kirill Levchenko , Christian Kreibich , Stefan Savage , Geoffrey M. Voelker, Unexpected means of protocol inference, Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, October 25-27, 2006, Rio de Janeriro, Brazil [doi>10.1145/1177080.1177123]
• 19: Andrew W. Moore , Denis Zuev, Internet traffic classification using bayesian analysis techniques, Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 06-10, 2005, Banff, Alberta, Canada
• 20: David Moore , Geoffrey M. Voelker , Stefan Savage, Inferring internet denial-of-service activity, Proceedings of the 10th conference on USENIX Security Symposium, p.2-2, August 13-17, 2001, Washington, D.C.
• 21: James Newsome , Brad Karp , Dawn Song, Polygraph: Automatically Generating Signatures for Polymorphic Worms, Proceedings of the 2005 IEEE Symposium on Security and Privacy, p.226-241, May 08-11, 2005 [doi>10.1109/SP.2005.15]
• 22: Leonard Pitt , Robert E. Reinke, Criteria for Polynomial-Time (Conceptual) Clustering, Machine Learning, v.2 n.4, p.371-396, April 1988 [doi>10.1023/A:1022825229661]
• 23: G. Portokalidis and H. Bos. Sweetbait: Zero-hour worm detection and containment using honeypots. Technical Report IR-CS-015, Vrije Universiteit, Amsterdam, The Netherlands, May 2005.
• 24: N. Provos. Honeyd: a virtual honeypot daemon. In Proceedings of the 10th DFN-CERT Workshop, February 2003.
• 25: Niels Provos, A virtual honeypot framework, Proceedings of the 13th conference on USENIX Security Symposium, p.1-1, August 09-13, 2004, San Diego, CA
• 26: J. Riordan, D. Zamboni, and Y. Duponchel. Lessons learned from Billy Goat, an accurate worm--detection system. Research Report RZ 3609 (#99619), IBM Zurich Research Laboratory, Saumerstrasse 4, CH--8803, Ruschlikon, Zurich, Switzerland, 2005.
• 27: Martin Roesch, Snort - Lightweight Intrusion Detection for Networks, Proceedings of the 13th USENIX conference on System administration, November 07-12, 1999, Seattle, Washington
• 28: Sumeet Singh , Cristian Estan , George Varghese , Stefan Savage, Automated worm fingerprinting, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.4-4, December 06-08, 2004, San Francisco, CA
• 29: S. Staniford--Chen, S. Cheung, R. Crawford, M. Dilger, J. Frank, J. Hoagland, K. Levitt, C. Wee, R. Yip, and D. Zerkle. GrIDS -- A graph-based intrusion detection system for large networks. In Proceedings of the 19th National Information Systems Security Conference, 1996.
• 30: The Snort Project. Snort users manual 2.6. Web page at http://www.snort.org/docs/snort_manual/2.8.0/snort_manual.pdf, October 2007.
• 31: K. Wang, G. Cretu, and S. Stolfo. Anomalous payload--based worm detection and signature generation. In Proceedings of the Eighth International Symposium on Recent Advances in Intrusion Detection (RAID 2005), September 2005.
• 32: Vinod Yegneswaran , Jonathon T. Giffin , Paul Barford , Somesh Jha, An architecture for generating semantics-aware signatures, Proceedings of the 14th conference on USENIX Security Symposium, p.7-7, July 31-August 05, 2005, Baltimore, MD
• 33: D. Zamboni, J. Riordan, and Y. Duponchel. Building and deploying Billy Goat: a worm-detection system. In 18th Annual FIRST Conference, June 2006.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.6 Security and Protection
          Subjects: Invasive software (e.g., viruses, worms, Trojan horses)

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.4 Knowledge Representation Formalisms and Methods
          Subjects: Representations (procedural and rule-based)

General Terms:
Experimentation, Security

Keywords:
automatic signature generation, data mining, honeypots, intrusion detection, malware identification