Analyzing network traffic to detect self-decrypting exploit code

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Analyzing network traffic to detect self-decrypting exploit code

Full text

Pdf (270 KB)

Source

ASIAN ACM Symposium on Information, Computer and Communications Security archive
Proceedings of the 2nd ACM symposium on Information, computer and communications security table of contents

Singapore

SESSION: Network security table of contents

Pages: 4 - 12

Year of Publication: 2007

ISBN:1-59593-574-6

Authors

Qinghua Zhang	North Carolina State University, Raleigh, NC
Douglas S. Reeves	North Carolina State University, Raleigh, NC
Peng Ning	North Carolina State University, Raleigh, NC
S. Purushothaman Iyer	North Carolina State University, Raleigh, NC

Sponsor

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 18, Downloads (12 Months): 137, Citation Count: 1

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1229285.1229291 What is a DOI?

ABSTRACT

Remotely-launched software exploits are a common way for attackers to intrude into vulnerable computer systems. As detection techniques improve, remote exploitation techniques are also evolving. Recent techniques for evasion of exploit detection include polymorphism (code encryption) and meta-morphism (code obfuscation). This paper addresses the problem of detecting in network traffic polymorphic remote exploits that are encrypted, and that self-decrypt before launching the intrusion. Such exploits pose a great challenge to existing malware detection techniques, partly due to the non-obvious starting location of the exploit code in the network payload.We describe a new method for detecting self-decrypting exploit codes. This method scans network traffic for the presence of a decryption routine, which is characteristic of such exploits. The proposed method uses static analysis and emulated instruction execution techniques. This improves the accuracy of determining the starting location and instructions of the decryption routine, even if self-modifying code is used. The method outperforms approaches that have been previously proposed, both in terms of detection capabilities, and in detection accuracy.The proposed method has been implemented and tested on current polymorphic exploits, including ones generated by state-of-the-art polymorphic engines. All exploits have been detected (i.e., a 100% detection rate), including those for which the decryption routine is dynamically coded, or self-modifying. The false positive rate is close to 0%. Running time is approximately linear in the size of the network payload being analyzed.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Common vulnerabilities and exposures. http://cve.mitre.org/cve/downloads/full-cve.csv.
	2	Computer Economics. http://www.computereconomics.com.
	3	Intel Architecture Software Developers Manual. Volume 2: Instruction Set Reference.
	4	Metasploit project. http://www.metasploit.org.
	5	The ADMmutate polymorphic engine. http://www.ktwo.ca/ADMmutate-0.8.4.tar.gz.
	6	The CLET polymorphism engine. http://www.phrack.org/show.php?p=61&a=9.
	7	Bro Intrusion Detection System, 2003. http://www.bro-ids.org.
	8	Snort: an open source network intrusion prevention and detection system, 2005. http://www.snort.org.
	9	P. Akritidis, E. Markatos, M. Polychronakis, and K. Anagnostakis. STRIDE: Polymorphic Sled Detection through Instruction Sequence Analysis. In Proceedings of the 20^th IFIP International Information Security Conference (SEC'05), pages 375--392, June 2005.
	10	R. Chinchani and E. Berg. A Fast Static Analysis Approach To Detect Exploit Code Inside Network Flows. In Proceedings of the 8^th International Symposium on Recent Advances in Intrusion Detection (RAID'05), pages 284--308, September 2005.
	11	Mihai Christodorescu , Somesh Jha , Sanjit A. Seshia , Dawn Song , Randal E. Bryant, Semantics-Aware Malware Detection, Proceedings of the 2005 IEEE Symposium on Security and Privacy, p.32-46, May 08-11, 2005 [doi>10.1109/SP.2005.20]
	12	Stuart McClure , James C. Foster , Mike Price, Sockets, Shellcode, Porting, & Coding: Reverse Engineering Exploits And Tool Coding For Security Professionals, Syngress Publishing, 2005
	13	C. Kruegel, W. Robertson, F. Valeur, and G. Vigna. Static Disassembly of Obfuscated Binaries. In Proceedings of the 13^th USENIX Security Symposium, pages 255--270, Auguest 2004.
	14	Zhichun Li , Manan Sanghi , Yan Chen , Ming-Yang Kao , Brian Chavez, Hamsa: Fast Signature Generation for Zero-day PolymorphicWorms with Provable Attack Resilience, Proceedings of the 2006 IEEE Symposium on Security and Privacy (S&P'06), p.32-47, May 21-24, 2006 [doi>10.1109/SP.2006.18]
	15	Steven S. Muchnick, Advanced compiler design and implementation, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
	16	James Newsome , Brad Karp , Dawn Song, Polygraph: Automatically Generating Signatures for Polymorphic Worms, Proceedings of the 2005 IEEE Symposium on Security and Privacy, p.226-241, May 08-11, 2005 [doi>10.1109/SP.2005.15]
	17	J. Newsome, B. Karp, and D. Song. Paragraph: Thwarting Signature Learning By Training Maliciously. In Proceedings of the 9^th International Symposium on Recent Advances in Intrusion Detection (RAID'06), September 2006.
	18	U. Payer, M. Lamberger, and P. Teufl. Hybrid engine for polymorphic code detection. In Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment(DIMVA'05), pages 19--31, July 2005.
	19	M. Polychronakis, K. Anagnostakis, and E. Markatos. Network-Level Polymorphic Shellcode Detection Using Emulation. In Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment(DIMVA '06), July 2006.
	20	Paul Royal , Mitch Halpin , David Dagon , Robert Edmonds , Wenke Lee, PolyUnpack: Automating the Hidden-Code Extraction of Unpack-Executing Malware, Proceedings of the 22nd Annual Computer Security Applications Conference on Annual Computer Security Applications Conference, p.289-300, December 11-15, 2006 [doi>10.1109/ACSAC.2006.38]
	21	S. Sidiroglou and A. Keromytis. Countering Network Worms Through Automatic Patch Generation. In Research Report, 2003.
	22	T. Toth and C. Kruegel. Accurate Buffer Overflow Detection via Abstract Payload Execution. In Proceedings of the 5^th International Symposium on Recent Advances in Intrusion Detection (RAID'02), pages 274--291, October 2002.
	23	X. Wang, C. Pan, P. Liu, and S. Zhu. SigFree: A Signature-free Buffer Overflow Attack Blocker. In Proceedings of the 15^th USENIX Security Symposium, pages 225--240, July 2006.
	24	J. Xu, Z. Kalbarczyk, and R. K. Iyer. Transparent runtime randomization for security. In Proceedings of the 22^th International Symposium on Reliable Distributed Systems (SRDS'03), pages 260--269, October 2003.
	25	V. Yegneswaran, J. Giffin, P. Barford, and S. Jha. An architecture for generating semantic-aware signatures. In Proceedings of the 14^th USENIX Security Symposium, pages 97--112, August 2005.