skip to main content
10.1145/1229285.1229291acmconferencesArticle/Chapter ViewAbstractPublication Pagesasia-ccsConference Proceedingsconference-collections
Article

Analyzing network traffic to detect self-decrypting exploit code

Published: 20 March 2007 Publication History

Abstract

Remotely-launched software exploits are a common way for attackers to intrude into vulnerable computer systems. As detection techniques improve, remote exploitation techniques are also evolving. Recent techniques for evasion of exploit detection include polymorphism (code encryption) and meta-morphism (code obfuscation). This paper addresses the problem of detecting in network traffic polymorphic remote exploits that are encrypted, and that self-decrypt before launching the intrusion. Such exploits pose a great challenge to existing malware detection techniques, partly due to the non-obvious starting location of the exploit code in the network payload.We describe a new method for detecting self-decrypting exploit codes. This method scans network traffic for the presence of a decryption routine, which is characteristic of such exploits. The proposed method uses static analysis and emulated instruction execution techniques. This improves the accuracy of determining the starting location and instructions of the decryption routine, even if self-modifying code is used. The method outperforms approaches that have been previously proposed, both in terms of detection capabilities, and in detection accuracy.The proposed method has been implemented and tested on current polymorphic exploits, including ones generated by state-of-the-art polymorphic engines. All exploits have been detected (i.e., a 100% detection rate), including those for which the decryption routine is dynamically coded, or self-modifying. The false positive rate is close to 0%. Running time is approximately linear in the size of the network payload being analyzed.

References

[1]
Common vulnerabilities and exposures. http://cve.mitre.org/cve/downloads/full-cve.csv.
[2]
Computer Economics. http://www.computereconomics.com.
[3]
Intel Architecture Software Developers Manual. Volume 2: Instruction Set Reference.
[4]
Metasploit project. http://www.metasploit.org.
[5]
The ADMmutate polymorphic engine. http://www.ktwo.ca/ADMmutate-0.8.4.tar.gz.
[6]
The CLET polymorphism engine. http://www.phrack.org/show.php?p=61&a=9.
[7]
Bro Intrusion Detection System, 2003. http://www.bro-ids.org.
[8]
Snort: an open source network intrusion prevention and detection system, 2005. http://www.snort.org.
[9]
P. Akritidis, E. Markatos, M. Polychronakis, and K. Anagnostakis. STRIDE: Polymorphic Sled Detection through Instruction Sequence Analysis. In Proceedings of the 20th IFIP International Information Security Conference (SEC'05), pages 375--392, June 2005.
[10]
R. Chinchani and E. Berg. A Fast Static Analysis Approach To Detect Exploit Code Inside Network Flows. In Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID'05), pages 284--308, September 2005.
[11]
M. Christodorescu, S. Jha, S. A. Seshia, D. Song, and R. E. Bryant. Semantics-Aware Malware Detection. In Proceedings of 2005 IEEE Symposium on Security and Privacy (S&P'05), pages 32--46, May 2005.
[12]
J. C Foster and M. Price. Sockets, Shellcode, Porting, & Coding: Reverse Engineering Exploits and Tool Coding for Security Professionals. Syngress Publishing, USA, 2005.
[13]
C. Kruegel, W. Robertson, F. Valeur, and G. Vigna. Static Disassembly of Obfuscated Binaries. In Proceedings of the 13th USENIX Security Symposium, pages 255--270, Auguest 2004.
[14]
Z. Li, M. Sanghi, Y. Chen, M. Kao, and B. Chavez. Hamsa: Fast signature generation for zero-day polymorphic worms with provable attack resilience. In Proceedings of 2006 IEEE Symposium on Security and Privacy (S&P'06), pages 32--47, May 2006.
[15]
S. S. Muchnick. Advanced Comiler Design Implementation. Morgan Kaufmann Publisher, CA, USA, 1997.
[16]
J. Newsome, B. Karp, and D. Song. Polygraph: Automatically Generating Signatures for Polymorphic Worms. In Proceedings of 2005 IEEE Symposium on Security and Privacy (S&P'05), pages 226--241, May 2005.
[17]
J. Newsome, B. Karp, and D. Song. Paragraph: Thwarting Signature Learning By Training Maliciously. In Proceedings of the 9th International Symposium on Recent Advances in Intrusion Detection (RAID'06), September 2006.
[18]
U. Payer, M. Lamberger, and P. Teufl. Hybrid engine for polymorphic code detection. In Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment(DIMVA'05), pages 19--31, July 2005.
[19]
M. Polychronakis, K. Anagnostakis, and E. Markatos. Network-Level Polymorphic Shellcode Detection Using Emulation. In Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment(DIMVA '06), July 2006.
[20]
P. Royal, M. Halpin, D. Dagon, R. Edmonds, and W. Lee. PolyUnpack: Automating the Hidden-Code Extraction of Unpack-Executing Malware. In Proceedings of the 22th Annual Computer Security Applications Conference (ACSAC'06), December 2006.
[21]
S. Sidiroglou and A. Keromytis. Countering Network Worms Through Automatic Patch Generation. In Research Report, 2003.
[22]
T. Toth and C. Kruegel. Accurate Buffer Overflow Detection via Abstract Payload Execution. In Proceedings of the 5th International Symposium on Recent Advances in Intrusion Detection (RAID'02), pages 274--291, October 2002.
[23]
X. Wang, C. Pan, P. Liu, and S. Zhu. SigFree: A Signature-free Buffer Overflow Attack Blocker. In Proceedings of the 15th USENIX Security Symposium, pages 225--240, July 2006.
[24]
J. Xu, Z. Kalbarczyk, and R. K. Iyer. Transparent runtime randomization for security. In Proceedings of the 22th International Symposium on Reliable Distributed Systems (SRDS'03), pages 260--269, October 2003.
[25]
V. Yegneswaran, J. Giffin, P. Barford, and S. Jha. An architecture for generating semantic-aware signatures. In Proceedings of the 14th USENIX Security Symposium, pages 97--112, August 2005.

Cited By

View all
  • (2023)Traffic-Oriented Shellcode Detection Based on VSMAdvances in Internet, Data & Web Technologies10.1007/978-3-031-26281-4_15(152-162)Online publication date: 12-Feb-2023
  • (2022)Binary Exploitation in Industrial Control Systems: Past, Present and FutureIEEE Access10.1109/ACCESS.2022.317192210(48242-48273)Online publication date: 2022
  • (2020)REDT: Remote exploitation detection technology for network infrastructureIOP Conference Series: Materials Science and Engineering10.1088/1757-899X/715/1/012035715(012035)Online publication date: 3-Jan-2020
  • Show More Cited By

Index Terms

  1. Analyzing network traffic to detect self-decrypting exploit code

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ASIACCS '07: Proceedings of the 2nd ACM symposium on Information, computer and communications security
    March 2007
    323 pages
    ISBN:1595935746
    DOI:10.1145/1229285
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 March 2007

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. decryption
    2. detection
    3. emulation
    4. exploit code
    5. polymorphic
    6. static analysis

    Qualifiers

    • Article

    Conference

    Asia CCS07
    Sponsor:

    Acceptance Rates

    ASIACCS '07 Paper Acceptance Rate 33 of 180 submissions, 18%;
    Overall Acceptance Rate 418 of 2,322 submissions, 18%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)13
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 09 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Traffic-Oriented Shellcode Detection Based on VSMAdvances in Internet, Data & Web Technologies10.1007/978-3-031-26281-4_15(152-162)Online publication date: 12-Feb-2023
    • (2022)Binary Exploitation in Industrial Control Systems: Past, Present and FutureIEEE Access10.1109/ACCESS.2022.317192210(48242-48273)Online publication date: 2022
    • (2020)REDT: Remote exploitation detection technology for network infrastructureIOP Conference Series: Materials Science and Engineering10.1088/1757-899X/715/1/012035715(012035)Online publication date: 3-Jan-2020
    • (2019)HEDGE: Efficient Traffic Classification of Encrypted and Compressed PacketsIEEE Transactions on Information Forensics and Security10.1109/TIFS.2019.291115614:11(2916-2926)Online publication date: Nov-2019
    • (2019)Classifying Malware Represented as Control Flow Graphs using Deep Graph Convolutional Neural Network2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)10.1109/DSN.2019.00020(52-63)Online publication date: Jun-2019
    • (2017)k-Depth Mimicry Attack to Secretly Embed Shellcode into PDF FilesInformation Science and Applications 201710.1007/978-981-10-4154-9_45(388-395)Online publication date: 18-Mar-2017
    • (2016)Semantics-aware detection of targeted attacks: a surveyJournal of Computer Virology and Hacking Techniques10.1007/s11416-016-0273-313:1(47-85)Online publication date: 2-May-2016
    • (2015)Identifying functions in binary code with reverse extended control flow graphsJournal of Software: Evolution and Process10.1002/smr.173327:10(793-820)Online publication date: 1-Oct-2015
    • (2014)NBA of obfuscated network vulnerabilities' exploitation hidden into HTTPS trafficThe 9th International Conference for Internet Technology and Secured Transactions (ICITST-2014)10.1109/ICITST.2014.7038827(310-317)Online publication date: Dec-2014
    • (2014)On Emulation-Based Network Intrusion Detection SystemsResearch in Attacks, Intrusions and Defenses10.1007/978-3-319-11379-1_19(384-404)Online publication date: 2014
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media