skip to main content
10.1145/1920261.1920305acmotherconferencesArticle/Chapter ViewAbstractPublication PagesacsacConference Proceedingsconference-collections
research-article

Comprehensive shellcode detection using runtime heuristics

Published:06 December 2010Publication History

ABSTRACT

A promising method for the detection of previously unknown code injection attacks is the identification of the shellcode that is part of the attack vector using payload execution. Existing systems based on this approach rely on the self-decrypting behavior of polymorphic code and can identify only that particular class of shellcode. Plain, and more importantly, metamorphic shellcode do not carry a decryption routine nor exhibit any self-modifications and thus both evade existing detection systems. In this paper, we present a comprehensive shellcode detection technique that uses a set of runtime heuristics to identify the presence of shellcode in arbitrary data streams. We have identified fundamental machine-level operations that are inescapably performed by different shellcode types, based on which we have designed heuristics that enable the detection of plain and metamorphic shellcode regardless of the use of self-decryption. We have implemented our technique in Gene, a code injection attack detection system based on passive network monitoring. Our experimental evaluation and real-world deployment show that Gene can effectively detect a large and diverse set of shellcode samples that are currently missed by existing detectors, while so far it has not generated any false positives.

References

  1. Goodfellas security research team. http://goodfellas.shellcode.com.ar/.Google ScholarGoogle Scholar
  2. The metasploit project. http://www.metasploit.com/.Google ScholarGoogle Scholar
  3. milw0rm. http://milw0rm.com/shellcode/win32/.Google ScholarGoogle Scholar
  4. Packet storm. http://www.packetstormsecurity.org/.Google ScholarGoogle Scholar
  5. Win32 assembly components, Dec. 2002. http://lsd-pl.net.Google ScholarGoogle Scholar
  6. Common shellcode naming initiative, 2009. http://nepenthes.carnivore.it/csni.Google ScholarGoogle Scholar
  7. Retrieving kernel32's base address, June 2009. http://www.harmonysecurity.com/blog/2009/06/retrieving-kernel32s-base-address.html.Google ScholarGoogle Scholar
  8. S. Andersson, A. Clark, and G. Mohay. Network-based buffer overflow detection by exploit code analysis. In Proceedings of the Asia Pacific Information Technology Security Conference (AusCERT), 2004.Google ScholarGoogle Scholar
  9. P. Baecher and M. Koetter. libemu, 2009. http://libemu.carnivore.it/.Google ScholarGoogle Scholar
  10. P. Bania. Evading network-level emulation, 2009. http://piotrbania.com/all/articles/pbania-evading-nemu2009.pdf.Google ScholarGoogle Scholar
  11. K. Borders, A. Prakash, and M. Zielinski. Spector: Automatically analyzing shell code. In Proceedings of the Annual Computer Security Applications Conference (ACSAC), 2007.Google ScholarGoogle ScholarCross RefCross Ref
  12. R. Chinchani and E. V. D. Berg. A fast static analysis approach to detect exploit code inside network flows. In Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID), Sept. 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. S. P. Chung and A. K. Mok. Swarm attacks against network-level emulation/analysis. In Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection (RAID), September 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. M. Egele, P. Wurzinger, C. Kruegel, and E. Kirda. Defending browsers against drive-by downloads: Mitigating heap-spraying code injection attacks. In Proceedings of the 6th international conference on Detection of Intrusions and Malware, & Vulnerability Assessment (DIMVA), 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. S. Ford, M. Cova, C. Kruegel, and G. Vigna. Wepawet, 2009. http://wepawet.cs.ucsb.edu/.Google ScholarGoogle Scholar
  16. I)ruid. Context-keyed payload encoding. Uninformed, 9, Oct. 2007.Google ScholarGoogle Scholar
  17. C. Kruegel, E. Kirda, D. Mutz, W. Robertson, and G. Vigna. Polymorphic worm detection using structural information of executables. In Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID), Sept. 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. J. Ma, J. Dunagan, H. J. Wang, S. Savage, and G. M. Voelker. Finding diversity in remote code injection exploits. In Proceedings of the 6th Internet Measurement Conference (IMC), 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. J. Mason, S. Small, F. Monrose, and G. MacManus. English shellcode. In Proceedings of the 16th ACM conference on Computer and communications security (CCS), 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. U. Payer, P. Teufl, and M. Lamberger. Hybrid engine for polymorphic shellcode detection. In Proceedings of the Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA), pages 19--31, July 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. M. Pietrek. A crash course on the depths of Win32#8482;structured exception handling, 1997. http://www.microsoft.com/msj/0197/exception/exception.aspx.Google ScholarGoogle Scholar
  22. M. Polychronakis, K. G. Anagnostakis, and E. P. Markatos. An empirical study of real-world polymorphic code injection attacks. In Proceedings of the 2nd USENIX Workshop on Large-scale Exploits and Emergent Threats (LEET), April 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. M. Polychronakis, E. P. Markatos, and K. G. Anagnostakis. Network-level polymorphic shellcode detection using emulation. In Proceedings of the Third Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA), July 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. M. Polychronakis, E. P. Markatos, and K. G. Anagnostakis. Emulation-based detection of non-self-contained polymorphic shellcode. In Proceedings of the 10th International Symposium on Recent Advances in Intrusion Detection (RAID), September 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. M. Shimamura and K. Kono. Yataglass: Network-level code emulation for analyzing memory-scanning attacks. In Proceedings of the 6th international conference on Detection of Intrusions and Malware, & Vulnerability Assessment (DIMVA), 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. sk. History and advances in windows shellcode. Phrack, 11(62), July 2004.Google ScholarGoogle Scholar
  27. Skape. Understanding windows shellcode, 2003. http://www.hick.org/code/skape/papers/win32-shellcode.pdf.Google ScholarGoogle Scholar
  28. Skape. Safely searching process virtual address space, 2004. http://www.hick.org/code/skape/papers/egghunt-shellcode.pdf.Google ScholarGoogle Scholar
  29. SkyLined. Finding the base address of kernel32 in Windows 7. http://skypher.com/index.php/2009/07/22/shellcode-finding-kernel32-in-windows-7/.Google ScholarGoogle Scholar
  30. SkyLined. SEH GetPC (XP SP3), July 2009. http://skypher.com/wiki/index.php/Hacking/Shellcode/Alphanumeric/ALPHA3/x86/ASCII/Mixedcase/SEH_GetPC_(XP_sp3).Google ScholarGoogle Scholar
  31. Y. Song, M. E. Locasto, A. Stavrou, A. D. Keromytis, and S. J. Stolfo. On the infeasibility of modeling polymorphic shellcode. In Proceedings of the 14th ACM conference on Computer and communications security (CCS), 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. P. Ször. The Art of Computer Virus Research and Defense. Addison-Wesley Professional, February 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. T. Toth and C. Kruegel. Accurate buffer overflow detection via abstract payload execution. In Proceedings of the 5th Symposium on Recent Advances in Intrusion Detection (RAID), Oct. 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. X. Wang, Y.-C. Jhi, S. Zhu, and P. Liu. Still: Exploit code detection via static taint and initialization analyses. In Proceedings of the Annual Computer Security Applications Conference (ACSAC), 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. X. Wang, C.-C. Pan, P. Liu, and S. Zhu. Sigfree: A signature-free buffer overflow attack blocker. In Proceedings of the USENIX Security Symposium, Aug. 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. B.-J. Wever. SEH Omelet Shellcode, 2009. http://code.google.com/p/w32-seh-omelet-shellcode/.Google ScholarGoogle Scholar
  37. G. Wicherski. Win32 egg search shellcode, 33 bytes. http://blog.oxff.net/2009/02/win32-egg-search-shellcode-33-bytes.html.Google ScholarGoogle Scholar
  38. Q. Zhang, D. S. Reeves, P. Ning, and S. P. Lyer. Analyzing network traffic to detect self-decrypting exploit code. In Proceedings of the 2nd ACM Symposium on Information, Computer and Communications Security (ASIACCS), 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Comprehensive shellcode detection using runtime heuristics

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Other conferences
        ACSAC '10: Proceedings of the 26th Annual Computer Security Applications Conference
        December 2010
        419 pages
        ISBN:9781450301336
        DOI:10.1145/1920261

        Copyright © 2010 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 6 December 2010

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

        Acceptance Rates

        Overall Acceptance Rate104of497submissions,21%

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader