article

Testing malware detectors

Authors:

Mihai Christodorescu,

Somesh JhaAuthors Info & Claims

ACM SIGSOFT Software Engineering Notes, Volume 29, Issue 4

Pages 34 - 44

https://doi.org/10.1145/1013886.1007518

Published: 01 July 2004 Publication History

Abstract

In today's interconnected world, malware, such as worms and viruses, can cause havoc. A malware detector (commonly known as virus scanner) attempts to identify malware. In spite of the importance of malware detectors, there is a dearth of testing techniques for evaluating them. We present a technique based on program obfuscation for generating tests for malware detectors. Our technique is geared towards evaluating the resilience of malware detectors to various obfuscation transformations commonly used by hackers to disguise malware. We also demonstrate that a hacker can leverage a malware detector's weakness in handling obfuscation transformations and can extract the signature used by a detector for a specific malware. We evaluate three widely-used commercial virus scanners using our techniques and discover that the resilience of these scanners to various obfuscations is very poor.

References

[1]

D. Angluin. Learning regular sets from queries and counterexamples. Information and Computation, 75:87--106, 1987.

Digital Library

[2]

K. Brunnstein. "Heureka-2" AntiVirus Tests. Virus Test Center, University of Hamburg, Computer Science Department, Mar. 2002. Published online at http://agn-www.informatik.uni-hamburg.de/vtc/en0203.htm. Last accessed: 16 Jan. 2004.

[3]

T. Chen and Y. Yu. On the relationship between partition and random testing. IEEE Transactions on Software Engineering, 20(12):977--980, Dec. 1994.

Digital Library

[4]

S. Chow, Y. Gu, H. Johnson, and V. Zakharov. An approach to the obfuscation of control-flow of sequential computer programs. In G. Davida and Y. Frankel, editors, Proceedings of the 4th International Information Security Conference (ISC'01), volume 2200 of Lecture Notes in Computer Science, pages 144--155, Malaga, Spain, Oct. 2001. Springer-Verlag.

Digital Library

[5]

C. Collberg, C. Thomborson, and D. Low. A taxonomy of obfuscating transformations. Technical Report 148, Department of Computer Science, University of Auckland, New Zealand, July 1997.

[6]

C. Collberg, C. Thomborson, and D. Low. Breaking abstractions and unstructuring data structures. In Proceedings of the International Conference on Computer Languages 1998 (ICCL'98), pages 28--39, Chicago, IL, USA, May 1998. IEEE Computer Society.

Digital Library

[7]

C. Collberg, C. Thomborson, and D. Low. Manufacturing cheap, resilient, and stealthy opaque constructs. In Proceedings of the 25th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL'98), San Diego, CA, USA, Jan. 1998. ACM Press.

Digital Library

[8]

D. W. Cooper. Adaptive testing. In Proceedings of the 2nd International Conference on Software Engineering (ICSE'76), pages 102--105, San Francisco, CA, USA, Oct. 1976. IEEE Computer Society Press.

Digital Library

[9]

T. Detristan, T. Ulenspiegel, Y. Malcom, and M. S. von Underduk. Polymorphic shellcode engine using spectrum analysis. Phrack, 11(61), Aug. 2003. Published online at http://www.phrack.org. Last accessed: 16 Jan. 2004.

[10]

J. W. Duran and S. C. Ntafos. An evaluation of random testing. IEEE Transactions on Software Engineering, 10(7):438--444, July 1984.

Digital Library

[11]

J. E. Forrester and B. P. Miller. An empirical study of the robustness of Windows NT applications using random testing. In Proceedings of the 4th USENIX Windows Systems Symposium, pages 59--68, Seattle, WA, USA, Aug. 2000.

Digital Library

[12]

P. G. Frankl, R. G. Hamlet, B. Littlewood, and L. Strigini. Choosing a testing method to deliver reliability. In Proceedings of the 19th International Conference on Software Engineering (ICSE'97), pages 68--78, Boston, MA, USA, May 1997.

Digital Library

[13]

S. Gordon and R. Ford. Real world anti-virus product reviews and evaluations -- the current state of affairs. In Proceedings of the 19th National Information Systems Security Conference (NISSC'96), pages 526--538, Baltimore, MD, USA, Oct. 1996. National Institute of Standards and Technology (NIST).

[14]

D. Hamlet and R. Taylor. Partition testing does not inspire confidence. IEEE Transactions on Software Engineering, 16(12):1402--1441, Dec. 1990.

Digital Library

[15]

R. Hildebrandt and A. Zeller. Simplifying failure-inducing input. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis 2000 (ISSTA'00), pages 135--145, Portland, OR, USA, 2000. ACM Press.

Digital Library

[16]

R. Hildebrandt and A. Zeller. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering, 28(2):183--200, Feb. 2002.

Digital Library

[17]

ICSA Labs. Anti-virus product certification. Published online at http://www.icsalabs.com/html/communities/antivirus/certification.shtml. Last accessed: 16 Jan. 2004.

[18]

E. Kaspersky. Virus List Encyclopedia, chapter Ways of Infection: Viruses without an Entry Point. Kaspersky Labs, 2002.

[19]

LURHQ Threat Intelligence Group. Sobig.a and the spam you received today. Technical report, LURHQ, 2003. Published online at http://www.lurhq.com/sobig.html. Last accessed on 16 Jan. 2004.

[20]

LURHQ Threat Intelligence Group. Sobig.e -Evolution of the worm. Technical report, LURHQ, 2003. Published online at http://www.lurhq.com/sobig-e.html. Last accessed on 16 Jan. 2004.

[21]

LURHQ Threat Intelligence Group. Sobig.f examined. Technical report, LURHQ, 2003. Published online at http://www.lurhq.com/sobig-f.html. Last accessed on 16 Jan. 2004.

[22]

A. Marinescu. Russian doll. Virus Bulletin, pages 7--9, Aug. 2003.

[23]

A. Marx. A guideline to anti-malware-software testing. In Proceedings of the 9th Annual European Institute for Computer Antivirus Research Conference (EICAR'00), 2000.

[24]

A. Marx. Retrospective testing -- how good heuristics really work. In Proceedings of the 2002 Virus Bulletin Conference (VB2002), New Orleans, LA, USA, Sept. 2002. Virus Bulletin.

[25]

McAfee AVERT. Virus information library. Published online at http://us.mcafee.com/virusInfo/default.asp. Last accessed: 16 Jan. 2004.

[26]

G. McGraw and G. Morrisett. Attacking malicious code: report to the Infosec research council. IEEE Software, 17(5):33--41, Sept./Oct. 2000.

Digital Library

[27]

B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):12--44, Dec. 1990.

Digital Library

[28]

B. P. Miller, D. Koski, C. P. Lee, V. Maganty, R. Murthy, A. Natarajan, and J. Steidl. Fuzz revisited: A re-examination of the reliability of UNIX utilities and services. Technical Report 1268, University of Wisconsin, Madison, Computer Sciences Department, Madison, WI, USA, Apr. 1995.

[29]

G. J. Myers. The Art of Software Testing. John Wiley & Sons, first edition, Feb. 1979.

Digital Library

[30]

S. C. Ntafos. On random and partition testing. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis 1998 (ISSTA'98), pages 42--48, Clearwater Beach, FL, USA, Mar. 1998. ACM Press.

Digital Library

[31]

S. C. Ntafos. On comparisons of random, partition, and proportional partition testing. IEEE Transactions on Software Engineering, 27(10):949--960, Oct. 2001.

Digital Library

[32]

Symantec Antivirus Research Center. Expanded threat list and virus encyclopedia. Published online at http://securityresponse.symantec.com/avcenter/venc/data/cih.html. Last accessed: 16 Jan. 2004.

[33]

P. Ször and P. Ferrie. Hunting for metamorphic. In Proceedings of 2001 Virus Bulletin Conference (VB2001), pages 123--144, September 2001.

[34]

TESO. Burneye ELF encryption program. Published online at http://teso.scene.at. Last accessed: 15 Jan. 2004.

[35]

The WildList Organization International. Frequently asked questions. Published online at http://www.wildlist.org/faq.htm. Last accessed: 16 Jan. 2004.

[36]

Virus Bulletin. VB 100% Award. Published online at http://www.virusbtn.com/vb100/about/100use.xml. Last accessed: 16 Jan. 2004.

[37]

C. Wang. A security architecture for survivability mechanisms. PhD thesis, University of Virginia, Oct. 2000.

Digital Library

[38]

West Coast Labs. Anti-virus Checkmark level 2. Published online at http://www.check-mark.com/checkmark/pdf/Checkmark_AV1.pdf. Last accessed: 16 Jan. 2004.

[39]

West Coast Labs. Anti-virus Checkmark level 2. Published online at http://www.check-mark.com/checkmark/pdf/Checkmark_AV2.pdf. Last accessed: 16 Jan. 2004.

[40]

E. J. Weyuker and B. Jeng. Analyzing partition testing strategies. IEEE Transactions on Software Engineering, 17(7):703--711, July 1991.

Digital Library

[41]

G. Wroblewski. General method of program code obfuscation. PhD thesis, Institute of Engineering Cybernetics, Wroclaw University of Technology, Wroclaw, Poland, 2002.

[42]

z0mbie. Automated reverse engineering: Mistfall engine. Published online at http://z0mbie.host.sk/autorev.txt. Last accessed: 16 Jan. 2004.

[43]

z0mbie. z0mbie's homepage. Published online at http://z0mbie.host.sk. Last accessed: 16 Jan. 2004.

Cited By

G SFaloutsos M(2024)MalFusion: Simple String Manipulations Confuse Malware Detection2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619782(113-121)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619782
Singh DKaur HSajid SSagar G(2024)A Comprehensive Review of Android Malware Detection TechniquesE3S Web of Conferences10.1051/e3sconf/202455601008556(01008)Online publication date: 9-Aug-2024
https://doi.org/10.1051/e3sconf/202455601008
Geng JWang JFang ZZhou YWu DGe W(2024)A survey of strategy-driven evasion methods for PE malware: Transformation, concealment, and attackComputers & Security10.1016/j.cose.2023.103595137(103595)Online publication date: Feb-2024
https://doi.org/10.1016/j.cose.2023.103595
Show More Cited By

Index Terms

Testing malware detectors
1. Security and privacy
  1. Systems security
    1. Operating systems security
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Testing malware detectors
ISSTA '04: Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis

In today's interconnected world, malware, such as worms and viruses, can cause havoc. A malware detector (commonly known as virus scanner) attempts to identify malware. In spite of the importance of malware detectors, there is a dearth of testing ...
Revealing Packed Malware

In concert with the ever-growing network applications, a significant increase in the spread of malware over the Internet has been observed. In cases where malware are the zero-day threats, generating their signatures for detection via anti-virus (AV) ...
Obfuscated malware detection using API call dependency
SecurIT '12: Proceedings of the First International Conference on Security of Internet of Things

Malwares pose a grave threat to security of a network and host systems. Many events such as Distributed Denial-of-Service attacks, spam emails etc., often have malwares as their root cause. So a great deal of research is being invested in detection and ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGSOFT Software Engineering Notes

ACM SIGSOFT Software Engineering Notes Volume 29, Issue 4

July 2004

284 pages

ISSN:0163-5948

DOI:10.1145/1013886

Issue’s Table of Contents

ISSTA '04: Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis
July 2004
294 pages
ISBN:1581138202
DOI:10.1145/1007512
General Chair:
George Avrunin
University of Massachusetts, USA
,
Program Chair:
Gregg Rothermel
University of Nebraska -- Lincoln, USA

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2004

Published in SIGSOFT Volume 29, Issue 4

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

207
Total Citations
View Citations
3,941
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

G SFaloutsos M(2024)MalFusion: Simple String Manipulations Confuse Malware Detection2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619782(113-121)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619782
Singh DKaur HSajid SSagar G(2024)A Comprehensive Review of Android Malware Detection TechniquesE3S Web of Conferences10.1051/e3sconf/202455601008556(01008)Online publication date: 9-Aug-2024
https://doi.org/10.1051/e3sconf/202455601008
Geng JWang JFang ZZhou YWu DGe W(2024)A survey of strategy-driven evasion methods for PE malware: Transformation, concealment, and attackComputers & Security10.1016/j.cose.2023.103595137(103595)Online publication date: Feb-2024
https://doi.org/10.1016/j.cose.2023.103595
Graux PLalande JViet Triem Tong VWilke P(2023)OATs’inside: Retrieving Object Behaviors From Native-based Obfuscated Android ApplicationsDigital Threats: Research and Practice10.1145/35849754:2(1-27)Online publication date: 10-Aug-2023
https://dl.acm.org/doi/10.1145/3584975
Bhowmick RIndra RGanguli IPaul JSil J(2023)Breaking Captcha System with Minimal Exertion through Deep Learning: Real-time Risk Assessment on Indian Government WebsitesDigital Threats: Research and Practice10.1145/35849744:2(1-24)Online publication date: 10-Aug-2023
https://dl.acm.org/doi/10.1145/3584974
Bridges ROesch SIannacone MHuffer KJewell BNichols JWeber BVerma MScofield DMiles CPlummer TDaniell MTall ABeaver JSmith J(2023)Beyond the Hype: An Evaluation of Commercially Available Machine Learning–based Malware DetectorsDigital Threats: Research and Practice10.1145/35674324:2(1-22)Online publication date: 10-Aug-2023
https://dl.acm.org/doi/10.1145/3567432
Vombatkere NSrikaanth D(2023)A Novel Information Stealing Malware2023 International Conference on Networking and Communications (ICNWC)10.1109/ICNWC57852.2023.10127455(1-7)Online publication date: 5-Apr-2023
https://doi.org/10.1109/ICNWC57852.2023.10127455
Bridges RWeber BBeaver JSmith JVerma MNorem SSpakes KWatson CNichols JJewell BIannacone MStahl CHuffer KOesch T(2023)AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386590(1620-1629)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386590
Boyko VVasilenko MSlatvinska V(2023)Linked List Systems for System Logs Protection from CyberattacksInformation Technology for Education, Science, and Technics10.1007/978-3-031-35467-0_15(224-234)Online publication date: 18-Jun-2023
https://doi.org/10.1007/978-3-031-35467-0_15
Pružinec JNguyen QBaldwin AGriffin JLiu YKiss ÁMarín BSaadatmand M(2022)KUBO: a framework for automated efficacy testing of anti-virus behavioral detection with procedure-based malware emulationProceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation10.1145/3548659.3561307(37-44)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3548659.3561307
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents