One Virus Engine Is Not Enough
The Case for Maximizing Network Protection with Multiple Anti-Virus Scanners

• Introduction
• A Review of Current Anti-Virus Engine Tests
• Test Results in a Nutshell
• The Case for Using Multiple Engines
• Additional Considerations in GFI MailSecurity
• About GFI
• Acknowledgments
  
  
• Appendix A
• Appendix B
• Appendix C
• Appendix D

Introduction

All responsible organizations protect their networks from virus attacks by installing an email security product. Yet, how does one choose the right solution out of the wide variety of virus scanning engines available? And is one anti-virus engine enough to protect the internal network from mass-mailing viruses, worms and other email-borne threats?

The tests detailed in this paper show that each virus scanner presents its own strengths and weaknesses. This means that no single anti-virus engine can fully protect against all possible threats. As a result, simultaneous use of more than one virus engine can achieve greater security than is technically possible when relying on only one anti-virus engine. The use of multiple virus engines also enables security administrators to be vendor independent when it comes to virus scanning, thereby able to use the best of breed virus engines available on the market.

Note: This paper does not cover desktop virus scanners. Its aim is to feature several popular virus-scanning engines and highlight the differences between each.

A Review of Current Anti-Virus Engine Tests

This paper examines the research currently available on leading anti-virus engines - namely, those developed by Trend Micro, Norton, Bit-Defender, McAfee and Norman - and studies their performance in three key areas:

• Overall detection rates of in the wild and zoo viruses;
  
• Their ability to scan through compressed and embedded files; and the coverage of non-virus malware

Detailed results for each set of tests are found in Appendices A, B and C respectively.

The results compiled in this paper are based on tests conducted by these anti-virus testing laboratories:

ICSA Labs - ICSA certification is regarded as the guarantee that a certain product is top notch and assures customers that the product has succeeded in a number of stringent tests.

West Coast Labs - The West Coast Checkmark has been developed as an independent testing and standards organization. Checkmark-certified products and services can be relied upon to an identified standard.

Virus Bulletin - The Virus Bulletin 100% award is awarded to those anti-virus products that detect all in the wild viruses during both on-demand and on-access scanning during testing.

AV-Test.org - This German organization of the University of Magdeburg consistently tests anti-virus software on behalf of companies and leading IT publications for client, server, UNIX and groupware products.

Virus TestCenter - The University of Hamburg Computer Science Department runs tests on anti-virus products and publishes the results in its Virus TestCenter, with an emphasis on the detection of zoo viruses.

For further information on each lab, please see Appendix D.

Test Results in a Nutshell

Considered together, the various test results show that no single anti-virus engine can fully protect against all possible threats (see Appendices A, B and C for full results).

For instance, Trend Micro does not scan ACE, B2 or TGZ compressed files, and it does not detect viruses compressed with the increasingly popular UPX. However, it excels in the MS Office files area, capturing all OLE objects embedded in such files in the AV-Test.org tests. Trend Micro's products also obtained good (but not full) results with non-virus malware.

While Norton AntiVirus achieves a good rate at detecting both ITW and zoo viruses, it fails to detect viruses compressed with packages such as UPX, Shrink, and ASPack. In the tests, it achieves an average detection rate of 75% of backdoors and Trojan files.

McAfee VirusScan yielded different results from different testing organizations regarding detection of ITW viruses on different platforms. According to the AV-Test.org tests of November 2001, VirusScan caught 99.5% of the in the wild (ITW) viruses. This product does not support compression formats RAR or ACE and does not detect viruses compressed with UPX and other similar products. However McAfee achieved good results in the non-virus malware section (ActiveX, backdoors and Trojans).

Norman's main strength seems to be in maintaining a high rate at detecting ITW and zoo viruses. However Norman is less powerful when the viruses are compressed with formats other than ZIP and ARJ, or using any self-extracting (SFX) archiving method such as WinZip.

BitDefender by SOFTWIN supports several compression formats like ACE, ARJ, RAR and ZIP. It also checks through files packed using popular packaged such as UPX, Neolite and ASPack. Yet, it missed one ITW file virus and caught 92% of all zoo viruses on test.

The Case for Using Multiple Engines

Given the inability of any individual anti-virus engine to provide full coverage against all email attacks, logic dictates that combining multiple engines will produce a more complete solution. In simple terms, if anti-virus products X and Y - each stronger in one area but weaker in another - are used together, their joined strength is likely to cover a wider range of security areas, and this way they can counteract each other's weak points.

Further analysis shows the validity of this theory. The tables below use data from the AV-Test.org tests of November 2001 to show the impact of using two or three virus scanning engines to increase protection.

Email security product A1 with Norman and BitDefender engines installed

This email product "A1" would cover 100% of ITW viruses, between 56.8% and 78.4% of the most popular compression methods, and 84% - 100% of samples from the "other malware" section.

Email security product A2 with McAfee, Norman and BitDefender engines installed

This product "A2" would cover 100% or ITW viruses, about 56.8% or more of the most popular compression methods, and 98.7% - 100% of samples from the "other Malware" section.

Another email security product - "B - uses the Norton virus-scanning engine. The table below shows the total coverage with this product:

A fourth email security product, "C" uses Trend's anti-virus engine, with the following results.

Comparing these four products, we notice that A2 has an advantage over the rest of the products, with A1 next on the performance list.

The table below gives a closer view of the compression area (where virus scanners tend to differ greatly in performance):

Here, one sees how BitDefender covers many more of the compression formats than the rest of the virus-scanning engines being used in this analysis. The email security product A2 would therefore provide a much more complete solution than a product making use of a single virus scanner.

As of the date of this writing, GFI MailSecurity for Exchange/SMTP was virtually the only product on the market to provide support for multiple virus engines, and the only one to offer the comprehensive protection of McAfee, Norman and BitDefender illustrated here.

Additional Considerations in GFI MailSecurity

While anti-virus protection is a critical component in protecting a network from email-related threats, virus protection alone cannot fully safeguard networks from email assaults. The fact that virus scanners only cover a portion of non-virus threats is well known. Therefore a fuller email security product should include features that protect against email-borne security threats apart from viruses, as well as multiple virus scanners.

Again, GFI MailSecurity for Exchange/SMTP provides a solution. In addition to the unusual simultaneous use of multiple virus engines discussed above, MailSecurity provides Email content & attachment checking - to quarantine dangerous emails; Exploit shield - to provide mail intrusion detection and defense; and an Email threats engine - to analyze and defuse HTML scripts, .exe files and more. This combination of features is unique in the industry, providing maximum protection against email-related network assaults. Other features of GFI MailSecurity include:

• Automatic removal of HTML scripts
  
• Automatic quarantining of Microsoft Word documents with macros
  
• Detects attachment extension hiding
  
• Rules-based configuration
  
• Apply rules to AD users or groups
  
• Approve/reject quarantined mail using the moderator client/email client/public folders
  
• Lexical analysis
  
• Seamless integration with Exchange Server 2000 through VS API
  
• Excellent value

GFI MailSecurity for Exchange/SMTP can be deployed at the gateway level, or at the information store level (based on the Exchange 2000 VS API). An evaluation version can be downloaded from: http://www.gfi.com/mailsecurity

About GFI

GFI has six offices in the US, UK, Germany, France, Australia and Malta, and has a worldwide network of distributors. GFI is the developer of FAXmaker, MailSecurity, Mail essentials and GFI LANguard, and has supplied applications to clients such as Microsoft, Telstra, Time Warner Cable, Shell Oil Lubricants, NASA, DHL, Caterpillar, BMW, the US IRS, and the USAF. GFI is a Microsoft Gold Certified Partner and has won the Microsoft Fusion 2000 (GEM) Packaged Application Partner of the Year award.

Acknowledgments

Special thanks to Andreas Marx [amarx@gega-it.de] of AV-Test.org for his help and contributions to this paper.

© 2002 GFI Software Ltd. All rights reserved. The information contained in this document represents the current view of GFI on the issues discussed as of the date of publication. Because GFI must respond to changing market conditions, it should not be interpreted to be a commitment on the part of GFI, and GFI cannot guarantee the accuracy of any information presented after the date of publication. This White Paper is for informational purposes only. GFI MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. FAXmaker, Mail essentials, MailSecurity and GFI LANguard and the FAXmaker, Mail essentials, MailSecurity and GFI LANguard logos and the GFI logo are either registered trademarks or trademarks of GFI Software Ltd. in the United States and/or other countries. Microsoft, Exchange Server, VS API, Word, and Windows NT/2000/XP are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other product or company names mentioned herein may be the trademarks of their respective owners. GFI. http://www.gfi.com info@gfi.com 1-888-2GFIFAX / +44-(0)20-8546 0640

APPENDIX A
Overall ITW and Zoo Virus Detection Rates

Because all anti-virus tests feature specific products such as antivirus products for Microsoft Exchange Server, or desktop anti-virus scanners, there are no generic anti-virus scanning engine tests available. Anti-virus engines on different platforms usually show small variations in performance. Sometimes anti-virus products for servers also make use of older engines that can make a difference in the AV test results.

In the wild (ITW) viruses are those reported by two or more researchers, meaning that these viruses have been reported as spreading in the real world, rather than just in a laboratory environment (see http://www.wildlist.org/ for more information.) Zoo viruses are those held in a collection by a virus researcher or testing agency. These viruses also may or may not be actively circulating.

The following results draw from tests performed by a number of anti-virus testing labs. (For more info on each of the testing labs, see Appendix D). Those virus scanners that hold ICSA and/or West Coast Labs Checkmark and/or Virus Bulletin (VB100%) certifications are certified to capture 100% of all wild list viruses as well as satisfying other security criteria.

Here is a summary of the findings per engine:

Trend Micro
Trend Micro offers a variety of products such as InterScan VirusWall, ScanMail for Exchange and Trend Virus Control System. Most of Trend Micro's products hold both ICSA and Checkmark certifications.

According to AV-Test.org (tests of July, September and November 2001), both PC-cillin (NT/2000/XP) and ScanMail (Exchange 2000 and Lotus) achieved 100% detection in their tests. Trend Micro's products were not submitted to Virus TestCenter (VTC); the VTC site claims this is because Trend Micro's scanner is deliberately trimmed to on-access scanning and detection of ITW viruses.

Trend Micro's PC-cillin has only recently been submitted to Virus Bulletin for review. It holds 2 consecutive VB100% awards. According to the February 2002 edition of Virus Bulletin, the tests did not produce any false positives. It produced a single false negative in the macro scanning segment. However, Trend Micro has a couple of misses in non-ITW polymorphic viruses segment.

Symantec Norton AntiVirus
Most Norton AntiVirus products hold ICSA and Checkmark certifications, meaning that these products detect 100% of all ITW viruses as well as satisfying a number of other criteria. This is verified by AV-Test.org tests (tests of November 2001, client tests of October 2001 and groupware tests of September 2001) with 100% detection rate of DOS, Windows as well as macro and script viruses. Norton AntiVirus achieved the same result in the VTC tests.

Norton AntiVirus 2002 was also awarded 16 VB100% awards between January 1998 and February 2002, making it one of the top rankers on the VB100% scale, along with Norman (16 awards) and ESET NOD (17 awards).

Symantec Norton AV achieves a very good rate at detecting almost all viruses, be they ITW or Zoo viruses.

McAfee VirusScan
VirusScan and NetShield by McAfee hold ICSA certification for on-demand/on-access and the West Coast Labs Checkmark. The reports for McAfee VirusScan on AV-Test.org (October 2001) show 99.5% detection rate for ITW viruses. VirusScan Corp achieved a 100% detection rate. This is very similar to the AV-Test.org results for zoo on-demand tests: VirusScan for WinNT/2000/XP achieved a 99.6% detection rate, and VirusScan Corp for WinNT/2000/Xp achieved a 99.9% detection. Such results show the discrepancies that can arise between different products using the same engine.

AV-Test.org tests on Antigen (Exchange 2000) using the McAfee engine produced a 99.8% rate of detection of ITW viruses.

In tests run by VTC (April 2001), McAfee achieved an average detection rate of 94.1% for zoo file viruses under DOS. It holds first place in the DOS, WinNT, Windows 98, and Windows 2000 anti-virus and anti-malware products section.

Norman
Norman Virus Control for Windows 95/98 is ICSA certified. Various versions of NVC also hold the West Coast Labs Checkmark.

Norman is one of the products to achieve the greatest number of VB100% awards (16 in all), meaning that it has long stood firm against ITW viruses.

AV-Test.org (tests of October and November 2001) also reports that NVC scored 100% on all ITW virus detection. Norman achieved a 97.1% detection rate in the October 2001 zoo on-demand tests and a 96.9% detection rate in the November 2001 tests. The tests done on groupware software by AV-Test.org in September 2001, resulted in Norman engine under Antigen catching 98.8%.

BitDefender
This product by SOFTWIN is certified by the ICSA labs.

Although BitDefender, formerly known as AVX, has not applied for any VB100% awards, it was named "perfect ITW scanner" in the VTC Windows 98 anti-virus tests of November 2001.

In the November 2001 tests conducted by AV-Test.org on, BitDefender achieved 99.8% detection of all ITW viruses and 92.1% of all zoo viruses.

APPENDIX B
Scanning through compressed and embedded files

AV-Test.org tests anti-virus scanners for their ability to scan emails that are compressed using different methods and for virus files embedded in other file formats.

The most popular compression methods tests by AV-Test are for these formats: ZIP, RAR, LHA, CAB, ARJ, ACE. Also tested are executable files that make use of these compression types and other compression methods such as ASPack. Because most of these compression methods allow the use of password protection, AV-Test.org also tests virus scanners to see if this prevents them from scanning the compressed files, and whether they issue an alert that the compressed file is password protected.

The tests check for UNIX file compression methods like GZ, TAR and BZ2, as most of these formats are opened using popular tools like WinZip and WinRAR.

AV-Tests.org also checks anti-virus engine performance with self-extracting (SFX) archives, that is, archives merged with an executable module that is used to extract files from the archive when executed. This means that no external programs are needed to extract the contents of an SFX archive; running it is enough.

Compression is probably the best well-known way to circumvent certain anti-virus products. Another way is to embed OLE objects (i.e., a virus) within Microsoft Office files. Many anti-virus products now also support checking for OLE objects within these files. AV-Test.org tests use of multiple OLE file type objects such as EXE and DOC viruses, as well as different types of MS Office formats, such as DOC and XLS. These tests also cater for DOC files are saved as HTML files, which could also contain macro viruses.

Here is a summary of the findings per engine:

Trend Micro
Trend Micro's PC-cillin, ScanMail for Exchange and SecureQ (using Trend Engine) do not check for files that are compressed using the ACE format. However, all three check the rest of the Windows compression formats, namely ARJ, CAB, LHA, RAR and ZIP. They also scan for nested packed archives in these file compression formats.

PC-cillin and ScanMail/SecureQ using Trend Engine virus-scan the UNIX formats GZ, TAR and TGZ but not the B2 and TBZ formats.

All three supports the ARJ, LHA and WINZIP self-extracting archive formats. However, they do not check for viruses compressed using ACE, RAR16 and RAR32 formats.

Compressed viruses protected using a password escape PC-cillin and ScanMail for Exchange. However SecureQ using Trend Engine issues alerts on compressed files that make use of a password, this probably being a feature added by SecureQ rather than the Trend engine.

PC-cillin, SecureQ using the Trend engine and ScanMail for Exchange catch viruses compressed using LzExe, Neolite and PkLite. However, they failed tests using other products, such as ASPack, PEPack, Petite, UPX, and some other compression packages. SecureQ using the Trend Engine also did not find viruses compressed using Neolite.

PC-cillin supports the detection of viruses that are embedded inside MS Office DOC, XLS, PPT, and RTF files, etc. It detects the various combinations tested by AV-Test.org, such as VBS viruses embedded inside DOC files, and EXE files embedded inside XLS files. Trend Micro's engine also supports password-protected DOC/XLS files for Office 95, 98 and 2000. The mail products that use this same engine - ScanMail for Exchange and SecureQ - obtained the same results in these tests.

Symantec Norton AntiVirus
Symantec supports the Windows compression formats ARJ, CAB, LHA, and ZIP and the UNIX formats GZ, TA and TGZ. Norton does not detect viruses compressed with ACE, RAR, B2 and TBZ. Norton AntiVirus can also scan recursively compressed file formats for ARJ, CAB, LHA, and ZIP.

Norton AntiVirus supports self-extracting archives created using WinZip and LHA. However it failed the tests using ACE, RAR16/32, or ARJ. Norton AntiVirus does not issue a warning when it encounters a password protected compressed file.

The product catches executable viruses compressed using LzExe and Neolite. Yet it failed to catch viruses compressed by popular executable compression programs such as UPX, Shrink and ASPack.

Norton AntiVirus detects viruses within DOC, PPT, RTF, SHS and XLS files. However it does not support DOC-MSO, PPT-MSO or XLS-MSO. It supports MSO-HTML DOC and XLS files (but not PPT MSO-HTML). MSO-HTML is an HTML file that can be opened and interpreted using MS Office applications such as Microsoft Word and Excel.

Norton AntiVirus scans password protected DOC files for Office 97 and 2000 but not DOC files created using Office 95. Norton scans XLS files created using Office 95/97/2000 for OLE objects even if the document is password protected.

McAfee VirusScan
McAfee can detect viruses within compression formats CAB, LHA and ZIP. It does not support virus scanning of the ACE, ARJ and RAR formats and the UNIX file compression formats. (The results of these tests depend on the McAfee product being tested e.g., the command line version of McAfee is more powerful than the graphic user interface - GUI - version because the latter does not support all the features available in the McAfee virus scanning engine.)

VirusScan succeeds in scanning compressed files within compressed files for the ZIP and LHA formats. It supports CAB files but not if they are found within another compressed file.

In the SFX testing conducted by AV-Test.org, McAfee detected the LHA format and executable viruses compressed using Neolite. It did not detect viruses compressed using the rest of the tested runtime compression packages, such as UPX and ASPack.

VirusScan scans objects in DOC, PPT, RTF, SHS and XLS files, objects within MSO-HTML files and DOC and XLS files protected by a password. It does not scan DOC-MSO and other non-HTML MSO file types.

Norman
Norman Virus Control (NVC) supports scanning ZIP and ARJ files as well as multiple packed archives in these formats. It disregards other Windows and UNIX compression formats such as ACE and TAR. In the tests by AV-Test.org, NVC did not detect viruses within any SFX archives created using packages such as WinZip and did not issue an alert when encountering a password protected compressed file.

NVC supports scanning through runtime compressed files using Ice, LzExe, PkLite and Neolite but does not detect viruses compressed with UPX, Shrink and other such packages.

NVC scans for viruses embedded as objects within DOC, PPT and XLS files. It scans SHS files for DOC, XLS, VBS and PPT files as embedded objects and DOC files embedded within MSO-HTML. It fails to scan certain OLE objects and MS Office formats such as EXE files embedded within a DOC file.

This anti-virus engine scans all 95, 98, and 2000 Office documents as tested by AV-Test.org, except DOC files for Office 95.

BitDefender
BitDefender scans through all Windows compression formats on test. This means that it catches ITW viruses compressed using ACE, ARJ, CAB, LHA, RAR and ZIP formats. It also supports the TAR and GZ UNIX compression formats.

BitDefender also scans recursive compressed files for Windows compression formats. It only supports RAR16 out of the SFX archives on test, meaning executables created using WinZip and other SFX utilities are not scanned for compressed viruses.

BitDefender supports runtime compression packages Ice, LzExe, ASPack, Neolite, PEPack and UPX but does not issue a warning when it encounters a password protected compressed file.

Embedded MS Office OLE Objects in DOC, SHS, and XLS files are scanned even if they are password protected. However viruses embedded within PPT, RTF, DOC/PPT/XLS-MSO or MSO-HTML are not detected.

APPENDIX C
Coverage of Non-Virus Malware

Most modern anti-virus scanners do not only scan for viruses. Due to the increase of non-virus malware, anti-virus scanning engines now also catch a variety of software that is found in the wild and used by attackers to try to infiltrate a system. Although anti-virus scanning engines cover these threats, most do not devote as many resources towards research in this field as they do towards virus research. Therefore the anti-virus engines may show huge differences in test results as compared to the results achieved by all in tests for ITW viruses, for example.

Having said that, determined attackers also try to circumvent the kind of protection offered by anti-virus engines by making use of undetected/non-ITW backdoor software. More details about this can be found in another GFI white paper about email exploits (see http://www.gfi.com/mailsecurity/wpexploitengine.htm).

Here is a summary of AV-Test.org's findings on the performance of each engine when facing non-virus malware:

Trend Micro
PC-cillin caught 100% of the eight malicious ActiveX controls on test. Out of 1,252 backdoors on test at AV-Test.org, it caught 1,239 of these, making the detection rate of 98.96%. Similarly, in the tests using Trojans, it caught 98.76%.

Symantec Norton AntiVirus
Norton AntiVirus caught all the malicious ActiveX controls on test. It caught 74.28% of the backdoors and 75.64% of the Trojans.

McAfee VirusScan
McAfee's VirusScan caught 100% of the eight malicious ActiveX controls on test. It caught 99.2% of the backdoors on test and 98.32% of the Trojans.

Norman
Norman missed one of the malicious ActiveX controls used in the tests. As regards the backdoors testing, it achieved 99.2%, reaching 98.32% in the Trojan horse testing.

BitDefender
BitDefender detected 7 out of the 8 ActiveX control tests. It caught 49.36% of the backdoors and 62.47% of the Trojans.

APPENDIX D
Overview of Anti-Virus Testing Laboratories

Individual commercial and non-commercial organizations perform periodic tests to determine several factors regarding anti-virus products. While some labs make their findings available to the general public in full, others simply award or certify the product. Most labs perform their tests periodically due to the active nature of computer viruses and malicious code. Leading anti-virus testing labs include:

ICSA Laboratories
ICSA certification is regarded as the guarantee that a product is top notch. It is awarded if a certain number of tests are passed. ICSA Labs perform periodic tests of anti-virus software based on different criteria and product types:

• On-Demand/On-Access Anti-Virus Product Certification
  
• Anti-Virus Product Cleaning Certification
  
• Anti-Virus Products for Internet Gateway E-mail Certification
  
• Anti-Virus Products for Microsoft Exchange Server Certification
  
• Anti-Virus Product for Lotus Notes Certification
  
• Anti-Virus Product for Security Service Providers
  
• Anti-Virus Product for Internet Service Providers
  
• On-Line Anti-Virus Scanners

ICSA certification is awarded to companies that satisfy a good number of standards in the AV tests.

West Coast Labs
The West Coast Lab's Checkmark has been developed as an independent testing and standards organization delivering a high degree of confidence that Checkmark-certified products and services can be relied upon to an identified standard.

West Coast Labs test products at regular intervals against set testing criteria.

Virus Bulletin
Virus Bulletin is a highly reputable and respected technical journal covering the tracks of computer viruses and anti-virus products. The Virus Bulletin 100% award is given to those anti-virus products that detect all In the Wild (ITW) viruses during both on-demand and on-access scanning during testing. These tests are widely recognized and therefore many customers make their choice based on this certification. ITW viruses are computer viruses that have been reported as spreading in the real world, rather than just in laboratory environments. This means that the tests conducted can be relied on. Test results are published in the Virus Bulletin journal and cover different products from participating vendors.

AV-Test.org
AV-Test is a project of the Business-Information-Workgroup at the Institute of Technical and Business Information Systems at the Otto-von-Guericke University in Magdeburg (Germany) in cooperation with GEGA IT-Solutions GbR. This organization consistently tests anti-virus software on behalf of companies and leading IT publications. Test results and the full details are freely available on the organization's website.

Virus TestCenter
The University of Hamburg Computer Science Department runs tests on anti-virus products and publishes the results in its Virus TestCenter. Full research results are published on the organization's website and are available to the general public. Virus TestCenter emphasizes the detection of zoo viruses, that is, viruses held in a collection by virus researchers or testing agencies; zoo viruses may or may not be actively circulating.

More information on each lab is available at each organization's web site:

AV-Test.org - http://www.av-test.org/
ICSA Labs - http://www.icsalabs.com/
Virus Bulletin - http://www.virusbtn.com/
Virus TestCenter - http://agn-www.informatik.uni-hamburg.de/vtc/
West Coast Lab - http://www.check-mark.com/

back to top