Some of the biggest names in the anti-virus industry have flunked detection tests for known malware samples on Windows Vista Service Pack 1.
Seventeen of 37 anti-malware products pitted against “in the wild” viruses on the latest version of Vista failed to obtain VB100 certification, an industry benchmark used to rate product quality, according to test results released by Virus Bulletin.
Among the software products failing to nab VB100 certification were McAfee VirusScan Enterprise, Trend Micro Internet Security, Norman Virus Control, Sophos Anti-Virus, Webroot Spy Sweeper with AntiVirus, Alwil Software’s Avast, BitDefender Antivirus 2008, PC Tools AntiVirus and VirusBuster Professional.
To gain VB100 from Virus Bulletin’s testers, a product must detect 100 percent of malicious Trojans, bots and viruses from a batch of “in the wild” samples maintained in the WildList Organization International’s database. The WildList contains a listing of viruses collected and reported by virus hunters during actual computer attacks.
The basic requirements for a VB100 passing grade are that a product detect, both on demand and on access, in its default settings, all malware known to be in the wild at the time of the review, and generate no false positives when scanning a set of clean files.
For live coverage of this year’s RSA Conference, click here.
However, as the results show, several brand-name anti-malware labs are still missing virus samples linked to known attacks.
In McAfee’s case, for example, the company’s VirusScan Enterprise 8.5.0i was described as simple and dependable with solid integration of Windows Vista’s UAC (User Account Control) feature. Virus Bulletin’s John Hawes said the product’s detection rates were “dependably excellent” during the tests until a single sample of the W32/Virut strain reared its ugly head.
Since that sample was in the WildList set, that was enough to deny McAfee a VB100 award, Hawes explained.
Trend Micro Internet Security, a three-user anti-malware product that retails for $49.99, also scored well on some detections but Hawes said some false positives led to the failing grade. “A small number of file infectors were missed in the WildList set and a couple of items in the clean set were labeled as ‘TROJ_Generic.’ As a result, Trend does not qualify for the VB100 award on this occasion,” he said.
Webroot Spy Sweeper with AntiVirus shares signatures with Sophos Anti-Virus; both failed because some samples of the tricky Virut variants were not detected.
While these results are a public relations embarrassment for the bigger anti-virus vendors, analysts say the results should be taken with a grain of salt.
“[T]here are a couple asterisks worth noting,” Paul Roberts, senior analyst in The 451 Group’s enterprise security research unit, wrote in a research note. “First of all: The platform in question-Vista SP1-was released shortly after the deadline for product submissions to VB. VB reviewer John Hawes … is up front about that fact that not every anti-malware vendor was even able to get a copy of SP1 for testing before submitting their wares to VB for certification.”
Roberts added, “Certifications like VB100, which are based largely on static file analysis, have gone a long way towards sustaining the signature-based detection model when others might serve consumers and enterprises better.”
He said most anti-virus companies already do blend behavior and signature-based detection methods, but warned that companies that rely heavily on the former, like BitDefender, tend to do worse on tests like the VB100.
“Does that mean BitDefender provides inferior protection to a company like, say Kingsoft, which did receive the award? Hardly, but the lack of certification still becomes a hook on which to hang competitive claims. Bottom line: You get punished for not using signatures, even if that’s the right or most effective thing to do,” Roberts said.
Roberts called for new testing methods to help “end the illusion of competence that current testing models perpetuate” and raise the bar for malware detection among established vendors.