Untangle Weaves Controversial Web With AV Test

Vendors contest the results of tests that pitted their anti-virus products against open source Clam AntiVirus.

It seems round two of the Untangle Anti-Virus Fight Club has begun.

McAfee officials and others are taking issue with the methodology of a live test that pitted proprietary anti-virus products from vendors such as McAfee and SonicWALL against the open source project ClamAV at LinuxWorld. The test was the brainchild of Untangle Chief Technology Officer Dirk Morris, who has said ClamAV was not getting treated fairly by testers.

His answer was the AV Fight Club, which involved products from McAfee, SonicWALL, Kaspersky Lab, Symantec, Sophos, Fortinet, FRISK Software International, WatchGuard Technologies and HAURI. Untangle, which provides an open source network gateway platform, uses ClamAV for its anti-virus protection.

The test consisted of three sets of viruses. The first batch was a basic test set from eicar.org that Morris described in a blog as a universal virus test. The second set was the "in-the-wild" test of viruses picked from Morris mailbox that he had received over the years in mass quantities, and the third group of viruses was submitted by users.

Heres what the study found: only ClamAV, Norton Antivirus 2007 and Kasperskys offering caught 100 percent of the viruses in the first two categories, and they were the top three in overall percentage. Sophos, FRISK and McAfee ranged in the 80 to 90 percent catch rate for the first two groups and 85.7, 85.7 and 74.3 percent overall respectively.


To read more about Kaspersky Labs new anti-virus tool, click here.

HAURI and the gateway appliances from SonicWALL and Fortinet caught about 60 percent of the viruses in the first two groups, though Fortinet and HAURI only caught 45.7 percent of the viruses overall. SonicWALL fared slightly better with a 54.3 percent overall catch rate.

Coming in last was WatchGuard, catching 5.6 percent in the first two virus-sets and 2.9 overall – an interesting finding since WatchGuard uses ClamAV.

"This to me, and many others, clearly suggests a problem with configuration," wrote security researcher David Harley in a white paper entitled "Untangling the Wheat from the Chaff in Comparative Anti-Virus Reviews".

"The wide variation in detection rates compared to the comparatively narrow ranges in most professional tests may also reflect configurational inconsistency, as well as an unreliable sample set."

Harley, head of a UK-based IT publishing firm called Small Blue-Green World, wrote that the testers seemed to fail to understand the importance of establishing a level playing field in terms of configuration, such as the level of heuristics set and archive scanning.

Page 2: Untangle Weaves Controversial Web With AV Test