Security software maker Secunia has published an interesting new piece of research that pits a dozen well-known AV and Internet security packages against live exploit testing to see how they fare in detecting the threats.
Based on the business proposition of the software company that I work for in my day job, I’ll avoid any philosophical debate over the value of exploit testing versus more traditional methods of AV assessment, and some observers are questioning the validity of Secunia’s findings (PDF) based on such issues, but I do think it’s an intriguing exercise at the very least.
Secunia’s methodology was to see if the security applications could detect active exploitation of available vulnerabilities since that’s how attackers would approach the scenario if they were trying to circumvent one of the products.
I mean, maybe you can argue that there are better ways of carrying out this research, but it is a pretty valid real-world approach IMHO. (But again, really, based on who I work for it should be.)
Anyway, here’s what the researchers themselves said:
“For a long time, we have been quite convinced that anti-virus products would exhibit poor performance in this discipline, given the name ‘anti-virus’ which suggests a limited focus (though customers may still expect to be protected). This is why we decided to test some more ‘high-end’ product bundles that are being marketed as comprehensive Internet Security Suites, thus leaving the impression that the user is ‘fully protected against all Internet threats.”
So Secunia tested the following twelve Internet Security Suites against live exploit code:
-McAfee Internet Security Suite 2009 -Norton Internet Security 2009 -Windows Live OneCare -ZoneAlarm Security Suite 8 -AVG Internet Security 8.0 -CA Internet Security Suite 2008 -F-secure Internet Security 2009 -TrendMicro Internet Security 2008 -BitDefender Internet Security Suite 2009 -Panda Internet Security 2009 -Kaspersky Internet Security 2009 -Norman Security Suite 7.10
The exploits themselves were developed in-house by Secunia and based on analysis of vulnerabilities that have been reported to security vendors over the last two years, and included:
-Proof of Concept (PoC) – The purpose of a PoC is to just trigger the vulnerability. It does not carry a payload. If a security product can reliably detect a PoC, then it can detect all attempts to exploit the vulnerability independent of the payload.
-GameOver PoC – The purpose of a GameOver PoC is to prove that code execution is possible by gaining control of the program flow, without actually launching any code.
-Exploit – Exploits carry a payload and will execute it if used against a vulnerable application.
And as the company points out: “In real life, an attacker would always use an exploit. However, if a security product cannot detect a PoC it also cannot detect an exploit reliably.”
So how did the security packages fare?
“The results clearly show that the major security vendors do not focus on vulnerabilities. Instead, they have a much more traditional approach, which leaves their customers exposed to new malware exploiting vulnerabilities,” Secunia said. “One could argue that this isn’t a problem, since no single product can offer a 100 percent protection. Yet, many of these suites clearly indicate that they are comprehensive and offer protection against ‘all’ Internet threats, thus many users would rightfully expect these suites to protect them against all current threats.”
The combination of over-reliance on AV and poor patching by most organizations and/or users “leaves the door wide open for professional Internet criminals,” the report concludes.
Of all the products, the company did single out Kaspersky and BitDefender for having onboard vulnerability scanning capabilities, which it said would help, but not solve, the exploitation issue.
The major takeaway was that people should really spend more time making sure they are fully patched and install multiple AV scanners to catch the broadest array of attacks.
Of course, Secunia could have also pointed out that something like an automated penetration testing solution might prove valuable within the context of such a discussion.
Oops, did I really just say that? ;)
Matt Hines has been following the IT industry for over a decade as a reporter and blogger, and has been specifically focused on the security space since 2003, including a previous stint writing for eWeek and contributing to the Security Watch blog. Hines is currently employed as marketing communications manager at Core Security Technologies, a Boston-based maker of security testing software. The views expressed herein do not necessarily represent the views of Core Security, and neither the company, nor its products and services will be actively discussed in the blog. Please send news, research or tips to [email protected].