Close
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Applications
    • Applications
    • Cybersecurity
    • Development
    • Networking

    Closing a Hole in AV Software Testing

    By
    Brian Prince
    -
    December 17, 2007
    Share
    Facebook
    Twitter
    Linkedin

      Behavioral-based detection has been a staple of anti-virus software for the past few years, but it remains absent from tests run by many labs rating the products. A group of security software providers and malware researchers are working to change that, but the question remains: Why has testing lagged so far behind the threat landscape?

      “I think it’s related to the time you need to perform a behavior-based test … and to the number of samples you can test against,” said Andreas Marx, CEO of AV-Test.org, one of the more widely used testing labs. “If you perform an in-the-wild test … like what most certification centers are doing, you need about 30 minutes to complete an on-demand and on-access test of 2,500 inactive samples. That’s quick. However, in the same time you might not be able to perform a test against a single active sample—and in order to get statistically meaningful results, you might need to test at least against 50 samples, better 100 samples.”

      Even then, the number of tested samples—50 to 100—appears to be low, and may not impress someone reading a report about the test, Marx said. In addition, the time required and costs associated with the testing are high, he said.

      For years, many testing labs have focused on signature-based detection capabilities, missing the behavioral aspect all together and making a true comparison of anti-virus products somewhat dubious.

      “It’s very hard to get an apples-to-apples comparison because everybody’s throwing different malware at it in different environments,” said Paul Roberts, an analyst with The 451 Group. “Even the way they are prepping the machines [and] how they are building the systems they are testing is different from trade pub to trade pub or testing lab to testing lab … so it’s very hard to get an apples-to-apples comparison.”

      Roberts described the anti-virus testing community as fairly conservative, adding that researchers take their time hashing over ideas and forming a consensus. Eric Howes, director of Malware Research at Sunbelt Software, had similar thoughts.

      “The problem is that testing methodologies become entrenched for several reasons,” Howes said. “First, obviously, is the reluctance to change something that has become comfortable and familiar. Second, spyware and the newer varieties of malware have been evolving so fast that they are effectively moving targets, making it more difficult to develop testing methodologies that adequately cover them. Third, it has been long apparent that any new testing methodologies for spyware and malware were going to involve work that was much more complicated and resource-intensive than what passed for AV and malware testing in the past.

      “Likewise,” he said, “it has proven harder to define basic standards and goals for such testing—much harder to define what constitutes a correct and satisfactory test—as the dynamics of the malware pose so many methodological challenges.”

      The challenges range from setting up the testing environment to what malware is chosen to the timeframe in which the tests are conducted in a world where servers blasting out malicious files are taken down seemingly as fast as they are set up. A group of security vendors and testers, including AV-Test.org, Symantec, Panda Security and others, in early December formed the Anti-Malware Testing Working Group to establish new guidelines for tests and do research.

      “It’s mainly a matter of time to get these new tests set up properly,” Marx said. “A clear methodology was one point which had to be solved. We’re now working on some better automation and helper tools to make the testing easier, faster and cheaper to perform.”

      Pedro Bustamante, senior research adviser at Panda Security, said he is hopeful new guidelines will push innovation, and added some companies may have to change their research, development and investment strategies around it.

      “Not all of the companies have done the necessary investments in developing new protection mechanisms that are able to cope with this new type of malware,” Bustamante said. “So there are some interests in keeping testing outdated in order to not make this situation apparent.”

      Still, all of the major security vendors use some form of behavioral-based detection, and Mike Montecillo, an analyst with Enterprise Management Associates, said those companies that haven’t put much of an emphasis on it are probably pursuing a broader security strategy beyond anti-virus and anti-malware products. In their own way, the new guidelines could help drive innovation, he said.

      “I definitely think there’s a market trend that’s already pushing innovation to a heuristic-based solution,” Montecillo said. “I think that signature-based algorithms will long be in place because they’re simple, they’re cost-effective and the technology behind it has already been developed. But what I think needs to happen is that that extra layer of heuristics-based detection algorithms need to be added onto the signature-based algorithms of the past.”

      Check out eWEEK.com’s Security Center for the latest security news, reviews and analysis. And for insights on security coverage around the Web, take a look at eWEEK’s Security Watch blog.

      Brian Prince

      MOST POPULAR ARTICLES

      Cybersecurity

      Visa’s Michael Jabbara on Cybersecurity and Digital...

      James Maguire - May 17, 2022 0
      I spoke with Michael Jabbara, VP and Global Head of Fraud Services at Visa, about the cybersecurity technology used to ensure the safe transfer...
      Read more
      Cloud

      Yotascale CEO Asim Razzaq on Controlling Multicloud...

      James Maguire - May 5, 2022 0
      Asim Razzaq, CEO of Yotascale, provides guidance on understanding—and containing—the complex cost structure of multicloud computing. Among the topics we covered:  As you survey the...
      Read more
      Big Data and Analytics

      GoodData CEO Roman Stanek on Business Intelligence...

      James Maguire - May 4, 2022 0
      I spoke with Roman Stanek, CEO of GoodData, about business intelligence, data as a service, and the frustration that many executives have with data...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Applications

      Cisco’s Thimaya Subaiya on Customer Experience in...

      James Maguire - May 10, 2022 0
      I spoke with Thimaya Subaiya, SVP and GM of Global Customer Experience at Cisco, about the factors that create good customer experience – and...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2021 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×