Only the Unknown Files Get Scanned

By Larry Seltzer  |  Posted 2008-11-07 Print this article Print


As a result, only unknown files need to be scanned, but as I said there are logical limits to that. Not finding malware in a file doesn't prove it's not malware. So now Symantec has added the concept of file reputation.

Imagine an asymptotic curve representing the prevalence of a particular file in the world. Well-known and legitimate files, like EXCEL.EXE, will be on the high side of the curve on the left. Further down are less-common files and the very popular malware. Further down, on what Symantec calls the "long tail" of the curve, are files that appear very infrequently. They may be legitimate, or they may be a new strain of malware with only a dozen copies in the world. The vast majority of new malware instances are on this part of the curve.

To get reputation data on files as users get them, Symantec gathers data from its customers who have opted in to what it calls the Norton Community Watch to send extra data back to the company. Symantec's not especially forthcoming about the algorithm for calculating reputation, but it's not hard to figure out some of the factors that would go into it.

Let's say a new file goes onto a system or a new Web site is visited. If no other checks, like blacklists, come up with anything and there is no existing reputation on the file, then an SHA256 is calculated, and it and some other data is uploaded to the Norton cloud. Here, according to Symantec, if the file is not in the database already, mysterious statistical formulae work their magic and emit a reputation number. The initial analysis may also include a more detailed heuristic analysis than is done on the PC. As other things happen on the systems on which these files exist, the reputation may enhance or suffer as a result.

This reputation system has actually been in effect for a year, and Symantec says 18 million users have it in their products. That's a pretty big sample. Does it work? That's something for testing to decide, and whatever their other problems, Symantec malware detection is always first-rate in any reputable test I've seen.

Symantec's approach makes sense to me, but it makes me shudder over the complication of it all. So many ways to check files, so many different inputs into the process. A simple solution like absolute whitelisting would be appealing, were it possible.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

For insights on security coverage around the Web, take a look at Security Center Editor Larry Seltzer's blog Cheap Hack.

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel