Lets Get Fuzzing

By Larry Seltzer  |  Posted 2007-04-27 Print this article Print

Opinion: When systems get as complex as Windows, the only way to keep up with the security flaws is with aggressive fuzz testing.

Ive said it before and Ill say it again: You just cant do enough testing of software systems. And as complexity of software systems rises, more testing is necessary. Do you think, for example, that a Boeing 747 is a complicated piece of machinery? I suspect it pales in comparison to Windows Vista for complexity. The recent ANI episode said a lot about the problem of complexity and testing. In fact, Microsoft does test aggressively and more than ever, and yet the ANI failure was more than anything else a failure of its testing.

There are many different kinds of testing you can do, but when it comes to the bizarre attacks of the ANI sort, nothing beats fuzzing. Microsoft knows this, and it has been an enthusiastic fuzzer for years.

Are you a security idiot, doing the wrong thing all the time? Click here to find out.

Fuzzing is a form of automated brute-force testing. You pick a protocol or a file format or an API, or whatever, and the fuzzer exercises it through a defined set of values or parameters. Intelligent fuzzing tests are somewhat picky about what they fuzz because you dont want to waste time testing cases that cant possibly occur. But picking the right set of cases can be hard, and you probably want to err more on the side of completeness than economy.

Whatever its motivations, it seems that Microsofts fuzzing cases were incomplete in the case of ANI files. Beyond Security, author of the beStorm testing tool, was curious about this too and wrote a quickie fuzzing module for the ANI file format. Glance at the module, which describes the ANI file format as input for beStorm to fuzz it. In short order it found the zero-day attack that caused all the ruckus.

How could Microsofts fuzzers have missed it? Actually, Microsofts Michael Howard just volunteered the answer to that question on the new Security Development Lifecycle blog:
It turns out none of the .ANI fuzz templates had a second "anih" record. This is now addressed, and we are continually enhancing our fuzzing tools to make sure they add manipulations that duplicate arbitrary object elements better.
Oopsie! This is especially sloppy in light of the fact that a previous severe vulnerability in the same format also involved multiple anih records.

Making the decision that something is not worth fuzzing is getting increasingly dangerous. Read the April 27 Symantec Security Response blog for an idea of how bizarre the attacks are getting. The malware writers smell blood in the water in the form of sloppy parsing and sloppy testing. You cant be too careful.

And fuzzing is a two-way weapon. Dont think that the people who found and exploited the ANI zero-day found it by flipping bits in a file by hand. They have their own fuzzers too, as do the vulnerability research companies like eEye and Determina (who found the ANI bug). This is why, with as much fuzzing as it already does, Microsoft needs to do more and more.

Heres a good candidate: Microsoft recently announced Silverlight, a Flash-like browser plug-in for multimedia content. Someones got to fuzz the hell out of this thing, and not only on IE, but on Firefox and Safari, both of which are supported.

As Joanna Rutkowska likes to say, computer systems these days are too complex to have real confidence in. The ANI failure was basically the result of sloppy testing, but even if Microsoft had tested well and found it, there are a large, unknown number of similar issues in a system as complex as Windows (or any other modern operating system). Until a better approach comes along, we need to have more testing and much more aggressive testing.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983. More from Larry Seltzer Check out eWEEK.coms Security Center for the latest security news, reviews and analysis. And for insights on security coverage around the Web, take a look at eWEEKs Security Watch blog.
Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel