Lets Get Fuzzing

Opinion: When systems get as complex as Windows, the only way to keep up with the security flaws is with aggressive fuzz testing.

Ive said it before and Ill say it again: You just cant do enough testing of software systems. And as complexity of software systems rises, more testing is necessary. Do you think, for example, that a Boeing 747 is a complicated piece of machinery? I suspect it pales in comparison to Windows Vista for complexity.

The recent ANI episode said a lot about the problem of complexity and testing. In fact, Microsoft does test aggressively and more than ever, and yet the ANI failure was more than anything else a failure of its testing.

There are many different kinds of testing you can do, but when it comes to the bizarre attacks of the ANI sort, nothing beats fuzzing. Microsoft knows this, and it has been an enthusiastic fuzzer for years.

Are you a security idiot, doing the wrong thing all the time? Click here to find out.

Resource Library:

Fuzzing is a form of automated brute-force testing. You pick a protocol or a file format or an API, or whatever, and the fuzzer exercises it through a defined set of values or parameters. Intelligent fuzzing tests are somewhat picky about what they fuzz because you dont want to waste time testing cases that cant possibly occur. But picking the right set of cases can be hard, and you probably want to err more on the side of completeness than economy.

Whatever its motivations, it seems that Microsofts fuzzing cases were incomplete in the case of ANI files. Beyond Security, author of the beStorm testing tool, was curious about this too and wrote a quickie fuzzing module for the ANI file format. Glance at the module, which describes the ANI file format as input for beStorm to fuzz it. In short order it found the zero-day attack that caused all the ruckus.

How could Microsofts fuzzers have missed it? Actually, Microsofts Michael Howard just volunteered the answer to that question on the new Security Development Lifecycle blog:

It turns out none of the .ANI fuzz templates had a second "anih" record. This is now addressed, and we are continually enhancing our fuzzing tools to make sure they add manipulations that duplicate arbitrary object elements better.

Oopsie! This is especially sloppy in light of the fact that a previous severe vulnerability in the same format also involved multiple anih records.

Making the decision that something is not worth fuzzing is getting increasingly dangerous. Read the April 27 Symantec Security Response blog for an idea of how bizarre the attacks are getting. The malware writers smell blood in the water in the form of sloppy parsing and sloppy testing. You cant be too careful.

And fuzzing is a two-way weapon. Dont think that the people who found and exploited the ANI zero-day found it by flipping bits in a file by hand. They have their own fuzzers too, as do the vulnerability research companies like eEye and Determina (who found the ANI bug). This is why, with as much fuzzing as it already does, Microsoft needs to do more and more.

Heres a good candidate: Microsoft recently announced Silverlight, a Flash-like browser plug-in for multimedia content. Someones got to fuzz the hell out of this thing, and not only on IE, but on Firefox and Safari, both of which are supported.

As Joanna Rutkowska likes to say, computer systems these days are too complex to have real confidence in. The ANI failure was basically the result of sloppy testing, but even if Microsoft had tested well and found it, there are a large, unknown number of similar issues in a system as complex as Windows (or any other modern operating system). Until a better approach comes along, we need to have more testing and much more aggressive testing.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

More from Larry Seltzer

Check out eWEEK.coms Security Center for the latest security news, reviews and analysis. And for insights on security coverage around the Web, take a look at eWEEKs Security Watch blog.

Reader Comments: Lets Get Fuzzing >>> Be the FIRST to comment on this article!

>>> More Network Security & Hardware Articles          >>> More By Larry Seltzer