Integer Overflows Add Up to Real Security Problems

By Larry Seltzer  |  Posted 2004-03-11 Print this article Print

How can a security hole occur in software just by adding two numbers together? "Integer overflows" are the hot topic in vulnerability nowadays and Security Center Editor Larry Seltzer describes in detail how it happens.

Most security vulnerabilities are software bugs—in the strict sense of the word. And most of these bugs would be considered innocuous, perhaps in an environment where people arent trying to break the program. But then, we come to the Internet. The most famous class of such bugs is the buffer overflow, by now the kind of term that makes it into your local paper when another Windows flaw makes the news. But in recent years a new type of vulnerability is being exploited more frequently: Integer manipulation bugs. Not all integer manipulation bugs are integer overflows—some of them are underflows. Still, the class of error is usually referred to generically as overflows.
The basic problem is that integers in computers have a finite range. For instance, the rage of a signed 16-bit integer is -32767 to 32767.
What happens if arithmetic moves the value out of that range? The number could easily turn out to be massively larger or smaller than the expectation of the programs logic. Another example is a number that turns out to be negative instead of positive, changing the result of an "if (a<b)" comparison from what it was originally designed to be. And then there are errors relating to the effects of integer promotion. When operations are made on integers of different sizes, say a short and a long, the smaller one is promoted temporarily to the larger size, and the result is potentially truncated back to the smaller size. So what can go wrong just because a number is not what it should be? Some of those numbers are used for important stuff. Consider this example (lifted without permission from Michael Howards excellent article on MSDN: Reviewing Code For Integer Manipulation Vulnerabilities):
	bool func(size_t cbSize) {
	   if (cbSize < 1024) {
	      // we never deal with a string trailing null
	      char *buf = new char[cbSize-1];

	      // do stuff

	      delete [] buf;

	      return true;
	   } else {
	      return false;
For the non-C coders out there, this function tests to see whether the variable cbSize is less than 1024, allocates it and then zeros out a buffer of cbSize-1 bytes, and then deletes it. The program carefully checks to make sure the number isnt too big, but what if its too small? We can see that size_t is unsigned, so it cant become negative, but it can get to be zero. When you subtract 1 from it, the value 0 wraps around to the 32-bit value 0xFFFFFFFF. Oops—you just allocated a 4GB block of memory, and unhappily, HAL wont be opening the pod bay door. Next Page: How Integer Errors are Exploited

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel