eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
In this era in which software, especially prominent software, must be presumed to be under attack, you need the best tools to defend yourself. Much has been made of security features built into Windows Vista, such as IE Protected Mode, which accrue to all users. However, programmers can easily gain new security defenses for their applications if they code Vista-specific sections in them.
Some of these facilities are famous and don’t actually involve any coding changes at all, at least not for their own sake. Take ASLR, or Address Space Layout Randomization. This feature loads program images into random locations, making it hard for shell code in many attacks to run reliably. Note that this defense presumes that the attack code has begun executing, but it prevents it from doing meaningful damage. All a programmer has to do is to link the program with a relatively modern version of the linker and use the /DYNAMICBASE linker switch.
What happens if attack code executes in an ASLR program? The program will almost certainly crash. This is a good thing, in the sense that it’s a lot better than the shell code in the attack program executing. This is a common theme among many of the new Vista-specific defenses.
DEP, or Data Execution Prevention-also known as NX support-is similar in effect. All reasonably modern processors, and certainly anything that can run Windows Vista, can be set to throw an exception when program code is run from an area marked as data. This is a fundamental technique for vulnerability exploits: Send data to a program that tricks the program into transferring program control to attack code sent by the attacker, and this code will inevitably be stored in a data area such as the heap. All programmers need to do to take advantage of NX is to link with the /NXCOMPAT switch. This is not a Vista-specific feature, by the way; it was added in Windows XP SP2, and even after all that time relatively few programs opt into it, which is both surprising and appalling. We should expect more from developers.
If you follow vulnerability reports, you have seen over the last few years many fewer buffer overflow exploits on the stack and many more on the heap. Many software-based systems are available to detect stack corruption. The heap is trickier to protect because the proper structure of the heap is specific to the management of it. Most applications use the operating system to manage memory on the heap, so the OS is the obvious place to protect the heap. Vista adds such a function, HeapSetInformation(). Microsoft’s Michael Howard blogs about HeapSetInformation here (and follows up on the subject here).
When you run this function, the Vista Heap Manager will monitor the heap for signs of corruption or even the potential to cause heap corruption, and terminate the program. All manner of bread-and-butter exploitation techniques are detected in this way, including buffer overruns, buffer underruns, double frees and bogus heap handles.
As I said above, most applications use the OS for heap management, but not all do. Many write their own heap management. I was once on a programming team that did this (this was 1983 and it wasn’t my decision). We did it for a few reasons: Our program was portable across multiple operating systems, and we wanted to be able to have consistent heap management across them. We also had the bias, which some still have, that our heap management was better than the operating system’s, which we believed to be slow and unreliable. This is an old battle between application programmers and the OS vendors who serve them.
In any event, HeapSetInformation() is of little or no use to programs that perform their own heap management. And custom heap managers do have security vulnerabilities now and then, such as the recent vulnerability in OpenOffice, fixed last week in OpenOffice 2.4.1. OpenOffice.org probably had the same rationales for building a custom heap manager that my team had 25 years ago.
Another common theme of these defenses is that they are likely to bring to the surface subtle bugs in your code. It’s entirely possible that your program has, for example, buffer overflows that are, in the normal course of things, innocuous. But start checking your heap for corruption, and it suddenly looks potentially sinister. Or if your program has self-modifying code and you turn on NX support, the processor will throw an exception when your program runs because you are running code in a data area.
There are ways for programs with self-modifying code (such as a virtual machine that performs JIT operations) to operate properly even within a program that is otherwise marked for DEP/NX. You need to use proper options on VirtualAlloc(), VirtualProtect() and VirtualFree(). For more on this and many other techniques discussed here, see “Windows Vista ISV Security” by Michael Howard and Matt Thomlinson.
It’s often presumed that any program crash is a potential opening for exploitation, but this is not at all true. Many crashes are not exploitable, and some, as shown here, are actually defense in action against exploitation. So next time a program crashes on you, don’t necessarily blame the programmer. It may have been for your own good.
Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.
For insights on security coverage around the Web, take a look at eWEEK.com Security Center Editor Larry Seltzer’s blog Cheap Hack.