Vistas Fortified Kernel Could Trouble Third-Party Apps

By Matt Hines  |  Posted 2006-08-10

Vistas Fortified Kernel Could Trouble Third-Party Apps

Researchers at Symantec are questioning whether security modifications added to the kernel of Microsofts Vista operating system could prevent the anti-virus company, and other third-party software makers, from enjoying the same level of integration theyve enjoyed with previous Windows operating systems.

As part of a research effort examining the next-generation operating systems kernel, the softwares very core, Symantecs analysts have been led to believe that Microsofts work to better protect the product may impede innovation by other security applications vendors.

At least one other company, consumer firewall software maker Agnitum, has also complained publicly that Vista wont allow the same level of kernel-access as earlier iterations of Windows.

If the assertion, which is based on assessments of beta versions of Vista, proves true in the final product, Cupertino, Calif.-based Symantec and other aftermarket Windows software makers could be challenged to advance their products as quickly as they have in years passed, researchers said.

"The challenge we have is that these technologies eliminate the potential for third parties to extend enhancements to the kernel," said Oliver Friedrichs, director of emerging technologies for Symantecs Security Response team.

"Weve traditionally used to this method to add security technologies into the kernel; with some of these new technologies, any tampering or modification to kernel will result in a blue screen, which means we cant use it."

Friedrichs and his team specifically identified one kernel modification used in the 64-bit version of Vista that could prove troublesome in such a manner.

The operating systems PatchGuard technology, which promises to prevent non-Microsoft programs from patching the Vista kernel, could make it impossible for Symantecs security applications to intercept system commands and protect users against certain types of malicious content, the researcher said.

"By hooking systems calls, we can see data passing through to the kernel and help protect against anything malicious," said Friedrichs.

Click here to read more about Vista kernel security issues.

"We have the alternative mechanisms that Microsoft has added to support this, but it limits the innovation we can make via kernel extensions in the future; there may also be new security technologies that evolve that need to access the kernel to do their job."

While Symantec roundly praises an overwhelming majority of the work Microsoft has done to improve the security of Vista, both in the kernel and throughout the product, the limited ability to integrate directly with the OS on its most fundamental level will cost third party Windows applications vendors in the long run, he said.

Friedrichs and other development experts at Symantec, the Windows anti-virus market leader and a longtime Microsoft partner, have publicly dissected beta versions of Vista, issuing a series of three reports identifying potential vulnerabilities in the software.

At the same time, the Symantec researchers have lauded Microsofts efforts, including its work to reduce vulnerabilities in the much-awaited operating systems underlying code base.

In their latest Vista report, Symantec researchers examined a series of technological modifications made to the operating systems kernel in the name of boosting security of desktop systems that will run the OS.

The review repeatedly praises the job Microsoft has done in creating a more secure system, including the addition of stricter requirements for types of software drivers that can be downloaded directly onto Vista PCs and the ability to monitor traffic with the kernel for suspicious activity.

The report also praises the Vista kernels onboard code integrity checks, support for a secure boot up mode, and the ability to restrict access to a Vista desktops physical memory.

The critique identifies only one potential kernel issue that the researchers contend could be circumvented to form an attack, related to the driver-signing and code scanning features.

Next Page: Competition.


While longtime partners, Microsoft and Symantec are also increasingly competing against each other in the security market, where Microsoft has already moved into the consumer anti-virus space and has plans to offer more products aimed at enterprises in the near future.

Many industry watchers have observed prior to the identification of the kernel issue that the two companies may have problems working together as they begin to compete for more of the same revenues.

Redmond, Wash.-based Microsoft has responded to Symantecs previous reports by pointing out that the beta releases dont represent Vista as a finished product, and more recently company officials began questioning why one of the software giants largest partners is giving its test versions such a high-profile undressing.

In response to the kernel access issues posed by Symantec over the use of PatchGuard, Stephen Toulouse, security program manager for Microsofts Security Technology Unit, said that the company may be overreacting. Microsofts goal in further locking down the kernel was to improve end user security, not to hinder the efforts of its partners in building security applications, or any other products, he said.

Microsoft itself will not be allowed to create programs that use the type of kernel systems calls that Symantec is worried about losing, and the methods that the software giant has given its partners to replace the direct kernel access will prevent developers from being forced to completely retool such products, Toulouse said.

The executive pointed out that PatchGuard is already used in several Microsoft products, including its 64-bit Windows XP and Windows Server 2003 SP1 operating system releases.

"When you allow the use of unsupported calls into the kernel it introduces the possibility for security and reliability problems, and weve already seen rootkits take advantage of this capability," said Toulouse.

"To the extent that people are talking about restrictions in place hampering their efforts, the trade off is that hackers cant [access the kernel] either; we wanted to level the playing field, so that malware writers no longer enjoyed the same advantage as third-party software makers."

Toulouse said that any third-party software modifications demanded by the use of PatchGuard should be "easily done," and he said that Microsoft has been working with vendors who might be affected by the modification.

Along with security technologies, certain types of video game protection software have used the kernel systems calls in the past. He said the kernel was never designed to be used in such a manner to begin with.

"Its never meant to be used this way; our prods dont do it, and it was never meant to be a function of the kernel as it introduces reliability and functional issues independent of security, along with those serious concerns," Toulouse said.

"Were working with everyone to try and provide that level of functionality they thought they could only get by using the undocumented commands; either everyone has equal access or no one has access."

As Microsoft wont use the kernel commands in its own security products, Toulouse said there should be little question over whether the company has adopted the PatchGaurd technology in Vista to promote its own competitive interests.

Industry watchers said it remains to be seen how the kernel protection measures may impact the development of third party security technologies in the long run, but analysts largely dismissed the idea that Microsoft is using the kernel defense tactic as a way to improve its own standing against Symantec or other security software providers.

"It certainly seems that Microsoft is trying to play nice with the security industry, and its not in their interest to make enemies, these companies need to work together to secure PCs and the Internet," said Natalie Lambert, analyst with Forrester Research, Cambridge, Mass.

"There will always be a lot of customers protected by Symantec who wont necessarily trust Microsoft or its security products to do the same job; Microsoft knows this, and its hard to believe that they would try something so aggressive."

However, other experts said the kernel issue could pose serious challenges to developers of HIPS (host intrusion protection systems) and other aftermarket security tools.

Andrew Jaquith, analyst with Boston-based Yankee Group, said that Microsoft could also conceivably use the design shift to its advantage as it enters the market for such technologies.

"PatchGuard will definitely make it harder for HIPS vendors to function in Vista; the third parties have two choices, they can continue to petition Microsoft to create an approved kernel hooking interface, or they could use black hat techniques to bypass it," said Jaquith.

"The anti-kernel hacking features could create a somewhat formal barrier to rivals in the security space if Microsoft uses the barrier to promote their own HIPS products, some which we will likely see in it ForeFront release; if thats the case, we may see some of the larger security companies run to antitrust regulators as fast as possible."

Check out eWEEK.coms for the latest security news, reviews and analysis. And for insights on security coverage around the Web, take a look at Security Center Editor Larry Seltzers Weblog.

Rocket Fuel