Officials with Core, which is based in Boston, announced the flaw here at the ongoing RSA Conference just as Microsoft Chairman Bill Gates delivered his keynote address. The issue illustrates the fact that unless third-party application vendors go to great lengths to integrate their products with Vistas security features, the technologies cannot take advantage of the operating systems malware-defense tools, Core officials said.
Core contends that a previously disclosed vulnerability in CAs BrightStor ARCserve Backup software, dubbed CVE-2007-0169, can be exploited to compromise systems running the new Vista operating system.
By exploiting the buffer overflow vulnerability in versions 9.01 through 11.5 of the CA software, along with its Enterprise Backup 10.5 and CA Server/Business Protection Suite r2 products, attackers could remotely execute arbitrary code on computers and potentially gain access to other systems, the automated penetration testing company said.
To craft an attack that takes advantage of the flaw, hackers need only manipulate slightly exploits designed to attack the same problem on systems running Microsofts earlier Windows XP and 2000 operating systems, Core maintains.
CA already has a security patch available that will allow users of the software to block the loophole.
One of the most significant benefits being touted by Microsoft in Vista is the products many security features, which claim stronger protection of the softwares kernel, on-board malware-fighting tools and the programs User Account Control system—meant to keep viruses from escalating privileges on infected machines to prevent them from proliferating themselves onto other devices.
However, unless application vendors such as CA go to great lengths to integrate with those features, the tools can easily be defeated, as in the case of the BrightStor exploit, said Max Caceres, director of product management at Core. Part of the problem is that building products that can tap into the Vista security protections is not an easy task, according to the vulnerability expert.
"Application vendors need to be diligent about making sure that their products take advantage of Vistas security features—they dont integrate with them by default—and as long as developers do not make the necessary adjustments, their products will remain vulnerable to the same issue we saw in Windows XP," Caceres said. "The exploit we found demonstrates that even if companies are running Vista, they can easily be exposed to third-party flaws."
The CA vulnerability specifically circumvents Vistas ALSR (Address Space Layout Randomization) technology, which is meant to prevent buffer overflow exploits, a common mode of malware attack. The technique is also widely used in by developers to secure open-source software programs.
Most independent software vendors are porting their products to run on Vista but would need to completely rewrite sections of the programs to take advantage of the feature, Core maintains.
CA representatives challenged Cores report, calling the information "misleading" and pointed out that the company has specifically instructed customers not to run the products in question on Vista systems.
"CA goes to great lengths to certify and test its products in a variety of real-world configurations," company officials said in a statement. "Core made reference to a so-called security vulnerability that could occur if IT organizations use certain versions of a CA product—but this is precisely why CA has not specified that its customers use those software versions with Vista."
CA said that its first general release of BrightStor ARCserve Backup for Microsoft Vista (ARCserve Backup r11.5 SP3) will arrive in several weeks, and that it will include a patch for the vulnerability mentioned in Cores report.
Ben Fathi, corporate vice president of development of Microsofts Windows Core Operating System Division, said that Core is attempting to make a mountain out of a molehill with its report and defended the argument that making the ASLR feature a default setting for every application integrating with Vista would break some older third-party products—which would not benefit their users.
"This is an old version of a third-party product that is being updated from what were told. I dont see the big deal," Fathi said. "And I dont see why this company is forwarding the idea that its hard to integrate with Vista security. Thats certainly not what were hearing from our partners."
Editors Note: This story was updated to include comments from Microsofts Ben Fathi.