Coverity, the San Franciso-based company managing the project under a $1.25 million grant, described the flaw as the "biggest security vulnerability" found in the X Window System code since 2000.
The X Window System, also called X11 or X, provides the toolkit and protocol to build GUIs for Unix and Unix-like operating systems. It is used to provide windowing for bit-map displays.
The X Window System also ships as an optional GUI with Macintosh computers from Apple.
Coverity Chief Technical Officer Ben Chelf said the flaw resulted from a missing parenthesis on a small piece of the program that checked the ID of the user.
It could be exploited to allow local users to execute code with root privileges, giving them the ability to overwrite system files or initiate denial-of-service attacks.
Coverity hailed the discovery as proof that its automated code scanning tool can discover serious flaws that the human eye might miss.
"This was caused by something as seemingly harmless as a missing closing parenthesis," Chelf said, describing the severity of the bug as a "worst-case scenario" for the X.Org Foundation that manages the X Windows System project.
Daniel Stone, release manager at X.Org, agreed that the vulnerability was "one of the most significant" discovered in recent memory.
"[This is] something that we find once every three to six years and is very close to Xs worst-case scenarios in terms of security," Stone said. "[Coveritys tool exposed] vulnerabilities in our code that likely wouldnt have been spotted with human eyes. Its attention to subtle detail throughout the entire code base—even parts you wouldnt normally examine manually—makes it a very valuable tool in checking your code base," he added.
The flaw, which affects X11R6.9.0 and X11R7.0.0, was fixed within a week of its discovery, and Chelf said Coverity has implemented a system to analyze the X Window System on a continuous basis to help prevent new defects from entering the project.
The flaw was pinpointed during automated code scanning that formed part of the "Vulnerability Discovery and Remediation Open Source Hardening Project," a broad federal initiative to perform daily security audits of approximately 40 open-source software packages, including Linux, Apache, MySQL and Sendmail.
Coverity is providing source code analysis technology, and software engineers at Stanford University are managing the project and maintaining a publicly available database of bugs and defects.
Anti-virus vendor Symantec is providing guidance as to where security gaps might be in certain open-source projects.
The purpose of the audit is to pinpoint buffer overflows, memory allocation bugs and other vulnerabilities that are a constant target for malicious hacking attacks. In addition to Linux, Apache, MySQL and Sendmail, the project will also pore over the code bases for FreeBSD, Mozilla, PostgreSQL and the GTK (GIMP Tool Kit) library.