Red Hat Digital Keys Violated by Intruder

By Larry Seltzer  |  Posted 2008-08-22 Print this article Print

Just about the most serious breach of security possible at an OS vendor happened to this company. Red Hat is releasing updated OpenSSH packages to address the compromise of its internal systems.

In perhaps the most appalling breach of security at a major operating system vendor, Red Hat has revealed that a compromise of its internal systems included the digital signing keys for its distributions. An Aug. 22 advisory from Red Hat announces new OpenSSH packages to deal with the problem:

In connection with the incident, the intruder was able to sign a small number of OpenSSH packages relating only to Red Hat Enterprise Linux 4 (i386 and x86_64 architectures only) and Red Hat Enterprise Linux 5 (x86_64 architecture only). As a precautionary measure, we are releasing an updated version of these packages, and have published a list of the tampered packages and how to detect them at 

In other words, the attacker was able to sign files with Red Hat's keys. Presumably these were not benign versions he signed. Red Hat stresses that there is no evidence that any such hacked copies got out through its normal distribution channels to its own customers, but it's possible that some mirrors picked up the code.

It all started with an obscure message announcing an "issue" in the "infrastructure systems." Clearly this was serious since the message added, "...we recommend you not download or update any additional packages on your Fedora systems." This was followed by a vague but happier progress report. Note that the initial reports are about Fedora, not Red Hat Enterprise Linux.

Today Red Hat confessed at least some of the seriousness of the matter. An "infrastructure report," e-mailed out to the fedora-announce-list, announced that Fedora servers were illegally accessed. One of them was a system used for digitally signing Fedora packages. Even though Red Hat execs are confident that the actual keys for Fedora weren't compromised, they have decided, out of "an abundance of caution," to convert to new keys. This is not a minor step: "This may require affirmative steps from every Fedora system owner or administrator. We will widely and clearly communicate any such steps to help users when available."

Fedora is hosted at Red Hat, but the two distributions are separate. Even so, Red Hat systems were also compromised, as described in the OpenSSH update advisory. The compromised Red Hat keys were used for Version 4 and Version 5, the current version. There is no announcement, at least for now, of new keys for RHEL. Perhaps this is unnecessary because the distribution mechanisms are different. Oh, and by the way, while Red Hat was issuing the new OpenSSH, it fixed a minor security flaw having to do with X.11 cookies.

In addition to getting new keys and code out there, Red Hat will have to revoke the old keys. I'm not certain enough of its tools-which I assume are based on OpenSSL and X.509 certificates-to know if it has an effective revocation mechanism. We'll learn more about this over time.

The SecuriTeam blog report on this makes an interesting observation that Netcraft's hosting history for briefly switched last week from running Red Hat Enterprise Linux to an older version of Fedora, and then back: - Linux Apache/2.2.3 Red Hat - 19-Aug-2008 - Linux Apache/2.2.0 Fedora - 16-Aug-2008 - Linux Apache/2.2.3 Red Hat - 19-Aug-2008

What could this mean? Just another abundance of caution, perhaps, but the same switch is not present in the history for

Personally, I'm just astonished at this, even though there have been internal compromises of distributions of operating systems and major applications before. Trojaned versions of OpenSSH have been distributed in the past. And in 2007 the distribution server for WordPress was compromised and malicious code inserted.

Imagine the horrifying fallout if such a thing happened at Microsoft. In fact, it sort of did happen once, back in 2001. One of the HTTP servers running Windows Update and serving download bits was hit by the CodeRed worm and taken down quickly. It's a stretch to argue that any users were affected. A CodeRed compromise primarily involved defacement of the home page (which I guess is why it was noticed quickly), an attempt to spread itself and, much later, launching a DOS against certain fixed IP addresses (including the White House). I'd argue that this Red Hat incident is a far worse and more dangerous scandal. For instance, it's not clear to me that Red Hat can be sure how long the keys were compromised. Months? Who knows?

The last few years have seen a lot of bloom coming off the open-source security rose. I suspect there won't really be heavy fallout for Red Hat because, as serious as this is, it's just not all that shocking. Our standards used to be a lot higher.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

For insights on security coverage around the Web, take a look at Security Center Editor Larry Seltzer's blog Cheap Hack.

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel