Red Hat Digital Keys Violated by Intruder

In perhaps the most appalling breach of security at a major operating system vendor, Red Hat has revealed that a compromise of its internal systems included the digital signing keys for its distributions. An Aug. 22 advisory from Red Hat announces new OpenSSH packages to deal with the problem:

In connection with the incident, the intruder was able to sign a small number of OpenSSH packages relating only to Red Hat Enterprise Linux 4 (i386 and x86_64 architectures only) and Red Hat Enterprise Linux 5 (x86_64 architecture only). As a precautionary measure, we are releasing an updated version of these packages, and have published a list of the tampered packages and how to detect them at http://www.redhat.com/security/data/openssh-blacklist.html. 

In other words, the attacker was able to sign files with Red Hat's keys. Presumably these were not benign versions he signed. Red Hat stresses that there is no evidence that any such hacked copies got out through its normal distribution channels to its own customers, but it's possible that some mirrors picked up the code.

It all started with an obscure message announcing an "issue" in the "infrastructure systems." Clearly this was serious since the message added, "…we recommend you not download or update any additional packages on your Fedora systems." This was followed by a vague but happier progress report. Note that the initial reports are about Fedora, not Red Hat Enterprise Linux.

Today Red Hat confessed at least some of the seriousness of the matter. An "infrastructure report," e-mailed out to the fedora-announce-list, announced that Fedora servers were illegally accessed. One of them was a system used for digitally signing Fedora packages. Even though Red Hat execs are confident that the actual keys for Fedora weren't compromised, they have decided, out of "an abundance of caution," to convert to new keys. This is not a minor step: "This may require affirmative steps from every Fedora system owner or administrator. We will widely and clearly communicate any such steps to help users when available."

Fedora is hosted at Red Hat, but the two distributions are separate. Even so, Red Hat systems were also compromised, as described in the OpenSSH update advisory. The compromised Red Hat keys were used for Version 4 and Version 5, the current version. There is no announcement, at least for now, of new keys for RHEL. Perhaps this is unnecessary because the distribution mechanisms are different. Oh, and by the way, while Red Hat was issuing the new OpenSSH, it fixed a minor security flaw having to do with X.11 cookies.

In addition to getting new keys and code out there, Red Hat will have to revoke the old keys. I'm not certain enough of its tools—which I assume are based on OpenSSL and X.509 certificates—to know if it has an effective revocation mechanism. We'll learn more about this over time.

The SecuriTeam blog report on this makes an interesting observation that Netcraft's hosting history for fedoraproject.org briefly switched last week from running Red Hat Enterprise Linux to an older version of Fedora, and then back:

209.132.176.122 - Linux Apache/2.2.3 Red Hat - 19-Aug-2008

209.132.176.122 - Linux Apache/2.2.0 Fedora - 16-Aug-2008

209.132.176.122 - Linux Apache/2.2.3 Red Hat - 19-Aug-2008

What could this mean? Just another abundance of caution, perhaps, but the same switch is not present in the history for redhat.com.

Personally, I'm just astonished at this, even though there have been internal compromises of distributions of operating systems and major applications before. Trojaned versions of OpenSSH have been distributed in the past. And in 2007 the distribution server for WordPress was compromised and malicious code inserted.

Imagine the horrifying fallout if such a thing happened at Microsoft. In fact, it sort of did happen once, back in 2001. One of the HTTP servers running Windows Update and serving download bits was hit by the CodeRed worm and taken down quickly. It's a stretch to argue that any users were affected. A CodeRed compromise primarily involved defacement of the home page (which I guess is why it was noticed quickly), an attempt to spread itself and, much later, launching a DOS against certain fixed IP addresses (including the White House). I'd argue that this Red Hat incident is a far worse and more dangerous scandal. For instance, it's not clear to me that Red Hat can be sure how long the keys were compromised. Months? Who knows?

The last few years have seen a lot of bloom coming off the open-source security rose. I suspect there won't really be heavy fallout for Red Hat because, as serious as this is, it's just not all that shocking. Our standards used to be a lot higher.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

For insights on security coverage around the Web, take a look at eWEEK.com Security Center Editor Larry Seltzer's blog Cheap Hack.