How the U.S. PRISM and Blarney Programs Mine Your Data for Intelligence

NEWS ANALYSIS: Two programs go beyond phone calls to email and cloud storage to gather info about terrorism and provide it to the NSA and British intelligence.

WASHINGTON, DC—The revelations by The Washington Post about two big data analysis operations named PRISM and Blarney dropped like a bombshell on the Washington intelligence and security communities.

But I’d already heard about PRISM a day earlier and was trying to put it into context when the story broke. What’s surprising was that a few details emerged at a conference I was covering for eWEEK about cyber-security and big data.

There, people near me were discussing something called “PRISM” as the topic of how cyber-security experts look for patterns in event data. At the time, the discussion, while intriguing, wasn’t in context and I wasn’t having much luck in the few intervening hours learning more. Now I know why.

But if PRISM was such a huge secret, why was it being discussed openly in a public meeting room at the Willard Inter-Continental Hotel? Was it because it wasn’t as secret as the government says it was?

Leaving aside the wheels-within-wheels that characterize discussions in Washington, it’s clear that both PRISM and Blarney were important projects. PRISM, according to the story in The Washington Post, is responsible for a huge harvest of intelligence, and is reportedly responsible for disrupting at least one terrorist plot in the U.S.

Here’s what’s going on. Intelligence services in the U.S. have entered into agreements, backed up with court orders from the Foreign Intelligence Surveillance Court (FISC), with a variety of Internet companies to get access to their data. This court is so secret it never publishes its findings and only the U.S. government is authorized to appear before it.

While the companies deny they are cooperating, which is required by the court orders, the fact is that Microsoft, Yahoo, Google, Facebook, AOL, Skype, YouTube, Apple, PalTalk and probably many others are all accessible to National Security Agency (NSA) scrutiny. In cases where the companies know about the surveillance, they're required by the court orders not to reveal that information. But many of the companies may not be aware that their servers are being penetrated by the NSA through the use of equipment installed in their data centers to which the NSA can send commands.

On all of these services, email is sampled as are other message types. Cloud storage is searched. So if you have documents on Google Drive, SkyDrive, iCloud or other items on the other services, you can assume that they’ve been searched for keywords. The NSA doesn’t exactly read your documents or email, but rather mines those for keywords in a vast big data dragnet. Depending on the keywords and the origin or destination of the email, or the context of the document or video, the information may be recorded.

In addition to this keyword search, the NSA is also sampling email traffic for metadata. This is similar to the telephone number search that’s being conducted with the sweep of call records.