eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
WASHINGTON, DC—The revelations by The Washington Post about two big data analysis operations named PRISM and Blarney dropped like a bombshell on the Washington intelligence and security communities.
But I’d already heard about PRISM a day earlier and was trying to put it into context when the story broke. What’s surprising was that a few details emerged at a conference I was covering for eWEEK about cyber-security and big data.
There, people near me were discussing something called “PRISM” as the topic of how cyber-security experts look for patterns in event data. At the time, the discussion, while intriguing, wasn’t in context and I wasn’t having much luck in the few intervening hours learning more. Now I know why.
But if PRISM was such a huge secret, why was it being discussed openly in a public meeting room at the Willard Inter-Continental Hotel? Was it because it wasn’t as secret as the government says it was?
Leaving aside the wheels-within-wheels that characterize discussions in Washington, it’s clear that both PRISM and Blarney were important projects. PRISM, according to the story in The Washington Post, is responsible for a huge harvest of intelligence, and is reportedly responsible for disrupting at least one terrorist plot in the U.S.
Here’s what’s going on. Intelligence services in the U.S. have entered into agreements, backed up with court orders from the Foreign Intelligence Surveillance Court (FISC), with a variety of Internet companies to get access to their data. This court is so secret it never publishes its findings and only the U.S. government is authorized to appear before it.
While the companies deny they are cooperating, which is required by the court orders, the fact is that Microsoft, Yahoo, Google, Facebook, AOL, Skype, YouTube, Apple, PalTalk and probably many others are all accessible to National Security Agency (NSA) scrutiny. In cases where the companies know about the surveillance, they’re required by the court orders not to reveal that information. But many of the companies may not be aware that their servers are being penetrated by the NSA through the use of equipment installed in their data centers to which the NSA can send commands.
On all of these services, email is sampled as are other message types. Cloud storage is searched. So if you have documents on Google Drive, SkyDrive, iCloud or other items on the other services, you can assume that they’ve been searched for keywords. The NSA doesn’t exactly read your documents or email, but rather mines those for keywords in a vast big data dragnet. Depending on the keywords and the origin or destination of the email, or the context of the document or video, the information may be recorded.
In addition to this keyword search, the NSA is also sampling email traffic for metadata. This is similar to the telephone number search that’s being conducted with the sweep of call records.
How the U.S. PRISM and Blarney Progams Mine Your Data for Intelligence
As in the case of the call records, the agency isn’t recording the content, but rather using the metadata to look for patterns. It’s the patterns in the data that raise the flag that a terrorist action is being discussed.
How, you wonder, is this even possible? In one sense, it’s not. Despite its significant capabilities, even the NSA can’t read all the email that travels through the Internet every day. Besides, trying to monitor such a huge percentage of spam isn’t likely to yield much beyond a clogged network gateway. But what the NSA does is take samples and flag those keywords. When the agency starts to detect specific combinations of keywords, paired with metadata from Blarney, then the specific sender or recipient is flagged for further analysis.
Blarney works in concert with PRISM by tracking email and other traffic as it passes through what the NSA calls “Internet choke points,” which probably refers to major ISPs and major routing centers, especially those in the San Francisco and Washington regions. Blarney then mines this traffic for metadata from email and other communications such as file transfers and multimedia files.
Depending on the nature of the information, the NSA may share the email details with another agency such as the Federal Bureau of Investigation (FBI) or the Central Intelligence Agency (CIA). The job is so vast that the NSA is sharing this job with British intelligence, which is doing its own searching and analysis.
You may also wonder how this is legal. Again, this is the subject of a court order by the Foreign Intelligence Surveillance Court (FISC), which acts on secret warrant requests from intelligence agencies. This court is sufficiently secret that initially its very existence was a secret. However that was revealed a few years ago. This court is also known for never having turned down a surveillance request by a U.S. intelligence agency.
Now that the existence of these programs is known, there’s been some discussion in Washington that the director of national intelligence may shut them down. That’s a fantasy. These programs are so successful at yielding actionable information that they are a primary source for critical intelligence. In addition, because a large majority of all global Internet traffic passes through the U.S. at some point in its journey, there’s little that terrorists or anyone else can do to prevent it.
What might happen, at least before someone files a Fourth Amendment lawsuit, is that terrorists overseas may stop communicating using email. This alone would curb their operations and while that’s not a bad thing, there are other worries.
Those other worries include whether the existence of this capability and its companion court orders may give other agencies, such as the Department of Justice, a way to circumvent the requirement for search warrants in its witch hunt for leaks to the news media. That would be a very bad thing indeed.