Big Data Analysis Can Detect Cyber-Attacks Before It's Too Late

NEWS ANALYSIS: Harnessing the wealth of data produced by cyber-security systems can give security professionals the insights they need to detect cyber-attacks as they occur.

WASHINGTON, D.C. —"Times have changed," warned security consultant Mischel Kwon, former director of US-CERT. "We can't afford to wait for an antivirus warning or to have malware trip a firewall."

Kwon was kicking off a high-level seminar of government cyber-security experts gathered by FedInsider, a management publication for federal government executives. The panel Kwon was chairing included speakers from agencies ranging from the Department of Homeland Security (DHS) to the North Atlantic Treaty Organization (NATO), all of whom were addressing the role of big data in providing warnings of cyber attacks as they were about to happen.

But dealing with the data that describes the attack environment is a daunting task, pointed out J. R. Reagan of Deloitte & Touche. "We are amassing huge amounts of event data," Reagan said, noting that the volume of data "is beginning to outstrip the human ability to see patterns."

He noted that one of the most significant advances in cyber-security is the ability to produce visualizations of that data that make it possible to see patterns that wouldn't be visible otherwise.

"We can see pictures about sixty thousand times faster than we can read text," Reagan explained. "Now we can see the point of attack." He said that by producing the right visualization of the event data, security officers can see patterns in the events leading up to the attack that they never would have been able to see otherwise, and as a result can see the attack as it starts. He said that by the time a security event actually happens, it's too late.

Reagan, who is also on the faculty of Johns Hopkins University, in Baltimore, said that event data from sensors throughout an enterprise contribute to the data that can be used in the cyber-security analytics used to understand an attack when it's just starting.

But to be able to produce visualizations that are useful requires a huge amount of data. "It can be a billion events a day," Kwon said, explaining that this could mean as much as 24 terabytes of event data daily. But all of that data can be subjected to an analytical process that reveals patterns in nearly real time, which is important. "Patterns grow and change quickly," Kwon said.

Visualizing all of that data isn't easy, said Curtis Levinson, U.S. cyber-defense adviser to NATO. "What are true events and what is background noise?" he wondered. Levinson said that collecting event data is complicated by the fact that it has to be shared, and to be shared it must be cleaned of all personally identifiable information. Only then, he said, can researchers produce really useful visualizations.

Wayne Rash

Wayne Rash

Wayne Rash is a freelance writer and editor with a 35 year history covering technology. He’s a frequent speaker on business, technology issues and enterprise computing. He covers Washington and...