Microsoft Testing Real-Time Botnet Threat Intelligence Data Feed

NEW YORK — Microsoft is planning to make the data it collected as part of its botnet takedown operations available as a real-time threat intelligence feed.

The company would distribute threat data obtained from captured botnets and other sources to foreign governments, law enforcement, Computer Emergency Response Teams and private corporations, two members of the Microsoft Digital Crimes Unit told attendees at the International Conference on Cyber-Security at Fordham University Jan. 11.

Partners would be able to access the information using application program interfaces (APIs) that would be provided free by Microsoft.

When the real-time threat intelligence data service goes live, Microsoft plans to offer three different feeds. Partners could look for malware infections that often are part of botnet activity, or correlate host data with information on various Internet scams, such as click fraud, to connect the dots between them, according to T.J. Campana, a senior program manager in the Microsoft Digital Crimes Unit. Microsoft declined to say when the new real-time feed would go live.

Sharing threat intelligence data was an oft-recurring theme at ICCS. Better coordination and information-sharing between law enforcement, security researchers and private organizations is essential in the battle against cyber-criminals, Barry Green, the president of the Internet Systems Consortium, told attendees on the first day of the conference. While public-private partnerships are essential, there needs to be more data-sharing and private-private collaboration within the industry, according to Green. RSA Netwitness already offers a real-time data feed of network threat intelligence data collected from multiple sources.

"We collect a tremendous amount of data from our global assets," said Campana.

Microsoft has been collecting botnet data from a number of sources. The company's vast Internet infrastructure, which includes a global load-balanced, 80G bps network, can take over whole botnets and point the infected hosts to servers that Microsoft controls, according to Campana. This way, Microsoft is able to capture the botnet activity for its data collection purposes while effectively stopping the malicious traffic.

Data obtained during the recent operations that helped shut down the Kelihos, Rustock and Waldec botnets in the United States would also be included in the service. The takedowns, such as the operation for Rustock in March 2011 and the Waldec campaign in 2010, were a significant source of data, according to Richard Domingues Boscovich, a senior attorney in the Microsoft DCU who spoke during the same session as Campana. Microsoft has been collecting the data for three years, using a manual process and working with various partners such as CERT, according to Boscovich.

The data harvested from the Kelihos botnet back in September is already stored in the system, which includes IP addresses of infected systems, Autonomous System (AS) numbers as assigned by regional Internet registries (RIR) and reputation data provided by Microsoft's Smart Data Network Services, Campana said. Microsoft will not include the personally identifiable information that had been collected by the botnet in the threat intelligence feed, according to Campana.

Microsoft partners wanted a way to access the botnet data Microsoft had accumulated in a faster and more efficient manner, which provided the impetus for the data service. Once live, the service would segment the data, and only the relevant pieces of data will be shared with the partner. For example, national or regional CERTS would be more likely to prefer seeing threat intelligence related to their geographic area. The service will also give small organizations the security intelligence necessary to battle large global botnets without having to invest in complex analysis and forensics or engaging the professionals capable of using those tools.

Microsoft has been beta-testing the system internally, according to Campana. The system appears to be a 70-node cluster running Apache's Hadoop software framework on top of Windows Server, Campana said. The open-source Hadoop software allows organizations to work with petabytes of data and thousands of nodes.

The third annual International Conference on Cyber Security: A White Hat Summit is a joint effort between the Federal Bureau of Investigation and Fordham University. Leaders from law enforcement, industry and academia discuss cyber-crime and real-life operations during the conference, which runs from Jan. 9 to Jan. 12 on the Fordham University campus in New York.