OPNids Integrates Machine Learning Into Open-Source Suricata IDS

EXCLUSIVE: A new open-source project integrates the Suricata intrusion detection system with the DragonFly Machine Learning Engine, which uses a streaming data analytics model to help make decisions.


A new open-source intrusion detection system (IDS) effort is officially getting underway on Nov. 5 with the launch of the OPNids project.

The OPNids effort is being led by threat hunting firm CounterFlow AI and security appliance provider Deciso, which also leads the Opensense security platform project. OPNids is built on top of the open-source Suricata IDS, providing a new layer of machine learning-based intelligence to help improve incident response and threat hunting activities.

"We created a pipeline that will actually take the Suricata logs and analyze the packets to provide context around any alerts," Randy Caldejon, CEO and co-founder of CounterFlow AI, told eWEEK. "We like to call this alert triage. It's like taking it to the last mile of what the analysts would do anyhow because typically when there's an alert, they want some context."

The Suricata project got started in 2009 by the Open Information Security Foundation as an alternative open-source option to the Snort IDS that was already in market.

"What we're doing with the machine learning is taking advantage of the data that is flowing at line rates from Suricata and rather than just storing the data to disk we figured we should analyze the data while it's in motion," he said. "What we have is not one single model; we built an agent so analysts can write their own scripts."

How It Works

At the core of OPNids is the DragonFly Machine Learning Engine (MLE), which uses a streaming data analytics model to ingest line rate network data from Suricata.

"Most machine learning techniques are what are traditionally known as batch techniques, where you get a big pool of data offline and you apply a sophisticated algorithm onto that," Andrew Fast, chief data scientist at CounterFlow AI, told eWEEK. "We are taking a different approach, with streaming analytics, which is a newer branch of machine learning that is not as widely used."

Fast explained that with streaming analytics rather than looking at a collected pool of data, the DragonFly MLE is collecting statistics and making decisions as the data flows through the IDS. The DragonFly MLE can also support offline training and then online scoring at wire speed, he said. With machine learning, training is used to help tune a model, while scoring is the output of the model.

Caldejon added that OPNids can also enable post-processing of data for additional analysis, with an approach known as Filebeat, which is commonly used in the Elastic Stack. As such, OPNids data can be forwarded to an Elastic search engine for additional analysis. Apache Kafka streaming is also supported, as is syslog for general log capabilities.

Moving forward, Caldejon said the plan is to also integrate OPNids with threat intelligence gateway capabilities, taking in third-party threat data as a factor that helps the MLE make decisions.

OPNids Pro

Alongside the open-source effort, there is a commercially supported version of OPNids in the works. The commercial version has a hardware edition where software is all preloaded and integrated. Additionally, Caldejon said the Pro edition of OPNids will include a packet caching capability. The packet caching capability will take network packets and write them to disk, saving the PCAP (packet capture) for additional offline analysis, he said.

Both the Pro and standard editions of OPNids are fully bundled application images including the MLE and Suricata. Caldejon said users can download and get the system running within 10 minutes.

"It's got Suricata, it's got the MLE, and it has a nice UI [user interface], so even unsophisticated users can manage the technology," he said. "Typically, a lot of open-source projects are all CLI [command line interface] and expect people to be experts at the command line, but we wanted to make this accessible to data scientists as well, and they usually don't have those skills, so yeah, OPNids is a one-stop shop, everything is bundled."

Sean Michael Kerner is a senior editor at eWEEK and InternetNews.com. Follow him on Twitter @TechJournalist.

Sean Michael Kerner

Sean Michael Kerner

Sean Michael Kerner is an Internet consultant, strategist, and contributor to several leading IT business web sites.