A top-level open-source project at the Apache Software Foundation (ASF), Apache NiFi is an easy-to-use system to process and distribute data. The acquisition will make it easy for users to automate and secure data flows and to collect, conduct and curate real-time business insights and actions derived from data in motion.
"The NiFi user interface and ease of extension have made it extremely easy to get up and running and even customize," said Craig Connell, CTO at Leverege, which develops software for users to manage and visualize large networks of diverse sensors, including Internet of things (IoT) environments. "It is great that NiFi also easily integrates with other parts of the Apache big data world like Spark, Kafka and Hadoop."
As a result of the acquisition, Hortonworks is introducing Hortonworks DataFlow powered by Apache NiFi, which is complementary to the Hortonworks Data Platform (HDP). The transaction is expected to close in the third quarter of 2015.
Explaining the impetus behind NiFi, Joe Witt, CTO at Onyara told eWEEK: "We saw that it was difficult to see and control how data flows from access points as small as a sensor back to the data center. And there was disjoint logging and no real bidirectional intelligence as data was in motion. So we created 'Niagara Files' to address this, which through the NSA Technology Transfer was donated to the Apache Software Foundation and became Apache NiFi. NiFi makes it easy to automate and secure IoAT [Internet of anything] data flows that collect, conduct and curate real-time business insights and actions derived from sensors, machines, geo-location devices, clicks, logs, social feeds, etc.—the IoAT."
In addition, Witt and his team recognized the need for more comprehensive security and governance when it comes to IoAT. As more and more data is generated from IoAT, including data from sensors, geo-location devices, server logs, clicks, machines, social feeds, as well as any other data source at the edge, securely ingesting and processing the data from the "jagged edge" is an issue, he said. Customers and developers have had no choice but to create custom, disjointed and loosely integrated solutions to solve the problem of analyzing data and providing insights. However, NiFi provides fine-grained provenance metadata that supports compliance and governance as well as secure end-to-end data routing, including encryption and compression.
"Hortonworks is focused on doing everything possible to enable our customers to transform their business through data-driven insights and actions," Rob Bearden, CEO of Hortonworks, said in a statement. "Onyara's impressive work on security and simplicity in NiFi, combined with their commitment to open-source makes for a perfect addition to our technology team."
Hortonworks officials said a new data paradigm that includes data from machines, sensors, geo-location devices, social feeds, click streams, server logs and more is fueling the Internet of anything and driving the need for trusted insights from data at the very edge to the data lake in real time with full fidelity. Many IoAT applications need two-way connections and security from the edge to the data center. This results in a "jagged edge" that increases the need for security but also data protection, governance and provenance. These applications also need access to both data in-motion and data at-rest.