eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
OpsClarity, which provides Web-scale application monitoring solutions, has announced that its Intelligent Monitoring offering now provides monitoring for a growing suite of open-source data processing frameworks.
The technologies supported include modern data processing frameworks like Apache Kafka, Apache Storm and Apache Spark, as well as datastores such as Elasticsearch, Cassandra and MongoDB.
The OpsClarity solution enables DevOps teams to gain visibility into how these technologies are dependent on each other and troubleshoot performance issues.
“Open source data processing frameworks have rapidly matured and gained enterprise adoption to provide immediate business value, whether it be to identify customer preferences on the fly, detect online fraud or IoT-enable the next electronic device in our homes,” said Amit Sasturkar, co-founder and CTO of OpsClarity, in a statement. “OpsClarity has deep domain understanding of these distributed and complex data processing frameworks and how they work together and has built an intelligent assistant that visualizes the entire environment, detects and correlates failures, and provides guided troubleshooting.”
Technologies such as Apache Kafka, Apache Spark and Apache Storm are constantly expanding the scope of what is possible with big data. However, most of these data processing frameworks are themselves a complex collection of several distributed and dynamic components such as producers and consumers and masters and slaves, OpsClarity said. Monitoring and managing these data processing frameworks and how they are dependent on each other is a nontrivial undertaking and usually requires highly experienced operations teams to manually identify the individual metrics, chart and plot them, and then correlate events across them.
OpsClarity uses advanced data science with a real-time streaming analytics approach to ingest large amounts of metrics and events data from a disparate set of the open-source frameworks while intelligently correlating metrics and events across them. The OpsClarity Intelligent Monitoring solution provides auto-discovery, visual topology, health analysis and troubleshooting capabilities for the open-source frameworks.
“Unresponsive applications, system failures and operational issues adversely impact customer satisfaction, revenue and brand loyalty for virtually any enterprise today,” said Holger Mueller, vice president and principal analyst at Constellation Research, in a statement. “The distributed and complex characteristics of modern data-first applications can add to these issues and make it harder than ever to troubleshoot problems. It is good to see vendors addressing this critical area with approaches that include analytics, data science, and proactive automation of key processes to keep up with the changes being driven by DevOps and web-scale architectures.”
After ingesting metrics and events data from the open-source frameworks, OpsClarity synthesizes the various metrics, alerts and signals into an intuitive visual service topology along with overlaid health status. This simplifies the effort required by DevOps to set up and troubleshoot these modern data frameworks.
The auto-discover feature automatically discovers all the components of various data processing frameworks and automatically configures a deep and specific collection of metrics, events, alerts, process and network data. For example, Kafka brokers, Spark masters/slaves and Storm supervisors/workers are auto-discovered and auto-configured.
With the visual topology capability, users can automatically discover the service connections and dependencies to generate a logical visual topology for the open-source data processing frameworks. The health analysis capability enables users to see data processing framework component health, prioritized anomalies and service-level metrics—all within the context of the topology. And the troubleshooting feature provides a specific and actionable anomaly detection and event correlation capability that allows for rapid root cause analysis.
OpsClarity emerged from stealth in December 2015 and announced its Intelligent Operations Platform, a solution to help DevOps teams intelligently understand and troubleshoot issues in their rapidly changing Web-scale application environments.
“Our vision is to build intelligent, data-driven software for today’s DevOps teams,” said Dhruv Jain, co-founder and CEO of OpsClarity, in a statement regarding the company’s launch. “The rise of cloud-native services and web-scale application architectures has resulted in massive change, scale and complexity challenges for operations. Managing this requires a next-generation solution that can automatically adapt to, and understand, dynamic application environments and proactively surface actionable insights. We are leveraging our roots in data science and large-scale streaming analytics to build a smart solution that brings new levels of visibility, focus and productivity to modern software operations.”
The OpsClarity Intelligent Operations Platform automatically detects application architecture and system relationships, applies built-in domain knowledge about modern application stacks and relevant service metrics, improves signal-to-noise ratio through advanced anomaly detection, and delivers key insights and tools needed for rapid event correlation and root-cause analysis, the company said.
“We were seeing tangible productivity benefits from leveraging web-scale application components, but it was a long and laborious process for our operations team to find root causes for issues in this complex new environment,” said Raj Madhuram, vice president of engineering for C1X, in a statement. “With OpsClarity we have intuitive visualizations, proactive anomaly detection, and powerful event correlation, allowing us to troubleshoot and pinpoint issues across our applications and infrastructure more quickly than ever before.”