eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
To manage the complexities of today’s cloud native infrastructure, there’s an increasing need for IT monitoring technology, often referred to as observability. As core building blocks of this trend, two sophisticated technologies are rapidly developing: Monitoring-as-Code and Infrastructure-as-Code.
Let’s dig into these emerging technologies and look at how they help monitor and manage complex cloud native systems.
Also see: Why Cloud Means Cloud Native
Infrastructure-as-Code: a Software-Defined IT Foundation
In the Infrastructure-as-Code model, there is still physical infrastructure, but it comes in the form of a descriptive model for software-defined compute, storage and networking resources that have been provisioned and subsequently delivered by cloud providers.
In IaC, there are no wires, plugs, hubs and routers; instead, there are ‘connection topologies’ defined by source code files. So of course there are cables and wires, but they live in the cloud data center and code defines which ones we use, on what day, for what workload and at what level of impedance and so on.
As we now traverse into the cloud-native environment – driven by Infrastructure-as-Code – with all the containerized Kubernetes granularity that comes with it, we still need to keep our eyes on the central challenge and look out for unexpected turns and speed bumps. Which brings us to Monitoring-as-Code.
Monitoring-as-Code: Moving From 1.0 to 2.0
The “always-on dashcam” that is Monitoring-as-Code (MaC) plays a key role in managing complex systems. A close cousin to (but not wholly synonymous with) Observability-as-Code (OaC), MaC is now rapidly evolving from a ‘simple’ data collection function to form part of the higher-tier observability strata that typify modern multi-cloud deployments.
In a world where DevOps, DevSecOps, and AIOps are all helping to create the comparatively new roles in Site Reliability Engineering (SRE), a new and more evolved use of monitoring will surely be needed as we move from static IT infrastructures to the dynamic models of IaC.
At its most basic level, Monitoring-as-Code (MaC) is an exercise in data collection via software agents, plugins and discrete elements of source code that work as ‘exporters’ to track system-level data and direct it to a repository. This is MaC level 1.0, if you will.
To get to MaC 2.0 and beyond, we need to think about building a wider observability lifecycle and enabling it with intelligence, diagnosis and extended abilities to achieve levels of automated remediation with what we might call codified reactive logic. In simple terms, the foundation blocks of IT and cloud software are shifting.
“One of the major tech trends for 2022 is going to be the integration of ‘observability and pipelines’ as adjacent markets,” said James Governor, co-founder of technology analyst house RedMonk. “Testing is shifting left, but it’s also shifting right, into production. Software delivery pipelines are now a mission critical application. We’re seeing observability players work more closely with companies providing CI/CD and software delivery tooling – to make life easier for customers.”
Governor says that so adjacent are the two work disciplines (or markets) of observability and software delivery that they are converging to enable what we can call “progressive delivery.”
Also see: AIOps Trends
Use Cases for ‘As Code’ Observability Tech
“As we move to progressive delivery, we can start focusing on use cases such as blue/green deployments, canarying, feature flags and dark launches, with progressive rollouts to specific user cohorts to limit the blast radius. Production then becomes part of the software delivery lifecycle,” explained Governor.
He says that the ‘as code’ revolution is driving all of this forward, enabling more sophisticated pipelines and extensions as organizations seek to modernize their app delivery, with better testing, more automation and observability baked in.
Current players in the Monitoring-as-Code space include Sumo Logic with its open source Sensu project. Described as an ‘observability pipeline’ built to deliver Monitoring-as-Code on any cloud, Sensu promises to consolidate monitoring tools and fill gaps in observability across data silos with automated diagnosis and self-healing. In sum, it supports the full transept of modern IT from bare metal to Kubernetes.
Sumo Logic acquired the Sensu project back in June of 2021 and CEO Caleb Hailey and co-founder Sean Porter joined the new parent company in roles designed to further their work. The pair conceived Sensu while working on an internal project to replace an out-of-date Nagios infrastructure monitoring installation for a cloud information archiving company.
Without necessarily deriding their worth, Porter himself called out MaC 1.0 tools for what they were: software agents designed to transit system file information to configuration management tools that might include Puppet, Chef, Helm, Terraform, Ansible or other. He described this as an action akin to retrofitting traditional monitoring tools to the modern DevOps paradigm.
This all brings us full circle into another modern application platform implementation paradigm. The world of CI/CD can take MaC deep into its structure if we make sure we adopt the evolved MaC 2.0 at its core.
MaC 1.0 in CI/CD means using monitoring tools to test application and related services operational health and wellbeing. MaC 2.0 in CI/CD means using monitoring tools with codified reactive logic to build and test applications. And to automate their operational wellbeing in live production environments.
As the use of MaC 2.0 grows in contemporary CI/CD environments that make significant use of containerization through technologies like Kubernetes, along with artificial intelligence, the ‘common packaging’ of containers can enable us to extend CI/CD into an even more formalized pipeline.
Sumo Logic’s Porter refers to these developments as the move to new new Infrastructure-as-Code, a term that clearly needs reinvention (“neo-IaC,” perhaps). Additionally, this also clearly applies to how we build CI/CD pipelines with more operational functions – including monitoring and observability – baked in.
More Closely Monitoring the CI/CD Pipeline
It’s important to create a more operationally intelligent, reactive and functional Continuous Integration/Continuous Delivery (CI/CD) pipeline that doesn’t just “do and view.” It also “sees, understands and reacts” so this new IaC substrate can help build a system to serve highly demanding cloud services needed by so-called high-velocity, cloud native enterprises.
“Monitoring-as-Code (MaC) solves a bane for many projects whereby unexpected issues during pre-production testing or deployments go undetected. We lose hours allowing failing tests to continue, then more time troubleshooting the problem, and of course, we miss the opportunity to investigate root cause at the point of failure,” explained Seng Phung-Lu, AVP of site reliability engineering at TD Bank.
Phung-Lu says that with MaC tooling deployed alongside an application via a single unified pipeline, he and his team are able to catch any issues early and avoid (as he puts it himself) having to “manually babysit” the testing and CI/CD process.
The Ultimate Goal: Continuous and Complete End-to-End Monitoring
The real value of these emerging technologies may be a new definition of what we mean by end-to-end monitoring.
Back in the day, end-to-end monitoring used to mean observability applied to everything from the device Operating System (OS) and application User Interface (UI) to the compute, storage, analytics functions and back again to the user. The new end-to-end monitoring is all of that, plus an ability to deliver monitoring across the entire breadth of infrastructure that all of the above runs on.
In the virtuous circle of MaC 2.0 inside an end-to-end neo-IaC environment, a new symbiotic relationship starts to develop; application metrics are collected and detected automatically in response to code changes and new deployments. Onward extension of the IT stack benefits from the “learned mistakes” (code execution failures) that have already been experienced and the stack itself draws upon its own evolutionary DNA.
About the Author:
Adrian Bridgwater is a leading journalist covering emerging technologies.