Chronosphere’s Ian Smith on Cloud Native Observability

hi I'm James Maguire and on today's e speaks we're talking about Trends and best practices and observability an emerging technology that enables companies to better understand what's going on with their technical infrastructure to discuss that I'm joined by someone who knows a lot about the topic with me is Ian Smith Field CTO at chronosphere Ian very good to have you with us today thanks James it's great to be here so I I've got some questions for you and I really want to talk about the observability market there's a lot going on there uh before we get started with that I'd like to get a sense uh please tell us what Kronos feere does I think obviously it's a well-known company so many people already know but for those who aren't as familiar what does coros do for for other companies sure chronosphere is the early observability platform that's built to empower devops teams Sr teams and controlling the speed scale and complexity that comes with the technology and organizational changes of a cloud native World particularly as you adopts microservices containers at at scale like all monitoring and observability Technology The Guiding Light of chronosphere is in facilitating Engineers resolving their infrastructure and application issues before they affect customer experiences and that bottom line uh but we're also trusted by the world's most Innovative Brands including door Dash snap and Zillow to go beyond those fundamentals so riging in costs improving develop productivity uh increasing that customer satisfaction and enabling more rapid Innovation to develop and maintain the competitive Advantage all right great certainly a market for that no doubt about it so let's talk about the observ observability market it feels like it's come up in the world the last few years I sense a lot more Buzz about the market and uh so so as as you surve surveyed the observability sector going into 2024 what trends do you see currently driving driving the sector yeah so I think there are some interesting ones not just in the tooling side of things but across the board and this sort of in my mind refers to the overarching observability strategy that I'm sure we'll talk about as we we go through but uh there's things like the adoption of Open Source particularly in the areas of instrumentation and data collection Prometheus has been popular for a long time uh but things like open cemetry I think are really uh you know hitting stride now and it's offering customers and Engineering organizations a lot a higher degree of control and choice and that I think is also underpinning a lot of what we see on the tool selection side of things as well uh you know historically there's probably been a little bit of uh best of breed SL I can assemble things myself where I have maybe a hodg podge of of different things for metrics different tool for tracing or APM um and a different tool for logging and and various other things sort of assemble together it's oftentimes I hear a lot of customers talking about oh well actually we have you know six seven eight nine different observability tools uh that touch that experience and there's a really big focus on tool consolidation now um but I think underpinning that is a really key Focus on what is the value I'm getting out observability and am I getting the correct outcomes that I'm looking for so I think there's in some ways an interestingly for uh those of us working in the vendor space there's a high degree of skepticism now about well what are you providing me other than features and functions um and then I think you know underpinning um lots of these conversations particularly with the current economic environment is am I getting value for money am I getting return on my investment so costs and utilization value uh those are also questions that people are heavily thinking about as well you know it seems like it's a key question and and not to delve too deeply in this but but how do you counter that skepticism when someone says hey wait a second do I do I really need uh an observability application seems like kind of a no-brainer but what what do you say to customers in answer to that oh for sure and organizations definitely believe in the value observability as a concept and obviously need the tooling but I think the skepticism comes in terms of of well what am I getting Beyond sort of the highle marketing message uh and then one of the things that I see in my role is a desire for customers to be able to understand uh what am I getting beyond the tools the features and the function and am I having a partner in this uh obviously things like customer support but it's like what is the thought leadership where are you going uh am I going to be with this particular vendor um and have increased value over time you know over the course of a multi-year relationship um or am I going to be able to only really take advantage of some features and functions and deal with lowan fruit where you know all of the value all the differentiation that I'm going to get is very heavily front-loaded in in the engagement gotcha that makes perfect sense all right so I I know there are some challenges in the observability sector some companies uh it doesn't work perfectly some want to do some things that's hard to do what do you see as common challenges that companies face with their observability deployment yeah so for example as companies think well maybe I'm not getting value out of something uh you were dealing with issues like well the the growth the cost of observability and this is not just just the the hard dollar cost um it can be the you know the the infrastructure cost it can be the cost of people involved in managing and extending the observability function so for example in this case just deploying open source observability backends is not a Panacea for handling cost but but handling that cost um is is really a big challenge we see often times uh companies coming to us talking about super linear growth in their costs so to be more specific let's say that your revenue is growing at a certain percentage maybe your infrastructure say for example your AWS or gcp cost may be growing faster than that because you're having to invest in R&D and expanding your Footprints but then observability is growing even faster than that and that's obviously a big concern to a finance organization it's like well our revenue is going at a certain rate uh costs are going uh a biggest cost maybe the think think on gcp is is or AWS is growing at even higher cost and of course there's fops Focus there um but then observability which we as we talked about is critical for for maintaining that customer experience that's even growing even faster so it's a very big challenge there um and then of course oh we decide to move but um the friction of moving can be very high because we've locked ourselves into a proprietary ecosystem and of course that ties back into what I said before in terms of the adoption of Open Source giving more more freedom um but doing those things and making sure that any purposeful action that you take in observability ties into an overarching strategy is something that I'm hearing uh customers have a challenge around it's it's not I can just buy a tool but now I need to think about specifically how does it fit into that strategy and I I really solving uh true problems there um on the cost side of things um you end up having to throw a lot of people at the problem uh because inherently many vendors are frankly disincentivized to help you manage cost right it's you see the dichotomy of particularly some vendors in the market go hey we've added cost management features for your your Cloud your AWS and your gcp and uh some of the more skeptical reception for the market is great when are you going to provide similar tooling so that my metrics or my logging bill is also decreasing or or even static yes right well all right you've talked a bit about how how to address those challenges if if if a company came to you and said you know Ian we're having some struggles what are some ideas and best practices some tips for actually handling those challenges yeah so part of my role U not as a you know necessarily tactically selling on a day-to-day basis but looking at the way the market is is going and having some of those conversations with with organizations as they they think about the bigger picture um and Al also seeing how uh you know companies have been successful in making some of these big Transformations they go down in the cloud native pathway I really think the core of it needs to be um being very deliberate about an observability strategy and so for that uh One customer they talked to really summed it up very well is we keep buying tool after tool after toll uh and we think that we're buying Solutions but internally when when I when I asked around I said well what is the problem we're actually trying to solve we weren't all on the same page and so how can you possibly buy something thinking it's the solution for for a problem if you haven't actually defined that problem up front so at a at a basic level the observability strategy not just tool selection it's what are we trying to accomplish and usually it's tied into the business objectives it may be we need to move to the cloud it may be that we need to adopt Cloud native really aggressively um and then how does engineering facilitate uh what the business needs and then going from this top down basis is what is the value we're looking to derive from reserv ability allows to really filter down to what matters and it may be you know the ability to control cost is incredibly important or you know when we previously uh settled on our observability tooling we're a much smaller company we had really big focus on observability being used by our most senior resources and they drove the evaluation now we may be you know 10 15 times more engineers and it's a very broad spread of experiences so we need tooling that facilitates us to not be bottlenecked on those most senior Engineers we need everyone including you know the the engineer who's been here four months is about to go on call how are they going to be able to take advantage of those things and not constantly page the same people who were very Reliant upon from an innovation standpoint so being deliberate about those strategic objectives and then filtering that down and going well maybe you don't even need to change an observability tool maybe it is that um you know the key of it is uh we need to direct a large portion of observability data into uh some other area maybe a data Lake because we've been AB abusing our observability tooling and these are all strategic initiatives that come out of really sort of stepping back and and looking at that bigger picture H interesting all right so let's drill down to what chronosphere does I mean in terms of really addressing the observability needs of its clients I mean what is the chronosphere advantage yeah I think fundamentally and I I've worked with a lot of uh vendors in this space unfortunately last 10 years for better or worse but to me one of those key differentiators the one that shines through a lot to our customers is that fundamentally in philosophically we're built to solve organizational pain which sounds very fluffy I realize that but to be more specific it you think about the kinds of things in a in a company that that sound bad or painful and constantly bring up friction so you know Finance complaining as I mentioned before about unpredictable overages or a bill rapidly escalating and the other side of it might be you know Engineers saying okay well I don't have the visibility I need or okay I I'll I'll drop that data because um that appears to be sort of you know the the most recent thing we've added and that's the thing that tipped us over the the edge of getting an over those are organizational pains it's you're losing visibility or people are are not seeing the value of that observability investment and so um being able to solve those problems um ties back to the DNA of the company and and where it came from which is ultimately uh the founders were at Uber and Uber because of its scale wasn't able to just go out and you know sort of have that n- joke reaction like let's buy this let's buy that they had to be very deliberate they had to come up with a strategy and they had to to drill down into what really mattered because they were going there was going to be a significant engineering investment to to build tools that would support their particular needs particularly because they were early adopters of cloud native and so um some of the the things that the team tackled there are very much part of the core chronosphere offering so the ability control cost you know we've sort of talked about at a surface level but one of the things chronosphere does allows to understand the value of your data who's using it what are they using it for uh how much are they utilizing it and be able to compare across those data sets maybe you have some data over there that's only used um for you know once every 3 months for capacity plan but it's very very small in its footprint there's data over here it's used every day for instant investigation but what pieces of that are actually core to uh what you need to answer those questions we also secondarily refine the data set down dynamically and we really from a business perspective only charge you for the data that matters and that you've chosen to to store with us so you can generate as much data as you want figure out what matters to you reshape that on the Fly and then pay for that and so that as a a core capability really helps you address that fundamental cost problem of well data is exploding I have all this stuff coming from candidates and microservices I don't know what's valuable and uh you know I'm under a lot of pressure from Finance particularly in the current economy not to double or triple that over the next 12 months right uh you Ed interesting phrase of course Cloud native I mean we know what CL native means in general but what does it mean in terms of the visability sector and and why is it an advantage yeah so there's from the observability perspective you tend to get a massive increase in data volumes if you think about it just as simple maybe you were you know a web first company you maybe deployed in five different regions in each of those regions you have big monolith uh maybe even just running in 1 VM it's uh generating you know certain data points about performance and those kinds of things you basically just have five sources of data and then you move to uh a containers uh deployment ignore microservices for a second you just have a containers deployment and you maybe you're looking at 50 to 100 different containers in every single region you've now increase the volume of data just because by nature of running those in containers uh 50 to 100 times then you add on microservices you have another multiplier on top of that I've decomposed my monolith into 15 different microservices so there's a volume of data problem which obviously speaks to the cost piece that I I mentioned before um but then a secondary aspect of it is things just got a lot more complicated for anyone to be able to operate and particularly diagnose where problems exist and so it feeds into another problem of okay um my Engineers used to be able to you know pretty readily keep all the context in their head um they had a you know enough tribal knowledge enough context built up that oh I can look at these logs I can look at these metrics and I can solve my problems but now with this complexity I don't know where the problem lies also I'm not talking to the teams who are uh building the new microservices um on a maybe weekly or monthly basis pardon me and so now I don't know where to look and so oh I need to adopt distributor tracing but I'm not a distributed tracing expert and so this this aspect and I alluded to before of like the audience uh sort of sophistication and the complexity problem um re really is a big struggle and making sure that the tooling can speak to everyone and also that you as an engineering organization are able to put your fingerprints on things you know build your workflows and not just be beholden to uh sort of a an unrealistic ideal that the vendor has decided oh this is great for a marketing purpose um this is the ABCDE sequence that you should always take your investigative problem um often times I hear Engineers are very frustrated because their most um their most senior resources like oh well it's super easy you just go from a to but the product doesn't support you going and skipping BC BC and D being able to bring those kinds of things to uh the the audience and allow them to have you know that distillation of institutional tribal knowledge um is very powerful the big question is the future of observability I think companies want to know so they can get ready now what do you see a few years ahead in terms of observability where's the sector going yeah I I think particularly as we mentioned before the open source instrumentation and defining what data is important to you and being in control of that Destiny is going to become very very common place as already we really have those sort of early adopters are in there and they're really pushing the development of things like open Telemetry uh but I think seeing the vendor adoption is also another piece of that and I think that's going to extend forward um into various other parts of it so you know how do I control that data how do I how do I interact with it as well I think that uh extending the skepticism thing that we mentioned before I think companies are going to become a lot more focused on what are the outcomes and the value and it's going to be less of uh necessar tactical features of course features are always going to be important but the the the level of in inspection of well what are the outcomes how can I uh be confident about the adoption of this how can I be confident about um lowering the bar entry for my junior Developers for example how can I be confident in the ability for my senior Engineers to not be on call as much so these are those fundamental things that maybe don't immediately map to a feature or function but uh the the the market is going to become very very picky about and and force sort of a a bit of a change there I do think that uh you know the AI uh as a concept has some very interesting applications I'll be a little bit of a wet blanket and say we we can't just pin our hopes on AI and just sort of sit back it's not just a case of let's pump all of our our data over to Aon Ai and and have it come back to us um but I do think particularly generative AI has uh very important ramifications about how do we express ourselves to the to an observability system and how can it express itself back to us to so to be specific uh anyone who spent a bunch of time in observability has probably struggled with uh either a point and click or a query language interface of I want this data and I want it in a certain way and ultimately that is you know expressing yourself to the system but from a natural language perspective you say look tell me what's going wrong with this part of the system and then be able to use those sort of same ways of expressing yourself to really narrow down the search you know discard all the data from non-paying customers let me focus in on just paying customers okay which customers are currently being affected what's common about the customers who are being affected how long has this been going on these are all things that we have been asking since the dawn of time of Monitoring Solutions um but there still quite difficult for us to express in the system so generative AI has that I think opportunity to to Really uh Advantage us there and then also the data coming back right it's it's one thing for a a Raw Feed of data to come through and you know fill up a terminal and hopefully we're we're P those points but even you know the very dense graphs and dashboards that we've we've built up they don't necessarily highlight to us exactly what we should be looking at so the system saying for example you know here's a prro of data here's a dashboard but you should probably look here here and here and also last time these things were caused by this this and this and this is what you did to solve them it's really just allowing the human decision-making factor to come to the four rather than that you know sort of very heavy manual exploration of the data that's often times relying on tribal knowledge interesting you it's it's it's a lot of good information I think I I definitely learned something thank you so much for sharing your expertise today and uh please come back and talk with us again sometime of course thank you so much James

This transcript was generated automatically from the video's captions and may contain errors.