DataRobot’s Venky Veeraraghavan on Handling AI’s Challenges

hi I'm James Maguire and on today's webcast we're talking about the challenges the companies face as they work to build out their AI deployment every company knows they need to get on board with AI but the difficulties are numerous including cost compliance deciding what vendor to hire to provide insight into handling those challenges I'm joined by someone who knows as much about the AI Market as anyone with me is vinkki vrag gavan Chief product officer at data robot venki absolutely thrilled to have you with us today nice to see you again James it's I'm very excited to be here and I'm I'm assuming that data robot's probably keeping you busy and the company appears to be in a grow a rapid growth phase I I I take you you're pretty much working these days yes we are working very hard and a lot of exciting work uh you know between the releases earlier this year and the next one coming up in the fall uh lots and lots of cool work to do great well I think the the question is clearly Rises are eager to gain competitive Advantage from AI or expand the advantage they already have but there's challenges so what do you see as the biggest challenge or or or some of the biggest challenges for companies who want to deploy AI so you know I would think about like three things there's a combination many of them like I try to think of them as top three one is sort of like is the AI that they're building out is it valuable is it valuable to the user is it give does it give business value help the bottom line or Top Line uh is it safe to have confidence in how it works Works can they can they believe the results can they let their employees and their customers use use Ai and sort of not make mistakes and not certainly not get on the Press uh and the third one is like do they have the expertise to run uh the solution in the long term so it's I think it's one thing to say you know a team got together and you know built a built a little PC and did a demo but what does it mean to run over time and that's when some of the things you talked about in terms of cost in terms of compliance uh how do you sort of make sure it runs long term and is the company set up for it well think on that safety point I'm I wouldn't say that a obvious is not dangerous but I think there's a number of unknowns especially for companies who aren't as familiar with it maybe an employer employee is going to enter some information into the the app that is is actually proprietary maybe the app might hallucinate uh there there's still a little bit of a we're not sure which direction this is going aspect to the safety question agree disagree no absolutely I think that's you know uh you know we talk about it as there's a confidence Gap it's not that it doesn't have benefit and you know it doesn't have lot huge promise absolutely does the question is like will you depend your business on it right and and you and for all the same all the reasons you talked about so I think like a lot of the work that people have to think through is how do you know that the solution is going to work as you intended it like the spec for that thing how do you know it's going to work what do you do when it doesn't work and that's one of the key differences you know that change from like sort of plain software to machine learning is that machine learning is sarcastic so you don't know it's always going to do do the right thing always give the right answer and so the the question is how do you build the systems in place so it's business process the UI that really solve for what happens when it doesn't do the right thing so imagine you said someone does provide uh pii data to the to the to the model can you do you know it first and then do what do you do about it second and so so I think like that is kind of what we call the confidence problem and I think that is definitely keeping a lot of interesting poc's from going to production because they just haven't quite nailed how to sort of make sure that they can control it and understand what it's doing in in at run time well okay that makes perfect sense I want to make sure I give you you mentioned these these three challenges and you address that one are you able to address the other twoo the answer is there's an answer I I'll give you my point of view on that you know on the value front which is like you know is it valuable you know the uh I would say last year there was massive fomo with Gen everyone just sort of got in there and sort of did something right and so right is good and you learn a lot and then you realize that okay not everything I did is necessarily valuable for the cost that it has so you know values like you know cost cost against benefit sometimes the benefit is not there and the and the benefit may not be just that you know it doesn't do the work but it's also did I get the output of the AI in front of the right people who can use it right and so you know Microsoft talks about copilot which it gets in front of all the users like when if you're doing an Enterprise app where is it or is it you know if you build an Enterprise model uh or a solution where does it show up do does it show up in Salesforce does it show up in maretto does it show up in your custom app and so I think one of the big sort of issues with value is like you have to get the the AI so to speak if you can talk talk about as a noun how do you get it in the place where users are using and doing their work right so the business processes the business workflows you have to get it in there and that is actually kind of not trival in fact you know you spent a lot of time in the last few decades when you're doing digital transformation it's like sort of getting all the information from the right place to the right place we talk about in bi the same thing so I think we have the exact same problem with with AI which is like it's very very easy to build an AI model a rag solution or a agent and you say this is awesome but then you say d how do I get in front of 100,000 agents or 100 you know million customers and how do I know that is the next level of work you do so you bridge that Gap makes sense I mean the the board might ask you are you using AI so at least you could say yes but then the question is is it really giving you value that's right and and and then when you start getting the bill then you start wondering like hey look you know i' see you know clearly the board liked it or the exacts who saw the demo liked it but then the question is like who's going to pay the bill if not everyone's using it and they and you can't measure it so I think that's one of the big things uh I would say you know that we certainly have a point of view which say like you know instead of just counting on novel new techniques obviously there's a lot of new novel new techniques going to come out like these agent agentic workflows but there are also like simpler ways you like you know the existing workflows that you have how do you in the AI into that and sort of really integrate such that it works as part of your you know your marketing campaigns and you know so that that I think is an important part of how uh you sort of bridge this value Gap um which is not just focus on like does it work in a demo but really think to like you know what business process is it improving can I AB test it can I make it can I measure the difference did you get a chance to talk about your three points and you talking about the sa I talk about two the third one is expertise you know on on the expertise front I would say uh uh you know AI has been traditionally the the space for uh data scientists uh and and you know analysts people people doing data science but as you look at gen you see that a lot more people need to be involved to actually make it work so whether it's a product manager or a product owner or a pmm a product marketing manager who owns a product who says I want to define the experience for my for my consumer so it's not a data science you're not building a model anymore you're building the user interface you're specking the user interface how does that how does that person come in uh you were talking about how do you sort of package this bigger solution which is not just a model but you know the integration with a database and integration with the application so you want to talk about now software developers were now involved and so having a sort of a a a common ative upon way in which all these folks work together they have the right tools and they are quite different people like you know some the pmm and uh the business owners might be no code developers are all code how you pull something together that makes that work I think is the is the third part of um sort of the the tool chain that we have developed and we have sort of adopt to actually make make Solutions work at scale well it brings us to the question of costs I want to get into cost but something you said reminds me you know hiring people with the expertise feels like at this point it's exceptionally expensive it may not be as expensive going forward a few years from now there'll be more people who have the expertise but now it feels like still that that person who really has the AI expertise can be kind of a a rare bird and so an expensive bird so to speak sure I I I think I I I think that expertise problem has become better but it's not done yet right and so I think for most Enterprises if you're you know if you're trying like five years ago even getting a sort of really good data science was a very very hard job I think a lot of companies now have great data science teams now the question is like you know do you have the rest of it and you know I think there a set of U data scientists are sort of building these Frontier models they're incredibly expensive we see all see news reports about that but most companies don't need that what they need is really F folks who understand the business know how to use this tool chain instead of collaborate to actually build the solution and J actually makes it lowers the barrier for expertise in a lot of ways to Sol solve these problems because you're not building these like esotic models that's coming in open source it's coming by apis and so in some ways that's been factored out and become more of a platform and now the you know most Enterprises have to sort of figure out how to use all that power as opposed to how to create all that power well all right then then and cost seems like it's it's the thorniest issue I'm imagining an executive out there and he or she thinks well we need to do this with AI what's it going to cost it's not like you know going to the car a lot and realize it's going to cost $35,000 like no there's so many variables to go into the cost of Enterprise AI from hiring to you know hiring a provider and thoughts on this I mean how can it be more predictable I would say it'll get more predictive as we have more and more use cases that have been successful so I think this is one of the general problems of a early early space is that there's you know there's no uh best practices has been written down that says how to do it so I think I I would say uh most of the customers we talk to they're all on this Learning Journey themselves and the question is how do you go through that Learning Journey quickly right and you know one example i' would say is like you know again we talk about 18 months ago seriously the only real game in town was open AI right and so everyone's like well I have to do it and it's a big bill to pay but fast forward 18 months and now you have thousands of models there are some Frontier models that many many provid as a frontier models there are lots of Open Source models and there a lot of like sort of both commercial and uh sort of uh self-built Tas specific model so now the question is you have all these things for your use case what is the right one right and so how do you sort of think through that that I think becomes a more interesting problem and you know you could say like I really want to use a very expensive Frontier Model if the return is super high and I I know it's totally fine but if you're trying to do a much smaller problem that like you know you're trying to save cents or dollars on a phone call uh then you may not want a $35 you know bill for each uh for each engagement ment from a from a llm and so I think you what you have to do is to figure out how do you pick the right performance benefit uh or cost performance on that curve how do you pick that and that that I think is one of the big problems that we have to solve for and uh we have some work hooking up in that space But I'd say it's you know it's a it's a problem that everyone faces today and so while it's not 100% uh you can do some amount of cost planning uh but in the end like I don't think someone can say hey look to solve this problem use this model it costs this much for X number of users like we haven't we don't have that Baseline yet right I I I wonder if that will ever appear maybe maybe not some of you said reminded me that 18 months ago open AI was the only game in town it seems that as AI is developing it appears that artificial intelligence will not be controlled by a few modic providers but will be about Partnerships and alliances agree disagree on that one uh no I I I definitely agree I I think there'll be a small number I I don't know it's one or you know three I think that there might be like you know a dozen you can think about anthropic and go here there's a number of people who are really investing like you know tens and hundreds of millions and billions of dollars on building these Frontier models I think I think that number just because the size of that uh the bill there'll be a small number of people building that those models but most of the time you don't need all of those models for doing all the basic work and so I think that's where you know you see the the work that La meta did with llama 3.1 like it is really really good and it's free and that has multiple sizes so you can sort of pick of start picking where where you want to be on that on that price price performance curve so I think uh what we'll see is a much more heterogeneous sort of like Model area and you can see a lot of the big companies uh and sort of companies like ours who are like offering customers a variety of models so there's a you a model Garden or Bedrock you know there's this idea of like look there are lots of models and you have to choose what's right for them and then really the interesting thing becomes how do you pick the right one right that is in the right space for correctness the right space for uh sort of the kinds of experience you want to build and the right cost structure to actually deliver that well I think that is definitely one of the questions toour how do you pick the right model someone's shopping out there for a model do they pick a proprietary model do they pick a small customized model and any thoughts on how to pick the right llm um so I would say today it still feels like at least to the best of my knowledge like a lot of work uh in uh manually doing this work and so uh at the of sounding a little nerdy like you know if you think about like a sort of classical rag pipeline there are lots of choices you to make you to make like how do you do chunking you to do how to do uh retrieval right and what are the embedding strategy you also have temperature and you know uh prompt strategy you also have sort of deployment and sort of things like that so if you think of all of them as hyperparameters what you really want to do is sort of say optimize them across the board against some Metric you're going to optimize for right and so that's you're saying hey look it is I'm I'm optimizing for accuracy or I'm optim optimizing for non-toxicity so pick your set of evaluation metrics and then you try to sort of pick through the space and each model becomes one of the hyper parameter so to speak that in which you work through it so that I think is the interesting problem for us to solve and funny enough like it resonates well with us because that's kind of what we did with classical ml with automl right which we sort of explored this big space and said like here are some interesting places where you can sort of look at and so um I I feel like there's a lot of interesting work we can do in that space um and uh sort of help very quickly figure out how to explore the large space of models and these parameters and then how you get to the small part that you want to exploit and go deep into sounds good actually okay so I want to make sure I give you a chance to talk about data robot uh what's new what the customers need to know what what's happening with data robot um so you know data robot we're always moving uh and so we had uh um sort of a nice uh spring launch and a summer launch now we start heading into the fall uh and so you know the big things we're working on is getting uh deeper on the Gen space sort of like making sure that we give customers more uh more capability sort of you know we've learned a lot from customers in terms of what we trying to uh What uh problems we've had sort of like you know getting these things deployed at scale and sort of taking that learning and putting it back so J is sort of clearly a huge part of it uh we continue to focus on our current customers who have sort of bought us for democratization and uh and sort of predictive AI so I think you know we one of the unique platforms that have both full tightly integrated uh and then I think the part we also sort of very uh sort of focusing on is trying to figure out you know we talked about the value Gap how exactly do you specialize and sort of create this end to end sort of flow uh that is sort of like that's repeatable right and so that c you know customers can say hey look I solved this churn problem once I want to do it again for a different department or I want to do it for a different uh use case how do I repeat and sort of like so we think about this like how do you sort of templatized so to speak a solution and then uh and then sort of uh uh sort of repeat it over and over again I think that's going to be one of those uh things that we have to develop as an industry to say what is the best practice you know can I can I see it in code and then can I improve it because that's kind of the how software developers think about uh about work and so I think that that's the space I think we sort of like doing a bunch of work I don't want to sort of uh uh out myself and sort of like scoop myself so I want to wait wait for uh wait for the launch to actually talk about all of that work sure sure you used an interesting word in there use the word democratized and I think when I when I think about the future of artificial intelligence Enterprise AI it's going to be about democratization and that's going to be accessible to all people in many different layers of the organization apart from that the F future of Enterprise AI does it involve democratization to you or what what do you see when you look to the future of Enterprise AI um you know I definitely think democratization I think but you know I'd say democratization especially in my uh line of work with sort of our our history has had sort of a little bit of a sort of an asteris hanging over it which is like you know we talked we talked about data scientists who were doing code and then we said like you know this this the citizen data scientists who can build all the stuff without any code and that has sort of been mixed we've certainly found some but it's not as many citizen data scientists uh as we the entire industry expected but I think that there is democratization a different way in the sense that more of your employees will be involved in either building AI or in sort of consuming Ai and sort of having a system and a tool set that will help sort of support that I think is definitely in the cards I I don't think it is a a sort of you know a high priest of data scientist who can sort of you know describe the answer and everything works from there it will be a lot more sort of uh built out uh with across the company we talked about you know marketing managers and uh product owners who can decide how you know what the voice of their bot is all the way to like software developers are going to be integrating all this and making sure that it it runs at scale so I think I definitely think of democratization in a different way now Rel to sort of how we started it about maybe five years ago well and one one last thought I mean the idea of you know no code AI it seems like it's almost an oxymoron I can actually bu an artificial intelligence application you know with a no code platform that's that's pretty wild that seems to open up a pretty big door it does and I think J has helped it a lot because um you know the the before you know we had drag and drop and things like that sort of sort of on the UI but now we can actually use a natural language interface and you know using a lot of the Coden capabilities you can actually do a lot more with it you can sort of say hey look I want to do a SQL script I can go do data prep for it I can use a gen to actually work with a a model to actually explain what the model is doing so you can say you know hey you predicted churn with some uh thing tell me why tell me which cohort and tell me you know what are the reasons you don't have to know you no longer have to like you know understand very esoteric charts uh you can actually talk to the model so so so I think there's a lot more uh Jenny I definitely massivly changes the the set of people who can interact with AI and and and use AI so I think that's absolutely true mhm Becky I think you said it fascinating as always I learned a ton please come back and talk with us again sometime anytime James happy to be here and uh thank you again for the opportunity

This transcript was generated automatically from the video's captions and may contain errors.