DataRobot’s Venky Veeraraghavan on Building Enterprise AI

hi I'm James Maguire and on today's e speaks we're talking about getting started with artificial intelligence how companies can begin to deploy this emerging Tech we're also taking a look at some of the issues and challenges of AI to discuss that I'm joined by a a major industry expert with with me is venki vag gavan Chief product officer of data Rob robot uh vany very good to have you with us today I'm very excited to be here thanks so of course a lot of people know data robot it's a very well-known company for those who aren't quite as familiar could you give us the the brief you know nutshell explanation what does data robot do so data robot is an endtoend AI platform it runs on all the clouds and on Prem uh and we help customers sort of build govern and operate uh their AI uh Solutions so basically for generative and predictive AI applications we provide a full end to end set of tools to help help them with that I'm I'm pretty sure there's a demand for that in the year 2023 no no doubt about it that's right all right so the topic of course is artificial intelligence so much interest in it in this year obviously the thing I sense though among Executives is there's a certain amount of ambivalence about companies as they adopt AI certainly there's the enthusiasm enormous you know excitement a lot of dollars flowing into the sector on the other hand there's a there's a sense that there's a concern for the risks and the expense you know compliance security data protection you're you're senstive of this conflict what do you see going on you know I think sort of broadly the you know everyone is doing AI I think they have to do AI I think the big issue is do do they have confidence in the solution does it work does it provide value and unlike you know the last generation of sort of code when sort of soft rate everything uh that was deterministic uh ML and is sort of deterministic or stochastic that means they're sort of concerned about like this Randomness in the output and whether it is sort of whe it's probabilistic sort of outputs from a sort of churn model or you know in the case of J sort of what will the bot say right will it will it SP our toxic output so people are nervous about that and so being able to sort of uh build your business process and your your entire company around how to take advantage of the power of AI um needs means you need to sort of understand sort of how to sort of control the uh the confidence issue uh and so that is really I think the the thing that is sort of holding people back um and and and the industry is sort of building uh tools and solutions to help with that well on that very topic it seems like one of the issues is is the idea of the large language model it seems like right underneath the headlines of of the AI excitement is well what what is our llm and where are we getting it from are we buying it up the Shelf are we renting it is it proprietary to us and that may also play a role in whether what our what our chatbot ultimately says or what what you know what prompt our application finally kicks out what is your sense of how companies are dealing with the llm issue are they are they needing to build their own can they buy it off the shelf what what what advice do you give people say um I would say you know the best way to think about llm is sort of a new level of platform uh just like how we had sort of had from onr to Cloud where someone else was sort of providing you the machines you know people have made that transition you should think about this uh sort of the same thing which is like these enormous models that are being built and trained on huge amounts of data uh that are being sort of being created as platforms so basically you consider that sort of as the core and then you use your data on top of that to actually make it sort of applicable to you now there are multiple approaches obviously just like there's no one single Cloud approach there's hybrid Cloud there's all kinds of other approaches there's a similar thing with AI with these larger language models there clearly the big providers people like Microsoft Google Amazon uh and then there's open source and then there's sort of the the proprietary sort of uh sort of the second is the anthropic here Etc and so I think we are sort of spoiled for choice right now almost too many choices and so the key thing is to figure out what really works for you in your use case they all are trained with different data with different mechanisms and so that's sort of an important sort of thing which is like it is important to of understand broadly what what the um models can do and how they're useful to your specific problem now I think more specifically I'd say generally I would say start with the larger models they're often commercial uh to understand discover where the use case is and what the value can provide because they're general purpose so they're quite forgiving and sort of the answers and they have a lot more knowledge and once you understand sort of where uh sort of a use case where the gener AI can actually provide value then you can start optimizing you can start saying hey look is there an equivalent smaller model that I can that can buy that can do it you know should I instead of running it you know making a call to a an API provider can I use uh something that I run in house uh in in my own bpc uh and then you can even do further fine-tuning and say like look I want to do fine tune and build my own model the thing I would say that I have not seen yet is this need to sort of build a model from scratch pre-training from scratch is extraordinary expensive um heavy on sort of both compute data and uh skill uh and most companies don't have that and so I'd say uh you only pre-train if you have uh pre-train if you have to and most people don't need to do that it's a little bit of a long answer but I think it's it is a process you go through no I think it's a really good answer I'm thinking I the one one key point there I'm hearing and I I want to make sure I understand should should almost every company think about doing some um some customization of the model or will some some companies not really not me hardly any customization of their large language model I think it totally depends on the use case um you know I think there's a a sort of there are some places where it's just summarizing English text uh and so that the or English or any language text uh that these models can do quite well if you're doing a very highly technical subject with lots of propri data imagine you're saying look I'm trying to help a customer walk through um you know provide customer support on Boeing's engines and how to sort of help sort of look at their manual it kind of needs the manual data and that is not pro that is sort of may not be available on the internet so I think like I think it depends really on it and the best way to do that is to evaluate the answer like you know if the model is not providing great answers in general it needs more grounding data it needs it might need more pre-training uh and so the idea is that you sort of work yourself from the general purpose to the specific uh by looking at sort of the the evaluation metrics Mak sense well all right I think the big question that I think you could probably give a four four hour answer to this question but I'll I'll ask it anyways for the for the nutshell answer is try to be quiet and fast well so say there's a company out there and certainly there are dozens of not hundreds of companies out there and they're they're relatively new to AI they want to get sorted they they know the the train is leaving this station what what advice would you give to these companies that are still relatively new AI adopters you know make sure you think about this or or here's here's some good tips about growing your AI deployment yeah you know to to say it very simply uh you know I paraphrase Nike which says just do it which means just get in like you know I think I see customers are and Prospects I I talked to a lot of them uh and they're sort of doing a lot of learning they're like understanding it they're looking at a lot of the risks and everything else and so I would say that is all good but the best way to do it is to start getting your hands dirty and so using a platform like the robot that's available sort of like with all the access to all the different um models so you can sort of literally just build your own sort of uh you know bot that says like I'm going to chat with my manuals right I'm going to ask or I'm going to look at my customer support tickets and so the idea is that get it started and then then by doing you'll learn a lot more I think one thing we have realized is that this is such a disruptive technology that you know it's such a what technology I'm sorry such a disruptive technology and yeah right you all of got got to know it exactly when ago I think yesterday was chat P's one- year birthday so right right and so I think we're just all learning by doing what its capabilities are so you know the I would suggest every company sort of get in get a you know use a tool like data robot uh build something out and then say like hey look it doesn't quite work the way I want or like it's not really as exciting as I thought it would be why not and then you can start like you know exploring the different things you could say look am I building a should I build a rag pattern should I you know Rags the retrieval augmented generation pattern should I put more data in it should I put different data should I change my prompting strategy should I pre-train my model or should I sort of uh fine-tune my model these are all great questions but you get those questions by actually trying on some conrete use case so come up with an use case uh get started and then uh and then sort of learn and iteratively sort of develop the idea and you know we've learned a lot from our customers doing exactly that you know that makes perfect sense I I I think there's probably some executive out there and i' I've heard analysts say the same thing i' say well oh no you need to you know you before you R rush in you need to carefully plan your strategy and I I can also hear some executive you know in response to your answer they might say well it's it's so expensive we we can't afford to dip our toe in the in in the water and just and just start to build because what if it's just an expensive science experiment are you saying that that early building they might learn from is just sort of the cost of doing business or what is your sense of that no you know I think some amount of uh some amount of expense you have to have because it's a new technology you haven't done it before so therefore it's net new uh but I think I don't think it has to be extraordinary expensive to actually try out and learn so you know I would say like discovering the value of sort of how to use generative AI uh does not have to be very expensive now deploying it at scale can be expensive and so what you want to do is to figure out like hey look if I'm using a very expensive model and a slow model uh on a use case that you know requires billions of rows then you're going to be like look this is going to be very expensive this solution won't work as designed what are my alternative so the idea that you know you just don't get we don't get stuck saying it could be expensive you say like how is it expensive what are my choices for make it cheaper right and so so so I think that it's in the specifics that actually uh you get the uh you start moving you know to your point about strategy you should have a broad strategy that says look you know I want to get into it I have a bunch of use cases I have something in marketing I have something in Customer Care uh you know pick sort of whatever is interesting to you but then actually start experimenting and then you can say like and have Gates saying like look we want go to production until we better understand our security principles like if I'm going to send pii out it may not be aligned with our policies but the idea is that the experimentation is when you sort of understand what uh what needs to be done and what you have to be careful for so it goes from a high level strategy to more of a implementation and sort of planning process uh and sort of making that sort of taking down one one level of detail okay well let's let's drill down into Data robot and and what exactly does you spoke up front you gave a broad view but what how how really is is data robot addressing the AI needs of its clients I mean it's it's such a a mixed broad Market this I mean I think even I think a lot of companies are are not even sure how to shop because they're not sure what all the various companies do so talk a bit a you would about R robot how how is addressing the AI needs of its clients so i' see you know three ways so one is first we help customers build their solution and so and you know for building that solution as I mentioned there's a lot of experimentation you know there there is so many different kinds of models there Vector databases there are prompting strategies how do you put all that together to actually find something that is useful to your use case so we provide a full platform that allows you to do that out of the box you have access to uh you know Azure open aai service the Google pal service the newly all Amazon Bedrock Services you also have access to other open source model so you can register your own so idea is first you have access to that you can you can sort of uh create your prompting strategy uh you can build your own Vector database uh pull it all together and then you can actually chat with all three of them at the same time and you can sort of see how uh you know how they work with each other so you can say like and you can really start getting a sense of what what you know what model works for what and so that way you are first you have confidence that you have a solution that works so that's the first part second thing is you want to sort of make sure that you can govern all these things so there are now lots of people who are working on the same problem we have software developers who building the application you have you know product owners or program managers uh project uh product managers who are sort of figuring out the voice of the bot the voice of the Red Bull bot proba is different than the voice of the Ikea bot that is not a develop thing it's a sort of product owner thing and so they have to figure out how to get the right thing and then you have the data scientists who are like building the evaluation metric so you want to govern all these personas talking to each other and transitioning the state so you want to make sure there's a great way to do that and D robot provides a registry and a full experience for managing that and finally you want to operate this at scale so when you want to sort of deploy this model you have to understand like hey look where is it being deployed is it being deployed inside your Cloud environment next to your application do you want to run it on a on a hyperscaler um and then once it's running how do you make sure you learning from from production right you how you know you understanding what the inputs are what questions are coming in what are the outputs how are you measuring the the value of sort of the measuring sort of the outputs and making sure that you can intervene so if there's highly toxic content or more most of the models are getting quite good at sort of toxicity out of the box but you know the the the bot is talking about something outside of the topic you wanted to do you're like you're you know you're saying talk to the manual but it's talking about music that's not what they want and so so how do you sort of how do you want to intervene and make sure it stays on stay stays in its Lane so we provide a structure really for building the solution for governing all the different artifacts and solution and then running in operating that at scale that's great and obviously of course there's a huge demand for that I I think the really interesting question that some people are are are grappling with is going ahead what what's going to happen with the hyperscalers of course offer some of these services and you you mentioned some of the hyperscalers offerings in there uh and then there's Standalone companies like data robot um are they not in opposition to each other the Standalone versus the hyperscalers are it was interesting you really mentioned about working with them so how does how does that play out will the hyperscalers own the future of AI in terms of being a vendor or not necessarily um you know I think like like with every single sort of platform application know I I worked at meop for a long time there was the office and windows thing like you know and so you said you know there are there things other than office yes there are and so I think we we will live in a world of coopertition we sort of you know we will build sort of uh sort of uh capabilities in the same space but the focus really is different hyperscalers are really focused on scale and infrastructure and sort of making that easy to use uh I used to work for one uh and I delivered sort of a lot of the sort of the work there and so I know sort of what we optimize for and while dat robot is sort of you know structurally able to sort of look at hey look we're taking all of that hyperscaler stuff for granted we actually build on top of them so we do build on top of the work that vertex does or azl does or Azure does or um um or Google vertex does and so the idea that you know and sagemaker so the idea that we build on top of that and and on the data platforms and we add additional value that really helps it really make it very simple for customers to sort of Do The Last Mile which is like where is the value in the solution so instead of you know once hyperscalers have abstract a lot of this hard work we say look let's build the UI and let's build the sdks and apis that make it really really easy to validate is the solution giving you value and how do you sort of run this at scale that's one the second one is you know we are an independent software vendor so isv and that allows us to be Cloud neutral now not every customer does not have a single Cloud to work on uh and even if they were to start a single Cloud they might make an acquisition or might get bought suddenly they back in multi Cloud environment we provide a single way to sort of have a cross Cloud version of how to sort of uh work with all of your um uh sort of with all of your Solutions so some application might be on one Cloud some might be the other cloud and finally we we run um on on Prem so in a hybrid environment and that most clouds cannot sort of reasonably say so I think we have a a space we in fact we are very closely partnered with all three clouds uh we've just announced a bunch of Partnerships with all of them uh and we actually work closely with them in on the product integration as well so we think it's a you know we think it's a uh thing that we both learn from each other and it's a it's a healthy place to uh to be that's great all right so it's not it's not an either or in the future it's it's going to be a Cooperative or coopertition sort of scenario makes perfect sense and that's if you look at the history of it that's that's really the history of it it's always the way things have worked uh all right the the future then beny here's here's the big question the future of AI and the Enterprise billions of dollars rests on the answer to this question because I I think companies want to know where's it going what's it going to look like it's a you know 2026 2027 and how can I get ahead of it now uh what is your sense when you look into your chist Paul what do you see for the the history the future of AI in the Enterprise you know I yeah I definitely think uh there's going to be a lot more I think there's a lot of uncertainty there's a lot of questions but this is such uh technology so much promised that I think we will definitely see a completely different worldview of sort of how AI is use and how sort of AI just accepted uh you know I'd say I worked through the sort of on Prem to the cloud transition and you know at that time it was like unclear whether you know it's a big deal you know 10 years in I think I started this in 2022 uh 10 12 years in um it's kind of the default and of course people you know on Prem but like it's the idea is that you know the cloud is s of the default starting place for this so I think that I think we'll be in a very different place so I think the large Arc as I mentioned earlier is definitely huge amounts of AI in the Enterprise now the question is how do you get there and I feel like it's like a river getting to the Sea it'll Meander and there'll be you know obstacles and you have sort of work around the hills and whatever else it is and the Rocks so I think that I think will definitely happen and so the way I think about this is the way you get ready for that is to have a platform or a tool set that allows you to sort of AC accommodate for those things like you know the the the the technology will change you know I would say you know the events of the last few days from with open AI like it is so far like one of the one of the best uh models there is but you know we were concerned about it you know whe the compan is going to exist uh and so so so even such a powerful and great company uh suddenly you know if you have a one one uh a single sort of Provider you have issues and so sort of you having optionality and changes not just for technology change in sense like which model is better but also like you know which vendor is viable and some will come and some will go and so having a platform that is open and has optionality for sort of solving for those things is super important and then making sure that you know you actually can measure the value you can measure the cost so you're not uh to sort of you know I had one customer who just said look I have 50 VMS in Amazon just running different Genera I thinks I have no idea who they are but I'm getting this huge bill okay that is not sustainable in the long run and then that is what causes people to sort of say hey look I want to turn the thing off because I can't make out the value so having a uh you know a platform that allows you to observe maybe even sort of behind the scenes passively sort of where people are being used and asking for business value versus cost you can have these great conversations for Roi you know I think I would say we are I think as we all know we are deep in the hype cycle and so there is a lot more investment and sort of innovation trying to find out but over time reality will set in saying like look I'm spending the money am I getting the value and so you really want to sort of focus on what are the use cases what is the value what is the measurable value that I can get and you need to sort of have a platform that will help you do that and that's kind of I think the way you sort of get from here to there which is the path is unclear because you like as I said finding a way to the sea but the the the core constructs are all there we we know the details we know what can happen the specifics may be different yeah I like your metaphor of the of the of the sea finding its way to the the ocean the river Finds Its way to the sea I think it's really pretty accurate it'll be under it it'll hit some obstacles I I I sense that one of those obstacles is sometimes I'm not sure if everyone in the sea Suite really understands the technology and that artificial intelligence is not like other Technologies and that it can learn by itself I mean cloud computing was a big step forward and we've had a number you know even mobile was a huge revolution in many ways but it feels like AI is almost bigger than all of them and it's a complicated technology you know neural networks and deep learning and what it all means and you know large language models I don't know if if if if you know the water flows to the Sea if sometimes all the all the executives fully understand what's going on yet maybe they don't need to but you you sense that the people in charge are are really understanding the technology uh you know I think I don't think everyone understands technology I think it's moving really fast that doesn't mean uh you know even cloud or any mobile there's so many complex things lower and lower I think the question is how do we simplify it and make it into you know into a set of things you know questions that the execs can answer at that level and so you really want to make sure that you know the at the higher level the Strategic level you have to figure out look should we invest how much should we invest what is the value of this thing what am I you know how do I protect myself um and how do I make sure that the company I don't put the company at risk those are questions at the seite level and I think absolutely people should learn enough to understand those things I think all the vendors us included you know are working very hard to sort of do the education that happens uh in every one of these sort of disruptions um but I don't think everyone has to understand how you know uh a KNN search works or a deep learning uh thing works you know i' would say look I'm in I've been in this industry for a long time I don't know all the details I don't read every single paper and so I think that level of abstraction I think is okay uh but I think that sort of being able to clearly sort of abstract what are the questions that only the executives can answer and sort of bring teeing those questions up and saying like look how can I ask those questions and then second what are my options right you know you know again with respect to privacy like I'm running on the cloud from one of these providers so if I send send the same data to another part of the cloud is that a good thing or a bad thing right depending on your policies it could be a good thing or a bad thing but like it's not like you're not trusting the cloud provider with your data it's just that you know how what what is the what how the what the terms of service for what the uh model is going to be used for those are the questions so so you do need to be fluent a little bit in sort of those details but it doesn't mean you have to understand every single underlying detail we we don't understand all the details about phones or the cloud for for that matter right right excellent Becky I think you said uh it's going to be a very fascinating sector to fall in the years ahead uh thank you so much for sharing your expertise today and uh please come back and talk with us again sometime awesome thank you James I I'd love to talk do that again thank thanks again

This transcript was generated automatically from the video's captions and may contain errors.