DataStax CPO Ed Anuff on Generative AI and the Enterprise

foreign speaks we're looking at the future of artificial intelligence infrastructure along the way we're discussing large language models the cloud private data centers and data analytics a lot of Big Ideas all related to AI to discuss that I'm joined by Ed enough Chief product officer at datastacks and happy Friday too and thanks for joining us today uh thank you James it's been uh it's great to be here with you today it's really interesting to me that uh obviously we've seen this enormous upswirl in artificial intelligence ever since chat GPT debuted at the end of 2022 and it's AI is not really new but it seems like there's a new awareness of it going on um so as you survey how AI is affecting your business what are a couple of key trends you see here in this year well so a couple of different things so first obviously we're all very um uh amazed and and everybody has as this heightened awareness of what's possible with generative Ai and the large language models and those are super exciting and we've we're seeing um you know just a huge amount of adoption uh of them particularly over the last six months um but one of the things I do like to to remind people is is that AI adoption has really been happening over the last two years and a lot of that has been around uh real-time predictive Ai and in fact um when I say over the last two years that's been among sort of mainstream businesses and Fortune 500 companies obviously companies like uber have been uh building their services on top of a real-time AI predictive AI is is tends to be the way that it's described um over the last you know five ten years actually right and and that's that's gone through you know a number of different terms we're really sort of predicting that next action or telling me when your car is going to arrive and all of that what about all right was there another one in there I don't want to interpret it was there another Trend that you saw um well I think the second thing of course is is now really you know how do we uh how do we Embrace um these within the application tier so a lot of a lot of what we've been seeing with AI use cases have been things that have been uh being driven by data scientists and data Engineers um and now a lot of the the the tooling the the open source the commercial products um are actually increasingly targeting the application developer and you see that that the application developers in general are becoming very fluent in these Technologies so combination of them becoming more knowledgeable as well as as the bar being lowered in terms of of the complexity of adopting these so now it's almost sort of any developer you're talk you talk to um is able to be productive and leverage this in some way like calling a GPT API AI is something that that almost any developer can can do and can be productive in doing so that means that applications are getting AI um in all sorts of ways and so that's really exciting it means that that we're just seeing this all over the place you know it's I think it's a really interesting point when you think about large language models models which of course chat GTP is an example they change how developers work in a couple ways one of course is the idea of low code no code really the democratization of tech and so you you don't necessarily need to be a highly experienced Highly Educated developer you can get in there and tweak the application you know you might be a regional sales manager you might be a you know a mid-level executive and suddenly you know low code no code is is democratizing technology so even the idea I mean developers are working differently but even the definition of who is the developers is changing yes absolutely I mean I think you're seeing um uh a couple of things in relation to that so first um AI is making an llms in particular under the hood um are making uh no code style or low code or no code style automation much easier because again it makes it possible to build tools that are more sort of do what I mean type of of an experience and do what I mean is is a uh you know expression that that has been around for many years from a product design standpoint but now it's actually the reality and so so it definitely is um is making that possible that said we're also seeing that just developers who are generating or writing code also have a lot of tools at their disposal that make it possible to do a tremendous amount more work and to just increase levels of productivity and whether they're just going into chat gbt and asking it to to write the project for them building a web application doing things like that you get very high quality uh results some people will point out and say oh sometimes the code needs to be edited by hand but um you know it still is a massive uptake in productivity and honestly you ask any software developer and their first version of what they write doesn't work work out of the box either so so the fact that they've got to go and hand edit some of what they get from GPT is is not a problem um and then you've got tools like copilot um which plugins exactly the GitHub tool which which also make you know you start writing what you want to do and it starts Auto completing based on what it seems studying all of this code so the the productivity for whether it's at the the no code low code level which as you said just making it possible for a power user right somebody who may not be you know may not be a coder but they know how to express what they want the process to be um or for for the developer who does need to build a bunch of stuff now maybe I don't need a huge team maybe I just need a developer who has kind of a vision of what their application needs to be and they can get large swaths of the code built for them and they and makes them immediately more productive so it allows every developer to be a 10x developer or even 100x developer and so that's pretty exciting yeah I mean I think the the trend toward the Democracy the democratization of tech has certainly been a mega Trend that is going on for a while but I you know the large language models do accelerate that um let's look at it from a different angle what what about those companies that they need to retain their data for any number of reasons Regulatory Compliance they need to retain their data in their private Data Center and we still have them not everyone is on the cloud 100 it's even a reality here in 2023. how can those companies best Implement artificial intelligence so that's a um that's that's an interesting um Challenge and so first of all I will say that there is no question that AI um is going to be sort of the second shoe to drop maybe depending on the time Horizon you look at maybe the third shoe to drop in terms of of uh I don't know if that's an analogy um uh but uh but but I'll coin it here anyway uh the the the um you know we we saw a um you know with the pandemic we saw sort of that accelerated transition to Cloud right and um and AI is going to be that next wave accelerating transition to Cloud and the reason for that is um what makes all of these AI um use cases possible are the size of the models the size of the ml models that are being used to power them and that those long large models I'm not specifically saying large language models just large models in general right um they require gpus yes Graphics processing units those are what make it possible to accelerate the Matrix math that makes doing these things possible right and by and large most Enterprises don't have a um a a whole bunch of these gpus they're created by Nvidia there's other things Google has something called a tensor Processing Unit Microsoft and AWS have have their own analogs to that but by and large what they do is they use racks and racks of these gpus and that's what makes this possible if you look at for example opening open AI has spent hundreds of millions of dollars on this Hardware right um and so so generally if you want this stuff it's in the cloud yes now um that said a lot of the early AI was built um really from the standpoint of let's get this stuff out the door very quickly and so what you're now seeing is a lot of optimizations that are happening and you're actually seeing that it's possible to take these models and actually um run them on on your iPhone um or for that matter within within the data center and there's some companies there's some startups um companies like um uh there's a company called third Ai and a few other startups that are that are um uh also going and figuring out how to run these models more efficiently on the hardware that's already within the data center so my my prediction is it will be possible to do these large models and large language models on in the data center but right now um if you really want to do this stuff and you want to deliver applications and experiences that are powered by this stuff and the cloud is the place to do it so so it really depends on your time Horizon I think it's going to light a fire it has lighted a fire under moving more data to the the cloud but there's a lot of reasons as you pointed out why some of that data can't get to the Cloud very quickly and so so you will see some options um for for people who have medical data Pro privacy intensive data that can't go there but it's going to slow things down and I think that that a lot of companies are really you know making the the the prediction and the call where they go and say look this is this is the final this is really sort of the the the the the you know the final straw we need to figure out how we get that data to the cloud and so so so there's going to be a little bit of push pull around this and it'll be the biggest sorry this is this if you when I talk to cios and when I talk to I.T leaders this is every one of them is grappling with this the issue of where whether or not their own facilities have the processing power to really run Ai and then how to you know sort of speak import that ad into their own data okay that's a that's a huge issue all right it's a huge issue data data has gravity and generally you want to bring your AI to the data and if that date and if the data is not in a place where the AI can run you've got a problem right right I want to clarify one point I just said something fascinating kind of in passing you're talking about running a running these models on your phone you meant I'm assuming I'm sort of a SAS uh SAS scenario the the phone was logging onto a system and the user was running it you know to see the user interface and the phone did you mean literally running it on your phone I mean literally running it so this is this is where um uh you know it's fascinating how fast things are moving but a few weeks ago um uh um actually earlier than that but but a few weeks ago you you actually had a couple of Open Source projects um actually released the means for compiling a model so that it can run on the neural processor that's on the iPhone wow and they took one of the models one of the large language models one that's called uh uh vicuna 13B so it's a 13 billion parameter model and they actually released an iPhone app I have it on my app it's not as full featured as GPT but it actually lets you do um uh you know GPT GP chat GPT style interactions running entirely locally on your iPhone and um and so so you're gonna see more of that first of all what is the name of the app you don't mind saying um so this particular one is um uh it's part of a project called mlc which is an ml compiler and so they released an app called mlc chat as a proof of concept that allows you to use the the vicuna 13B uh model which is a large language model and you can just run it on your iPhone now now your iPhone does get very hot while it's running it's using a lot of compute but it actually runs and it runs very quickly uh on on the phone and so um uh you know I throw that out there so as an example of just how fast this space is moving you're going to have these models running in a lot of places and it's part of what I call the optimization phase of llm a lot of what we've seen thus far has been more about the art of the possible of going and saying hey if we have we go and throw 100 million dollars worth of gpus at something can we make this AI experience nobody dreamed possible can we actually make it possible but now you've got a whole wave wave of Engineers and researchers that are now going and saying how can we reduce down the size of these models how can we create means by these models can run on on the hardware that we have and so so you're going to see a whole flurry of activity by these optimizers uh what I call these these engineers and developers and researchers who are who are tackling the optimization problem and so you're going to be surprised you are going to see local execution of this stuff you know that I find absolutely fascinating I would almost uh resort to the exaggeration here for a moment and say like running AI on an iPhone feels like a moment in human history I mean who would have thought this little piece of metal in our pocket could actually run artificial intelligence that's pretty wild um yeah and I suspect I I suspect again I can't predict but I suspect that it's something that Apple anticipated which has been why they've been packing these phones with so much AI specific capabilities um we'll we'll see I'm I'm looking forward to like like everybody I'm looking forward to Apples uh upcoming worldwide developer conference and and and seeing if they do more to take advantage of it they certainly have been sponsoring a lot of this this sort of thing so so there might be some interesting things along that line again pure speculation on my part sure you know the the other idea about running um Artificial Intelligence on one's you know small phone uh or a powerful phone actually is it's it again ties into that trend of the democratization of technology so it's not a Mainframe somewhere you know staffed by you know highly paid data scientists it's some sort of data scientist it's a citizen citizen developer uh assistant taking action and he or she is equipped with artificial intelligence on their own you know phone which is pretty amazing um yes yeah well let's all right well then we're I was going to ask about the future we're already in the future but before I do that I think we should take a moment and talk about data Stacks in particular how is how is data Stacks playing in this sector what does all this mean to data sex in your view well so it's really exciting for us um so a couple of things so first of all again um we we firmly believe that that AI needs data and and needs very large data sets and if you look at where data Stacks plays with our Cassandra technology this is where people go and store their their large data sets this is why Enterprise companies and uh and Technology startups alike uh select Cassandra and select data Stacks uh again and again and you look at some of the largest companies in the world um that are leaders in AI you look at companies like uber and you look at companies like Netflix they standardize on Cassandra as their database for storing these large data sets a lot of that data is real-time event data and we talked a little bit about real-time AI at the beginning so um as companies use these large sets of events things I click on things I put in my shopping cart um perhaps Logistics data and they want to make predictions as to what's going to happen next they use these data sets and so we've been playing within this for a while now um both us specifically you know as as a company and within our enterprise software and our cloud services but also just the Cassandra ecosystem that we build on top of has been doing this quite a bit and so this is pretty exciting for us um that said as we've seen generative AI happening that also requires this set of of data and it's these large language models they use databases like Cassandra to build a history and a memory right because when you interact with a large language model you wanted to know all the previous interactions right and so these these Services um chat type applications as well as as all of these generative AI use cases um they build on top of Cassandra to build that long-term memory and so we are doing a lot of stuff um uh you know we're doing it within open source and we um uh you know and and it and with as I said within our products to make it possible if you're using Lang chain if you're building uh chat GPT plugins out of the box you'll be able to use our cloud services and our open source to empower these so so um so it's super exciting for us all these use cases are data intensive and there's a lot of the properties of nosql databases in particular as well as the type of database that Cassandra is that lend themselves very very well for these use cases so we're all in on it interesting okay really interesting um I want to talk about the future also but I I want to stop for a moment and then try to formula this question about who is what sector what kind of companies are going to control artificial intelligence the most the question can't be answered but I'll try to ask it anyway so that we see the large hyperscalers they have such sort of gravitational pull because as you say they've got all those racks of you know processing power and they've got the gpus lined up you know row up to row and so in a sense one might say the hyperscalers will own the future of artificial intelligence because they have the greatest platforms to really run things on the other hand there's a whole host of Standalone vendors data and AI vendors that are doing Innovative things and you know so they're doing some pretty interesting things so which of those two camps is going to control the future of artificial intelligence more in your in your view if that question could even be asked oh it's it's a wonderful question to ask it's a question that a lot of people talk about um a lot and it's interesting almost any conversation that I have with other technologists um technology leaders um this question comes up again and again and um the answer is um it's very hard to predict and and the reason why is certainly the hyperscalers have a major role to play um but you know about a week ago there was a there was a whole thing where there was a leaked memo out of Google where one of their researchers raised the question of of do the hyperscalers actually have a competitive mode or is the tension going to shift to open source there's that's a legitimate question there's a lot of things happening within open source models I'm sorry I understand the question someone has to do the hyperskillers have a competitive mode I mean they are except for a moment thank you all right do this because the nature of Open Source would have would erode the the hyperscalers moats perhaps yes because what's happening is that you're finding that people can take open source models and they can actually enhance them with their own data and build more powerful models that might be better for specific use cases than than the the large models that the hyperscalers are providing so this is already happening quite a bit in the developer communities okay that's pretty that's pretty interesting it is it is and then you also have the this you pointed out the startups that are also doing that where they're creating um they're building either their own models or they're actually adapting um you know the the models that they get from the hyperscalers and they're creating better better uh Services tailored for specific use cases so my my my personal answer to this question is um it is there's not going to be one size fits all it is going to be about going and saying okay you know do I want to build a specific model for um retail engagement or do I want to build a specific model for Logistics or do I want to build a specific model for legal documents or do I you know those will be things that startups will Tackle open source will tackle some some variants of that um and I think that the the hyperscalers will will continue to do two things one is they will provide the place where all these models run because they do have the hardware and the compute um I I personally when I worked at Google I actually toured one of the data centers and it's the size of a small town right right it literally is we we spent you know uh you know something like two hours walking from one side of it to to the other um and so so so that's that's obviously a physical advantage that's very important the second thing is they will they will be providers of what are called the foundational models because they need the the large the entire Data Center Corpus of the web to train these and build them and only a few companies can afford to do that right but you'll have open source models you'll have domain specific models you'll have companies doing purpose-built training or tuning on top of these foundational models and so and then the other thing is as we talked about you'll also have versions of these models that are designed to run on your iPhone and in other special Hardware right so it's changing every day it Eve you know on any given day I'm trying to keep up with this stuff between you know between the morning and by the afternoon I see five major Union announcements or models released or startups announced right it's it's a I've been you know for for folks I'm sure you're seeing this I I've been I've been you know working in Tech since since uh since the early 90s here here in Silicon Valley and the I haven't seen anything like this since 1994 1995 it it is sort of the internet hitting all over again I I agree like since November Everything Has Changed it's pretty amazing yes I mean the the world you're you're you're portraying I mean I think about past decades going back to the 90s for example there were there were incumbent technology vendors you know large Enterprise vendors in a certain sector and they would see a promising startup and they would let that startup get a certain amount of heft and then they would just simply buy that startup so they could sort of the incumbents would sort of digest the interesting startups but it feels like in the world you're portraying there's too many other things going on if if the hyperskillers are now the incumbents there's almost too much going on for them to Simply reach out and digest the interesting startups because there's too much going on I I think there'll be some I think there will be some digestion I mean remember um uh you know if we go back to again uh these historical analogies are imprecise but you know um uh uh uh you know Netscape did end up at AOL right and uh and and so so so so so you will see you will see both um Acquisitions and consolidation but you'll also see new players that emerge that um uh I'm sure for the next five years we will see new startups some of them already I mean obviously we have some examples of open Ai and others um who who have emerged you know so I mean obviously they were working behind the scenes for a while but to the general public sort of emerged out of nowhere and we'll see a whole slew of those so it's going to be a really interesting time um for the you know it will be a challenging time for for you know the enter and an exciting time for the businesses and Enterprises is that want to adopt this technology it it'll be just like back in the 90s where suddenly every company went from what's the internet and what's the web to I need a website I need a right you know I'm a retailer I suddenly need to be selling online well the same thing like there are companies that you know you're gonna have a company today that is going to some point in the next six months are gonna be like we have to build a chat GPT plugin and they're gonna be like what's chat GPT I mean right you know I mean most people at that point are going to say that but but you know it's it's we're at that State where where new things are going to be entering into the mainstream and the people the leaders who have been representing the the digital transformation or the digital engagement or the digital presence are suddenly have an entirely new front that they have to play Within um and and you know the last two waves for them might have been mobile and then before that was you know the web I mean these are it's it's going to be on that level of scale totally and I think I think you said it I mean this weekend could you continue on for another three hours I think but you know in the interesting time I I guess I better cut us off but wow really really it's exciting it's it's exciting yeah yeah it's just like a bit of a gold rush and then who knows what's going to happen with it and um it's yeah definitely well I guess I will allow myself one more short query is is there I said I mean I find the whole thing very exciting but is there not like a little bit of a hint of worries slash doom and the whole thing too or is it might be in a naysayer here I think that there's going to be disruption without a doubt I I don't know the nature of that disruption there are things we certainly um you know there the the there is the obvious thing which is the ability of these large language models to produce very high quality text right and if you think about it most of us are knowledge workers of some sort right and yes and um and and you know we produce content whether whether we're talking about articles or or or product requirements or customer uh you know facing marketing content or whatever all of us myself included we your knowledge words and and um and we're we probably you know are all using chat GPT to some degree to to help uh refine and formulate our ideas and and um and so there'll be some impact on that and certainly um uh GPT gpt4 is capable of producing very high quality code and so there is a slippery slope between right now a lot of us are using it as an amplification there's a slippery slope between amplification and elimination right like yeah maybe I just don't mean people at all right um it's too soon to tell it's too soon to tell it'll it'll be a little bit about is the long and short of it it will it will people who are doing more sort of wrote work um you know will will be somewhat challenged people who use this stuff again I'm old enough to remember when you know being able to use a search engine was was both disruptive and and but also a force multiplier right right you know and so so there'll be a lot of that um uh you know we'll all have to keep a close eye on it I'm I'm it is a legitimate question I think about it a lot um as does everybody um um yeah other than you know I I I've lived through a couple of phases of disruption um uh and uh and and maybe this is the final one maybe we're all put out of work um but uh uh but yeah yeah right and I think you said it that is an open question and all this it is in a sense it's unfolding it as we speak it is unfolded very rapidly uh at any rate um thank you so much for your Insight it was fascinating I hope you come back and talk with us again sometime I love to do it it's great conversation thank you

This transcript was generated automatically from the video's captions and may contain errors.