Prove AI’s Greg Whalen on Challenges in AI Governance

hi I'm dreames Maguire and on today's e speaks we're talking about artificial intelligence governance which is one of the most challenging areas in the tech sector the job of AI governance is to manage a technology that's changing faster than regulations can be written so how do we do it to discuss that I'm joined by Greg whan CTO of prove AI Greg good to have you with us today good to be here thank you so there's no doubt I mean ai go is a tough topic before we dive into that can you give us a nutshell version of what Pro AI does what is the company's Core Business yeah absolutely so that proof AI our our core sort of mission is to drisk AI development right so um there's lots of companies in that space right I don't think our the value that we're or the what we're trying to solve here is necessarily unique I think this is a Hot Topic um everywhere is how we how do we drisk AI developments right what we do differently though is is to focus on this as um as to something where we want to drisk things without putting in processes and without putting in unnatural things that slow development down right so so our our focus is to look at the history of software development and and just note that right when we've talked about hey things are moving too fast things are risky which which has repeated itself many times right in the world of cloud software development open- Source software data science and analytics right people keep saying wait things are going too fast let's slow everything down and historically has never been the right it's never been the right Solution that's never what's actually happened so right what what our mission and what we think that our our Unique Kind of approach to um to drisking AI development right is is basically to to focus on something that's very developer focused and something where right over time we want to limit the amount of people involved in the in the process right and focus on metrics focus on getting the right um listeners in place with the models that we're developing and ultimately get to a get to a place where we're all comfortable about what our what our AI development and what our models are producing right and and not have to worry about calling unnecessary meetings and being all worked up about whether whatever they're doing is compliant ethical or or simply the right thing right so I mean that's that's what our mission is we do use a decentralized ledger under the hood right to unlock certain use cases that are very hard to do with traditional databases such as third party verifiability and things like that we think those are core parts of the U of what we need to solve right if you have a full and complete vision of of solving this sort of drisking AI sure okay makes perfect sense uh certainly companies do want to be moving as fast as possible at the same time balancing the need for regulation and governance so we can get into that um but I I think that the interesting thing about governance the compelling urgent question about governance is that obviously AI governance is a major challenge for companies for any number of reasons I think privacy is an issue security I think sometimes staff members will input sensitive information into a large language model that can be you know a security issue but what what do you see are some of the most difficult current challenges involving AI governance yeah yeah absolutely so um I I sort of have a high level view of what the actual challenge is like the root cause so so like the one you gave is well how do we know that like training data how do we know what its Providence is right how do we know if it's really data that we want to use or should use um yeah that's that's a a horrendously difficult problem right and then we if we go back in root cause well why are we even in that situation to begin with right why are we even worried about that problem right I I think of a lot the biggest challenge is is the sort of the the natural inclination to try to fit whatever processes people use today with software development to AI right so right that the problem that you stated about data governance and data lineage lineage that exists purely because that has worked perfectly well for traditional software developments like and through through today right it was it was that was probably the least amount of effort necessary to get a good result right and it's it's because you didn't update data sets too often and if you did update data sets then you know okay you you probably it was probably less effort to have you know a human review of something and perhaps a declaration right there wasn't really a need to to actually you know operationalize this in a way that uh you know that could be that could remove people from the process so right sort of bumping this up a level right that's actually the biggest challenge that I think people that people have is they're set in a way of thinking of llms are software which they are they are software sure um however there's an attempt to say well if they're software shouldn't they conform to the ways that we've been governing software for the last 5 years 10 years right and the the answer is no but but they go they go through this process and they they naturally come up with well we need a human review here we need a workflow here we need a sign off here and they create very complicated chains of stuff right that that ultimately are not they're not ever going to support the speed of development that you're going to get Beyond those one or two initial models right that that you want to roll out right so you we Focus instead like I was saying earlier about Pro AI right we focus on events collection the idea is is if you have granular granular enough data like in terms of events right you collect events um you you know that they're trusted and storing them in a in the right place is as good if not better than a than a declaration or a sign off well let me if I interrupt just one I guess it seems there's this very powerful very quickly changing thing called artificial intelligence you're saying it's can be governed in a fair Fairly limited way without many bees without many cross cross Communications among colleagues it feels like it's a some might say without knowing all the details that's a pretty hands-off approach to governance am I correct in here thinking that yeah y yep and and that sounds scary right it absolutely right it does yeah for sure right right um however right I like to just draw the parallels with Cloud software deployment and like when we go back to the the Web 2.0 error right trying to do a single deployment right um scaring like I remember you know operating in this world where we would say you know what we need to do we need to do two deployments a week and everybody would lose their mind and say no that's way too dangerous you how are you gonna do that we'd have to have spin up a whole compliance team just to look at everything you've you've done like and then be be comfortable with the sign off we'd have to do user acceptance testing this is impossible um you know there's so again the solution that we've gotten to though is to get to a point where yeah that the amount of information and the amount of automated testing the am the amount of certainty and confidence we've got you can get to with high observability systems has given us the confidence to do thousands if not tens of thousands of releases of software you know each week so you know we're not we're not saying remove humans from the process entirely right that's um that would be dangerous right I guess the the the philosophy is here is to let's not make complicated there's no reason for people to spend their days looking at outputs of stuff when you could easily just you know program in checks and balances right for these things to automate this and get us to a point where we're very specific about what's okay what's not okay very specific about what our what our criteria are in our threshold for acceptable or not acceptable and being able to you know handle the exceptional events and shut things down and roll back to previous non-states which which is exactly what we do with traditional software and pipelines today well all right on that on that note of automation I think it brings us naturally to the the point that you know given that AI supports automation should we expect a shift toward automated governance perhaps we will exist in a world where where humans barely need to touch it and in fact the the governance process such as it is is actually totally automated is that does that world will that world ever exist yeah so that that's I I I believe so I believe it has to because otherwise you know we're going to spend way too much time um and effort with people in roles that simply you know are are designed to sort of approve things that that probably the probably the the amount of information they would have to look at right would wouldn't actually be possible for for a person to reasonably um you know approve of right so so things like take like the data set um piece before that you you brought up right so does this you know three pte data set contain any information that we should use right you could imagine a workflow where we say man we should just have somebody who who decides whether this information is acceptable or not all right and can sign off on its use um yeah that so we don't think that that's ever going to work as data sets continue to get bigger and they become harder to sift through with more and more inputs or or columns if that's what you want to call so right this is an example where you know at some point it's going to be impossible for an individual to to make a clear yes or no this is okay this is not okay decision by looking at a single artifact and that's what much of compliance you know is is about today it's about looking at artifacts and deciding whether you know the artifacts examined exhibit sufficient controls right that that follow either standards whether they're an internal standard or an external standard so he say well well logically if we're going to have you know if we're going to have immense data sets that um are are very hard to trace we're going to have neter ministic llms right not completely deterministic llms or at least their determinism is very hard to you know to fully test right then sorry to interrupt could you explain what do you mean by deterministic or non-deterministic large language model can you explain what that means sure sure it just means how certain are we that do we know every possible output right from these models right so traditional software right you can build in a relatively finite number of test cases where you're you're sure that you have full cover coverage or mostly full coverage of every possible output right and you know that because the software you've written is is very clearly deterministic right it's very it you there's a very clear analysis that a human can do to say that I am confident that we have full testing of this software code right because it's it's clear what it does from top to bottom you know when you have billions of uh you know billions of neurons right um in in an llm right the only thing that you've been able to observe with your llm is the input data and things like the number of parameters The Prompt right and the overall sort of you know how the llm itself works you know the problem is is that there's no way that you can have complete certainty that if you give you know question a that it's always going to give you you know acceptable answer B right right there's there's too many issues where you know you'd have to build so many test cases out to be completely sure of how this model is going to function that there's always going to be some non-determinism here where non-determinism simply means is I can't predict the output meaning you know test case a does not give me confidence that I have 90% you know test coverage that's that's what I mean there well let's let's delve into the world of agentic AI it feels like agentic AI is is the Hot Topic within artificial intelligence these days so what about agentic AI in governance are these virtual workers supporting governance or do they create their own governance issues yeah absolutely so I think they make it harder I think they make their own governance issues and I think they make governance more difficult um so I mean especially especially as we Empower and Link these these agentic AIS right so if if we step back and just say let's take very basic examples of of agentic AI like let's take um something like you know I've built a customer support application and and maybe you know a very Advanced one or you know might rely on SE agents to do things like resetting a password providing a refund doing sentiment analysis right and these are all semi-autonomous and that they right they can take action they're empowered to take actions on behalf of you know of a customer right right but in a in a in a even a relatively small number of them let's say five of them you get into all sorts of unexpected challenges like well how do we know that this agent is really talking to this other agent that I've actually author ized them to talk to right how do we know that there's a clear chain of trust between agent a and Agent B what happens if there's a mistake what happens if there's a you know some sort of attack and right different agents start talking to you know or masquerading as as um you know as other agents out right how do I how do I the rogue agent right and these get this this starts to become a real problem if we imagine that agents aren't necessarily all provided by the same person right so like in the customer for example maybe there's a specialized agent I Outsource it to a third party who's very good at sentiment analysis right and that's part of my solution but you know I've then told my agent to trust this agent over here right how do how how do these how can these agents then autonomously determine and be sure that yes I am talking to the agent I'm supposed to be I'm supposed to be using outputs from this agent right and then and then what happens when something goes wrong you know how do I trace back and figure out who actually made the who actually made the error right was it agent a was it Agent B am I going to rely on multiple kind of people looking through logs or am I going to want this written like to a decentralized ledger in a central place a single you know factual source of truth that multiple parties can go to and say aha the problem was is sentiment detection went off and it triggered a whole bunch of events that mishandled this customer issue right or was it you know nope there was a there was a refund sort of processor over here that you know that used erroneous logic right to to basically trigger some action but it wasn't prompted right incorrectly you get all these all these things that start to be real problematic um you know when when introducing multiple agents that each are supposed to be the best at their job right and have more autonomy than they did previously right right well I think that to wrap up our conversation I think one of the the biggest issues the most pressing issues is all the companies that are not developing AIS AI apps and they're working with large language models I mean that's a that's a lot of new development you know companies going in New Directions how does that impact AI governance yeah y so yeah like like the last example it it it just makes it harder right more more parties are involved um and right we can expect then that there's going to be more desire like for for cross collaboration and the selling of Aid driven solutions to third parties right so this has always been you know always a challenge right but typically still in today's world a lot of AI powered Solutions are still Solutions right I don't buy in many cases I don't buy a model from a third party right I buy a solution from a third party that's right that sits on top of that model that sits on top of it it it has slas it has um you know known behaviors um everything is everything about it is you know there are guarantees given to me right um when you have when you have this sudden surge of interest in company either developing or working with you know llms and and models um you know what you can expect them is there's just just going to be multiple parties um in the picture so you know what happens then is well how do I know that the model that I've bought from party a doesn't include data that I that I personally don't want my organization to use you know how do I know that when they change something that I can be alerted of it and how do I know since the blast radius is particularly big right a small small change to their model could it could could in theory create a huge you know trigger a whole bunch of bad bad answers on my end right that that open me up to liability right you how do I you know how do I solve those types of um problems right because ultimately if it's my solution I'm on the hook it doesn't right under under today's legislation right it doesn't really matter where I bought the the llm from right it's still my solution and thus I'm I'm accountable and liable for for anything that it does right so ex you can imagine examples in industrial use cases you know maintenance types of things automated sort of maintenance manuals where I buy an industrial product from part they have a sort of Maintenance chatbot that they that they provide to me I have thousands of workers that rely on that documentation to do preventative maintenance right I mean what happens then if I start buying a model from these sort of third party third parties to to do my own predictive maintenance right and suddenly either bad information gets in or or um or it becomes configured the wrong way like you know you know what happens if one of my people actually get hurt or they do the wrong type of thing to you know to an industrial machine because of a a decision that was built on a on a Model that I purchased from a third party right very tricky things that get involved but this is exactly the type of stuff that will come out as more companies develop and sell you know AI Solutions um it just gets more more hard MH Greg I think you said it a lot of good stuff it's certainly going to be a fascinating area to se to fall I mean there so many issu involved with AI governance thank you for sharing your expertise today and I hope you come back and talk with us again sometime absolutely likewise thank you for your time appreciate it

This transcript was generated automatically from the video's captions and may contain errors.