Senzing CEO Jeff Jonas on Entity Resolution in Data Analytics

foreign speaks we're talking about data analytics and we're exploring the idea of entity resolution we're going to find out what that is and how it can help you handle your data more effectively to discuss that I'm joined by a major industry expert with me is Jeff Jonas CEO founder and chief scientist at senzing Jeff good to have you with us today uh it's good to be here I've uh I'm working on a cold I'm working on getting over a cold I'm in Sacramento and I have an Iron Man in two days so if I if I start sniveling it's not because you're making me sad so you're not only doing an Iron Man you're doing an Iron Man with it with with a with an impending cold when I'm on the tail end you know yeah but I will have a cold and I I'm you know I'll race with the cold you can't race with a a flu okay that's actually dangerous but yeah cold I you know you are you are an Iron Man I sir I might have to give geared on that um I I definitely we want to talk about entity resolution which is there's there's an irony involved in that topic we'll get to that but first I see on your on your bio page on the sensing website it it calls you the big data Whisperer which I really like and no less an expert than a National Geographic refers to you as the wizard of Big Data obviously some big shoes so I you know I mean what what is Big what is big in Big Data this year I'm supposed to know by the way that does remind me somebody told me some struggles they were having with data and then they said it was they were called it big data and it was in a spreadsheet okay I just looked at him and said look if it's in his spreadsheet it's not big data oh is that true if it's if it's not if it's that's interesting why is it not big data like you know we have an Excel spreadsheet with with 5 000 cells cramped full of data that's and actually and Excel is the number one big data application I think to this day oddly enough so why is happening manipulate data but when I think of big I think of hundreds of millions of Records a billion records now some would use the word big to just mean diverse data right like lots of diversity and data uh unstructured data videos right right text and so sometimes people refer to that but when I hear the word Big Data I think about the things you can do within a diverse and sufficiently large set of data to make higher quality predictions well well maybe and I'm making to find a point here but maybe big data is a chunk of data data even just a mere 5000 cells in an Excel spreadsheet and the point being that we're going to try to glean inside from it rather than just gather it together and look at it I mean just maybe the gleaning of the Insight is is maybe that's the whole point of Big Data slash data analytics well I think you could have smolding and get some well the question is how big I guess it's in the audible holder if you're struggling with it because and you need to do a lot of manual stuff and there's 50 000 rows probably does feel like big data right it's like almost gonna take months okay fine it's big data but for me big data is like hey we have 4 000 data sources we have a billion records we're trying to get a better understanding of 100 million people so that we can better understand who's going to be traveling one and where and give them travel promotions right before their trip right okay that's bigger data yeah right big and even even financially resourceful data so all right so so that so we know what big data is what's here in 2022 what's what's making a difference in the world of of big data and data analytics what's a trend that you see going on out there well aside from any resolution what we're going to get to which is always been there and growing in people's awareness I I would say this though more generally is when your algorithms stop giving you enough accuracy you're still getting too many false positives and too many false negatives it's and you're struggling to get much more traction in in accuracy gain it's probably not the algo it's probably you're lacking a sufficient observation space ah and I think of it as fantasy analytics every now and then I'll talk to somebody okay what's your business goals they'll explain their business goals and go great good great what dating can you use to solve that and they go ABC and I go ABC that's all not even a Divine being could use ABC and make these prediction that you want and so you have to widen your observation space and get more data often that data is in-house it's data you already own it's the Yahoo puzzle pieces over there wait sorry so it's not the algorithm but actually it sounds like what you're saying is it's the human and the human has not created a certain judgment context around the algo is that what's going on well no I would say it's the humans uh it needs a little more inspiration at times to say well don't you think some data over there and that data over there would help the Algos do a better job Algos are great but when they putter out when they're redlining huh when you're when your deck of phds say that over the next three years they can maybe get one or two more percent out and you're thinking that's not even close to be making us feel more competitive I'm just saying add some more fuel and fuel is diversity in the data more more data diverse data and but but aren't we also don't we also need to ask better questions the most more relevant questions of the data whether it's bigger or not you definitely could have humans thinking of other questions although I'm a a fan of something called that as I called Data finds data relevance finds you and the reason is here's where this happened I was talking to a counterterrorism intelligence analyst shortly after 9 11. and I said what do you wish you could have any if you could have anything what do you wish you could have she said I wish I could get answers to my questions faster well I mean that's a reasonable thing I'm well sure that's yeah of course that's what she wants yeah but it felt insane to me to tell you the truth it was a two move Checkmate really because I look at her and I go hey man what if it wasn't a smart question today but if you asked the question in a few weeks that's more data's you know arrived what if it's a great question in two weeks she goes oh that could happen and I said what are the chances that you you could ask every small question every day she looked at me completely defeated and just said I can't and I remember I should have been thinking we're all going to die because she's a senator intelligence analyst you know but really it popped in my head that there's a type of system and I've been really fascinated with this in my own work my own architecture Journey assistance where the data finds the data and the things that are relevant find you it's systems that help direct human attention that's interesting it means the data is the data is the question well let me let me make sure I understand something that the data finds the relevant data finds you that sounds pretty optimistic to me actually I would I think there's probably a bunch of companies out there struggling they're not the the relevant data isn't finding them they're I know well I know I'm talking about an aspiration here though like if you could if you could wish for something do you want a bank of people that have to think of questions but because the question might not have been the perfect question until a few more days or next month they have to ask them again or what if there could be a different kind of future and really I you know the way to think about it is to me it's like a transaction is like another puzzle piece a puzzle piece shows up it's got no contact it's just a single piece it's like a pixel and then the thing you do is say hey how does this piece the thing I just observed relate to the other things that we've already already know so it's like figuring out where the puzzle fits in the on the table you know in the half assembled puzzle and after the the thing that's happening you'll go hey how does this relate to the other things that I know punch the puzzle piece in somewhere then you look at the other puzzle pieces around you go is that important should we tell somebody that's data finding data well then let me ask the same question in a different way if there if there's three parts of the system there's the actual data platform because the human is running it and then there's a data being fed into that data platform which of those three elements typically is is the weakness occurring and the the the the data going in the humans running it or the platform itself I don't really think it's the platform or the humans I think it's the the uh the inspiration about what kinds of data sets one should bring to bear like why are you ignoring the blue and the magenta puzzle pieces from over there don't you think those would help you make better decisions to find synthetic identities or to uh you know better do it do a better job uh preventing uh insider trading and so it's really about what observation space are you harnessing and what is your ability to make sense of it it's really about sense making and that and that actually drives to uh at IBM I was the chief scientist of context Computing and context is better understanding something by taking into account the things around it right right and so you want to understand your data you want to understand the pieces around it and the first form of context is entity resolution if you if you think it's three people each with a transaction but it's really one person you got the wrong context you can't make good predictions well let's all right let's let's dive into entity resolution and how it helps with data management I think the interesting thing about data entity resolution is it's also called uh duplicate deletion which has like that's the problem that's the irony isn't it yeah it itself is a system of eliminating duplicates correct me if I'm wrong on that so and in itself has many versions so it needs to be eliminated the duplicates of its of its own name to be duplicated but anyway what what is entity resolution as in your words it's recognizing that two records are about the same person even though they're described differently they don't have a join key it's like they both don't have the exact same driver's license number so I mean that's easy to I'll say that's entity resolution but it's not a join key but you know so the things about any resolutions you're trying to take two records one might say Elizabeth one might say best the month and day the date of birth might be transposed the address is super messy right and then conversely sometimes records are so close to the same but they're different I could show you two records that have tons of differences you'll be like those are the same people then I could show you two records that are identical except one's as Jr versus Sr or Junior Senior they'd be like all these are different people isn't that funny you can twiddle a whole bunch of bits and be like yeah same and you can Twitter one bit so energy resolution is figuring out same possibly the same and then if to do it well you have to build a graph and know kind of who's related to to who who's who have you decided it's not the same or a possible match because it helps future matching so it's it's building an entity resolved Network or graph if you will see resolution a real is a major problem in data analytics systems these days or is it a minor little thing or is it really mucking up the entire system it's just everywhere it's I don't know if I if you spend enough time with somebody and they they get the glasses on they look around and they were like ah I mean like just to give an example you have duplicates in your phone all bad you know the first time you met them you put it in wait a second well that's an entity resolution problem hmm you you didn't recognize the duplicate imagine that you're the only curator on your own phone imagine a bank or a Social Service Agency or an insurance company right how many duplicates they've got this is the the bane of challenges that come Downstream on Lower quality predictions that's why people send me hair care product ads you know but it's I mean it's an it's an MDM kyc AML cdps Insider threat vendor supply chain sanction screening I mean pick something basically if it's a business information system with people in it prospects in it customers are prospects people are companies vendors job applicants those are all entity resolution is needed to bring um you know meaning to that is it is it it's about expert counting to tell you the truth if you can't count you can't predict is it really is it 20 cases of Omicron reported you know 20 reported cases or one case reported 20 times like literally you can't count you can't predict the thing about it just to go on for a second is some companies have 10 20 30 50 people working on this a hundred people oh this should just be a mundane little data quality thing so that's that's what I'm trying to do in the world that's what my company is trying to do is like hey man uh would you write your own spell checker just plug in a spell check API same with entity resolution all right well let's let's dig into sensing I mean it's I want to I want to the real question is how do we how do we deal with with entity resolution how do we you know work with it and and get rid of all those duplicates that's what sensing does and talk to me about how sentencing achieves this you know what we do is uh sensing is an as a SDK it's for developers we don't build cars we build Transmissions so people Building Systems will have an entity resolution component maybe they like it maybe they don't maybe they want on the cloud maybe they want a real time and people download our SDK and in three API calls it all runs local by the way there's no privacy issues with data flowing to us it's not a SAS hosted Services locally hosted it's on your cloud or Prem and they just compile it into their Java python or C project and you pass it a Json message with names and addresses and phones and emails and whatever kind of bits you got about the identity and it'll tell you who's who who might be who and who's related just we're just trying to make it trivial you know work work on more important things well can you give an example of of how it has worked or an actual use case like how is how is sensing how's that solution really made a difference for a company what you know one of our my team's favorite examples is a non-profit that runs it it's called ericstates.org and they are modernizing voter registration in America people move and then forget to unregister right that's illegal by the way if you've ever moved and forgot to unregister oh you lawbreaker but then you get people you get election rolls they do get longer and longer and longer because people have been unregistered so this non-profit serving blue and red States uses entity resolution from I think at 36 dates it matches voter data and DMV data it's run by the election officials and it allows election officers to reach out to voters and say it seems like you moved did you move and do you want to still be registered that's an entity resolution thing because you had somebody that's shown up in a different state now it's got a different driver's license and trying to figure out it's the same person in your state do you think that the company that the companies and organizations running data analytics programs are are fully aware of problem with with duplicate entries like this or is it like they're going on we're mainly you know out of it I've been thinking about this recently I want to test it out right now there's two kinds of people in this world there were people that know they have energy resolution problems and know they if they fixed it it would fix a lot of things right and then there's everybody else so so you're saying most people are not fully cognizant of the importance of entity resolution you'll you'll find places in the organization that that are fully aware of of that um uh at the moment the way we view the world at sensing is we're not in the education business if one hasn't yet realized it we're not trying to educate them there's a lot of companies doing energy resolution we're just letting them talk about why it's important at sentencing we're we're available for those that kind of know what it is they know why they need it now they're trying to figure out how to do it real time higher scale they want to they want to be able to do it maybe for any money laundering uh or or better know your customer uh problems we're really really good at catching bad guys clever bad people that try to obscure their identity it's a it's a subtlety in in um entity resolution is that there's I'll say a legacy method where you're trying to take a record and matching it to another record you've seen But clever bad people don't use the same name managers and phone on every record Like Only the idiots would do that right right yeah but so if you want to catch clever bad people uh who are you know using their middle name and fudging up a little bit on their date of birth and you know getting a digit off you'd need to be able to need to be able to do something called entity Centric learning where you where you learn the different names you learn their nickname you learn the different addresses you learn that you've been using three days of births but the first time I built a robust version of this was for Vegas it was called Nora non-obvious relationship awareness and we were helping with uh the casinos there I mean there was somebody that had 32 different names different socials five different dates of birth okay they'd used to use different company how do you catch somebody like that entity Centric learned entity Centric learning does that well what about all those those large you know brand name analytics platforms out there we could name the the large companies and everyone would know them but are you saying that they do not have this tool set built into their their platform they probably do they're probably batch gotcha but you're not well you can search resolve data in real time but many of them uh boil the ocean and create an imprint of of Who's Who and who's related but then it's static until you reload it next time okay okay most many of them only do the uh figured out who's who but don't do the relationship so you can't see uh the graph of you know same household or groups that share the same email and um yeah and almost none of them are doing entity Centric learning which is really important for fraud uh loan fraud like there's a number of PPP loan fraud programs that we were used in and you're trying to figure out hey they got the same loan now some were some organizations got the same loan with exactly the same identifiers like everything the same and others did a little more effort to look a little bit different oh well then then why why is it you've talked about this a little bit but what why is it difficult because you think this is a problem that that analytics platforms have dealt with a lot but what what is challenging about entity resolution man there are a thousand edge cases ah okay and it gets it is fascinating you know you meet people that have already spent decades on it I'm telling you this is the the art of the dark science okay okay you you like you just in the subject of names how to compare names if when you compare names you know yeah you can have name Roots like dick Dicky Richie and Ricardo and Elizabeth and Beth okay how do names transliterate into other cultures and then and then by the way and the culture of the how to match a Spanish name with five parts is quite a bit different than how you'd match a Chinese name and it turns out if the name is Arabic then you would really should know that the word's been in Hajj aren't part of the name hmm right just names then you go on let's you know Social Security numbers okay certain numbers are valid and ranges and certain aren't some social security numbers were used on billboards in the 70s for advertising okay they're probably not too valid at this point right it's it just goes on and on and on and what how what has happened historically is as you get more data sources and more features like oh now we got Twitter handles oh right um yeah so as you get more features you would you know you you get more data sources and more features and you have more combinations of things if this and this is true and that doesn't disagree it's good and how do you set the weights and as you get bigger and more robust systems with big data you end up with like a pretty deep list of combinations of things that need to match and there's only a few people in your company that may be the experts at that maybe they've left and nobody even remembers all the config right and that's that's one of the big breakthroughs that we had as a as a new technique I came up with called principle-based entity resolution hmm and the uh which which is give give us enough well I will first give you the Reader's Digest version you have a kid a lot of kids you have a kid you catch them throwing looks at cars see tell them don't throw rocks at cars right so the next day the throne rocks are buses like I don't throw rocks at buses a couple months later they're throwing ball bearings at SUVs like hey don't throw ball bearings at SUVs how many rules do you need are they going to be in their 30s you're like don't don't throw iPads at people on Segways right right you'd need 10 000 rules but if you can teach a principal like hey don't throw things at other people's stuff yes with some exceptions like they have scissors and they're running at you and they're evil okay okay sure but if you you got to get to principles and the big breakthrough on our engine at sensing which is our sixth generation engine I sold my fifth generation engine to IBM and this project that we have is a project where I was for 50 million dollars if we start over we can do a ride we did it on this new method about principles and uh yeah so it turns out there's just a few very small number relative number of principles and they work across people companies cars boats planes vessels I guess you could use it for asteroids our main specialty is people company data but we've really generalized it so you can do all of those all at the same time without any config it's a it is the real breakthrough here here in 2022 it's it is the Breakthrough what if we you know look ahead several years a few years middle of the decade um what what is the future of entity resolution will we get to a point where like well no the systems really have that down right we're used AI you know we're we're we're all good with density resolution we figured it out is that going to happen and or what what do you see in the future well what we're on the cusp of right now at sensing is just democratizing it used to be only the most elite organizations in the world like literally if you don't have a million dollars to spend a more you can't have good stuff you just can't afford it I'm trying to make sure that the really the the smallest initiatives non-profits that have for them their big data is you know 12 000 records that have duplicates in there non-profit mailing list I'm trying to make sure everybody in the world can have world-class entity resolution and it's going to take a little while to get that available to everybody I mean it's available now but to get the word out make sure we're discoverable but we're trying to just fix this so people can move on and work on other more interesting things I like that it's really great that you're making it accessible to those small organizations that's a great service yeah it's it's really exciting you know uh an example of a place that was used in this last year is in the state of Utah the Clean Slate law is you know if if you had certain kinds of felonies like for example around marijuana you're gonna have your felony expunged and that means you get access again to voting and certain kinds of jobs and it's a matching problem there's entity resolution in there and our sentencing product was used for that um the folks at code for America did that with our product and you know a quarter of a million people had their records accurately expunged it's just it's it's just accessible to everybody now yeah I wonder if there's somewhere out there there's a there's a data privacy expert about any individual thinks well you know big brother is getting better and better at tracking our records we like the sound of entity resolution but we don't like the fact that they're they're getting better at tracking us is there an issue there or not I I I hear you're voting example people actually can make a vote again so it's actually positive but yeah what are your thoughts about data analytics and similarities uh it's it's worse than that I worry about uh somebody in uh uh in a far away place outside the U.S using entity resolution to Target a minority ethnic group right yeah right I mean like you just have to think you have to think about that uh on one level it's like I gotta we have a pencil over here and now most people use pencils for good things but every now and then somebody writes a hit list or stabs somebody in the neck and you know with the number two pencil so you know we made a caucus a a cognitive decision to not host anybody's data I also didn't want to ever have a breach so it can't get repurposed and used for ways people didn't anticipate right the consequence on that is that organizations can use it and there's no phone home we don't know how it's used for the most part I will tell you though it helps organizations make sense of the data they own they got the blue puzzle pieces in this hand the red puzzle pieces in that hand and they're missing the obvious now uh you know what marketing sure but you know fraud synthetic identities the amount of waste that's happening in healthcare fraud is just stunning is it okay it's really big yeah I've heard some estimates I can't remember they're all so big I can't even fathom them right large numbers not a privacy side I do a lot of work in the Privacy community and we've been uh we actually announced sentencing for the first time in international privacy day at the Privacy by Design conference because we've been making so many privacy features now I'll tell you what one of them is it's a it's data tethering it's if you happen to have a watch list like in Vegas there's a list of people who are banned from gaming right okay they're on the exclusionary list by The Regulators but if there's a record on there that got there inappropriately or it had an error and now it's fouling an innocent person data tethering is when it when it when a data source is contributing to entity resolution if you correct a record or delete it you should be able to find exactly that record in the system and take it out so it helps people I mean in terms of the civil liberty issue it actually it can help people false positives is one of the worst things you can do to people okay so if you're if you're on the bad list you should not be on the bad list until you really unless you really deserve to be in the Bible so yeah it's a liberating aspect yeah yeah and we also done a lot of work on favoring false negatives like you just don't want to be tapping on somebody's shoulder and by the way whether it's you're not letting them get access to immigration because you're doing a bad match okay or you're not there's somebody there's a there's a special promotional offer for a lowest interest rate and they just think you shouldn't have a lower interest rate so somebody's not even getting that maybe that's the case but bad matching is a is a problem so that's another privacy by Design aspect in sensing is to favor the false negative yeah Jeff I think you said it it's fascinating I think it will be a major force in data analytics in the years to come um thanks so much for sharing your Insight today thanks I've enjoyed I've dedicated my life to this so it's really all I got it it Matt it matters too I think it matters to the larger world it's it's needed and uh we're trying to make sure everybody can have it so and that makes it fun that the team and I most of the people on the team have been with me for 20 plus years you know wow we actually love doing this we haven't had a core engineer leave the company and since we've launched in five six years like we are passionate about this is all we do that's great all right well let's talk again soon it's been great talking

This transcript was generated automatically from the video's captions and may contain errors.