Expert Panel on Data Mesh: Capital One Software and Pitney Bowes

foreign speaks we're talking about the concept of data mesh we're looking at why more companies are adopting data mesh and we're also discussing how to operationalize data mesh to provide Insight I'm joined by two industry experts with me is anamate senior director leading the customer Solutions architecture space for Capital One software Anna thanks for joining us today hello James thank you for having me Anna please tell us about what you're focused on a Capital One software what sort of projects do you normally lead oh happy to um first of all thank you for um inviting me to participate in this conversation um alongside Vishal here from Pitney Bowes as we share our perspectives and and experiences in this space um I've been with Capital One software for about two years now um and capital on software for those of you who don't know it is a brand new Enterprise B2B line of business of Capital One which is focused on providing data management solutions for companies operating in the cloud so as you mentioned I I do leave the customer Solutions architecture space for uh Capital One software and specifically uh my team is focusing now on our first ever officially launched product by Capital One software called slingshot which is focused on enabling businesses to maximize their snowflake investment uh at a very high level so as part of that role I do focus on working directly with our Prospect and existing customers to how help them identify and understand which challenging aspects of the data management ecosystem can be addressed by some of our solutions that we develop at get one software sounds good and as you mentioned we are joined by Vishal Shah data architect manager at Pitney Bowes Michelle good to have you on board today thank you very much James and thanks Anna for the introduction also so uh as as James mentioned I'm Vishal Shah from Pitney Bowes um I've been with Pitney Bowes for about 23 years now uh and going um I am actually in charge of various data related Solutions whether it's um a data analytics Solutions or data engineering Solutions um I've been working with data all my life from the time I started working in the computer and say I've always been working with data so um you know the whole data journey of Pitney Bowes I was a very active part of that um and then a few years from before like a couple of years back we actually came to our data mesh Journey where we moved our architecture to our data related architecture and um we have got some significant benefit from that architecture and um you know I I play one of a core role in making sure that that architecture is completely managed govern uh within the pennibles room so thank you very much for having me sure all right so let's let's dig into the topic of data mesh I've certainly seen data mesh get a lot more attention recently the question is why why are companies moving toward data mesh architecture what's the advantage of a traditional data infrastructure Anna your thoughts sir that's a great question James and very timely for sure and I think there's a lot to impact here so let's dive into um this space a little bit uh I would like to first start by just setting the stage with what data mesh is right at its core and just in the most simple terms it is an architectural concept but also a paradigm shift that distributes the handling of data in an organization to the individual lines of business or domains or individual teams so in other words it is a new approach to managing and distributing data within large organizations so in in that sense data mesh the parts from the traditional approach of centralizing their responsibilities under one large data team and instead allows companies to access and analyze data at scale so the core idea here being that decentralizing data ownership and management across the organization um by by treating that data as a product so data mesh in my opinion has emerged as an important framework to help companies scale in a well-managed cloud data ecosystem in a complex data environment in which volumes and and sources of data are growing exponentially by the day there are four main principles that support the data mesh architecture that I would like to just run through real quick at a super high level first one being data as a product so think think of this as data teams applying product thinking to their data sets so in other words an organization can assign a product owner to a data and apply the same rigorous product principles to data assets to provide real value to its end consumers so these data products should be developed versioned and managed as as a software would be the second principle being data ownership and this translates into Data ownership being Federated among domain experts who are responsible for producing assets for analysis and business intelligence the third one being the self-service data platforms principle in other words this would be a central platform that would handle the underlying infrastructure needs of processing data while providing the tools necessary for a domain-specific autonomy and last but not least the Federated computational governance principle of datamesh which essentially translates into a universal set of essentially defined standards that ensure Conformity to data quality security policy compliance across all these different data domains and data owners so with with that Baseline being established um of what data mesh is and its key principles uh now I'm going to go back to your why question James um and I would say that there are at least three main reasons why companies are moving towards a data mesh architecture in my perspective in no particular or order I would say um scalability and Agility um as the domain teams within this construct can respond quickly to uh changing business requirements without centralized data teams for every related task which typically adds a lot of delays in bottlenecks I would say the other main reason would be reduced data silos which has historically uh plagued teams in traditional architectures so by promoting data as a product approach each domain can take ownership of the data and make it available for consumption by other teams and last but not least I would say that their reason in my opinion would be the empowerment of domain experts to manage all of the products in their business units independently without having to rely on centralized data engineering teams but even on a very very basic level for people to understand data mesh is like a lot of people you know are moving to the cloud and normally their Journey to the cloud is centralize all the data in one place so that people can use it but very soon they realize that centralizing the data in one place means that there is one team which is managing all the data so there's when when when when when when those teams are contacted for various kinds of analytical Solutions operational Solutions there is resource constraint that team alone cannot do everything that team does not know how the data is being used that team does not know what is the business that this data is going to be used for um so when data mesh came in and when it was introduced the whole purpose of that was to basically take that ownership from one Central team doing all the data related activity and create a cross-functional team which will consist of the actual data person the engineer itself there'll be a bi person which is a business analyst there would be a uh you know a dashboarding person which actually takes the data and creates dashboard there'll be data owners so people who are actually generating the data and what this does this cross functional table then do is it will basically they'll come together and come up with a data set or data product as Anna mentioned to act to make it available for people to use and the whole beauty of that is that any person in the team now knows what the data is where it is coming from how it is being used and what is the impact on it not they don't not one person needs to be always contacted saying how do I get the data or not one person has to be told how do I use the data because the team is already consisting of all those so there's knowledge sharing within the team and there's also a responsibility like ownership of the whole team instead of one person or one team owning it so at a very basic level that's what data mesh gives it gives you the control on what people are consuming the control on how you will publish something and how people consume it and what is the impact it's going to create like you can you and the data governance portion that Anna mentioned that assists this whole piece by ensuring that there's full audit control on what is being generated and what has been published by documenting it setting up obserability setting up data quality metrics so that governance helps people to make sure that what you're publishing is always audited basically and becomes a trustworthy system well Michelle it seems like a big aspect of this is the self-service aspect because it's going to democratize access to the data and enable a company to move much faster absolutely absolutely that's a very very good point it does does help democratize the data because now when you make the data as a product you can now ensure that all the disciplines and principles that you introduce in any kind of product you take care in the data and the data space also um you you are basically ensuring that the data which is being published is um is complete it's uh it's it's fully trustworthy uh it is documented um and it it has full um audit control on it saying that if something goes wrong people are automatically notified there's full Automation and then there's also the self-serve piece of that uh data product makes this data product available to other people without even contacting someone you can actually subscribe to the data product and start using it so that is the beauty of that cell service definitely a big big piece for the data mesh all right let's talk about the data mesh Journeys at Capital One and at Pitney Bowes how did data mesh evolve for your respective companies on on 100 data measure evolve a Capital One um before I I speak about the specific data mesh Journey at Capital One I just want to take a step back and and just provide a little bit of context about our broader Capital One journey and kind of how we got to deciding data mesh was right for us good so um Capital One has undergone a journey to reimagine our entire data ecosystem uh for the opportunities and demands of managing data in the cloud so back in 2017 we began by modernizing our data ecosystem in the cloud you know by building our data infrastructure storage and and the processes supporting these aspects from a cloud-first perspective which included the adoption of snowflake as our data Cloud platform so the cloud provided access to data from more sources but at the same time we encountered challenges in managing this exponential growth in data that was stored in different places and then meeting at the same time the demands of all the users with all the different use cases across the board which you would typically encounter in a large organization like Capital One so early on in that Journey we applied the product management and user-centered design principles to solve some of our initial data management challenges and um in taking this approach we actually learned that there were many players working together in our data ecosystem and scaling this very intricate ecosystem actually meant that all these different groups and all these different users needed to work together seamlessly so we noticed that there were different data experiences for all these different groups and users across publishing across consumption governance infrastructure management and some of these user experiences included for example you know those individuals responsible for publishing high quality data to a shared environment um it included business analysts data scientists that were looking to use the data to inform their business decisions there were people focused on enforcing data governance policies across the company to keep us in check and there were teams managing the infrastructure that made all of these use cases possible so if you think about all these different use cases and needs this was the perfect combination to create room for miscommunication and and creating point in time solutions by having these units silent in managing their experiences um so how does this all tie into the data mesh concept UMass James well by going through this exercise we took away a lot of learnings that we actually used to inform the way in which we designed and operationalized data measured Capital One so we knew we wanted to give a greater ownership over data to our lines of business but without sacrificing the in the important company's standards and compliance policies for data um so this led us to uh two-pronged approach to operationalize data mesh which consisted of centralizing uh policy through tools that were built into a central platform and enabling the Federation of data management across all of our lines of business so just to give you a understanding of what these two buckets are for the centralization aspect of the work that that was done uh you know we first broke out our lines of business into organizations and the units of specific data responsibility with their own hierarchy and each organization across the board had the same exact set of roles supporting their data needs we then defined um a common enterprise-wide standard for metadata curation right because not all data is created equal so for example in our case at Capital One the data that was used in regulatory reports uh required a much different standard of governance than for example you know the data that was in the staging tables we also defined the data quality standards for the shared data sets so we used um a very fine balance of a sloped governance approach which increased the governance's controls around access and security for each of these data buckets so for example production data needed to pass the more rigorous set of data quality checks right then data that a user may store in their private user spaces uh we also defined entitlement patterns Based on data sensitivity so what we learned uh early in our journey was that each data set was protected by its own entitlement which led to our users taking days if not weeks to figure out the right permission to request to obtain the specific data set but then only to find out that 66 percent of the time the data that they wanted was either poor quality or it wasn't at all what they needed so we tried to solve this issue by creating what we call the the business organization sensitivity combination which means that a user could access all non-sensitive data across the board in one place and only submit a new request for Access when escalating permissions and last but not least shocker we had to make this easy right for both data practitioners to follow as well as you know the data governance teams to enforce because we found that the vast majority of folks really wanted to do the right thing and be a good corporate Steward but all of the confusing and opaque policies that we had prevented them from getting there so these are are some of the the key steps we took for the centralization uh aspect and then for the Federation aspect you know one of the main concepts of data mesh is uh the Federation of ownership which is which is the idea that the ownership of data as a product should lie with the person who actually knows the data best right the the expert so a key step to federating the data management responsibly um was in providing um what we call a usability layer that was oriented by the jobs that needed to be done rather than a capability of the product like you know data cataloging or data quality so when we Define these jobs to be done we focused on uh publishing uh data as a new data product protecting sensitive data reconciling an infrastructure bill this all became jobs to be done so um to solve this we created the self-service user specific experience layer for like data producer consumers Risk Managers business data owners platform owners and so forth um to enable them to get the jobs done so through this approach we were able to actually create more accountability for the data quality improve the discovery of the data and actually build greater Data Trust across the board in our organization I know it sounds like there's a lot to unpack there one thing I'd like you to unpack a little bit further are those four user experiences you just mentioned yeah definitely it's it's a lot to impact there because uh uh operationalizing data mesh is it's not a a small project and that doesn't take a small amount of effort so as I was uh describing the Federation aspect of our two-pronged approach for operationalizing data mesh I was mentioning how we enabled uh self-service for uh data user specific experiences and the four experiences that I mentioned uh were the data producer the data consumer the risk manager and the business data platform owner so just to explain a little bit more what these user experiences are when you think about the data producer these are those individuals responsible for you know pushing a new data product and for these types of users we enabled a self-service data producer experience that empowered our lines of business to manage their their own data the second user experience that I mentioned is the data consumer so typically in a large organization with hundreds of thousands of tables it can be difficult for analysts and data scientists to find and Trust And even get access to the data they need so we enabled this self-service data consuming experience so that these consumers could go to one place to you know discover evaluate and get access to the data they need the third bucket that I mentioned is the risk manager user experience so this experience is for those individuals defining and enforcing data protection policies risk policy security policies uh and we enabled an experience that could make them feel confident knowing that the the automation of the governance was in place through these custom workflows and dynamic Warehouse provisioning so these workflows um ensure that the fully auditable Purge process and the ability to properly make changes to the data in a production environment and last but not least it was the business data platform owner experience um so data mesh does not only require Federation of data but also the Federation of the infrastructure supporting the data and these processes as well so for those individuals charged with managing infrastructure provisioning and cost we provided self-service tools that were focused at optimizing the cloud costs while effectively scaling cloud data and actually actually for this last user experience I will say this we brought this specific capability to Market with slingshot which is the product that I mentioned earlier in our conversation and is actually a product that can give companies the building blocks for data mesh and building their own data mesh Journey so um I mentioned that slingshot is focused on helping companies make the best on this not like investment but at the same time it's also a product that can help with those building blocks for data mesh what about on your end how did Pitney Bowes evolve with uh with data mesh what was your experience you would be surprised actually our journey actually also started in 2016 where we we moved to the cloud um first where we wanted to move all our on-premise data onto the cloud um and the main Vision behind that was that instead of 10 different teams using this call doing ETL and getting the data on their own and then creating a siled solution we wanted to have the data pulled in from each of the source system once into a data Lake and then give people access to it to that one source data so that they can all the people are looking at one version of the data they don't they are not doing their own ETL their own transformation is all done one time and exposed it so we started with the centralized data Lake and then we loaded all that centralized data into snowflake to be used towards analytical use cases but then very soon we realized that having one team do all the work of getting the data transforming it putting it into snowflake creating aggregations for analysts to use it was becoming a lot of resource constraint and plus everything all those people did not have full knowledge of the business portion of the data so they basically um allowed so we when we created the whole data mesh architecture we ensured that the engineers the bi person the uh the analysts they all come together we have uh uh a meat like we create a cross-functional theme uh meeting recurring meeting which they meet on a regular Cadence to see what what the data is how the data is being used who are the consumers of the data what is the business impact of creating this data product so that team basically will make a collective decision and because that team involves people from different experiences related to the different parts of the business it made that data product that much more efficient because then that data product that we are publishing we clearly know the business impact of that data so we could then technically monetize that piece that okay if we use this data we know that the solution is coming out of it and we were able to create some very very high efficient Solutions like Predictive Analytics like operational efficiencies like Revenue growth analytics we did a lot of those kind of work using the data coming out of the data mesh and our journey also like as I said started in 2016 into going to the cloud and by 2018 we were in full Cloud 95 96 of the data was in the cloud and then in it in 2019 20 time frame we started investing in our data mesh and that's where all the initial portion we also in fact I just forgot to mention one more thing we also have a data governance committee that we have created and that committee consists of people from various business units who basically come up with different kind of policies that we are going to implement on this data mesh those policies include how are we treating sensitive data that policy includes how are we going to transform the data into into Data products that policy would include how people are going to access the data like what kind of access does each person need to get so those are the things that uh we that data governance committee used to basically uh uh uh make the decision on and again the data governance committee like the cross-function team also used to meet in a regular Cadence so that everyone is keep up to date and then all this information that we are sharing is all fully documented in our conference pages and in our in our document system so that new people coming up will basically be able to look at that and pretty much connect with the system uh immediately and these committees or these teams they are not constant so you could have me as a part of one team doing one data product once I finish that work I can still go to another data product but I'm still the accountable person of the First Data domain so my accountability does not stop when the data domain is published my accountability will always remain with it but I might be able to work on multiple data domains at the same time so that what that gave is that automatically gave that internal knowledge sharing between com between in the company you know you no longer need to have that individual training because people can interact with each other and get that information so that was uh that was our journey basically so I think there's probably a lot of companies wondering is data mesh an optimal solution for every company Anna what's your opinion on this um data mesh um right now uh may not be the right answer for everyone um I think when considering data mesh as an approach I would encourage companies to think about a few key aspects to make that determination um I think they should understand how complex and and how vast their data inventory is because data match is best suited for companies dealing with significant data complexity and scale so I'm talking about organizations where the data landscape is growing rapidly with diverse data sources complex data models and and multiple data consumers across the various domains as well as their producers I also uh think that they should ask themselves if their organization is a domain-centric business structure because data mesh does align very well with organizations where different domains or business units have a distinct responsibilities um and that relatively autonomous in nature so each domain may have unique data needs and specialized knowledge about their specific data so if a business already operates in this fashion making the transition to a data mesh architecture could probably be a good fit for the use case and I also think that if if they they didn't ask themselves if they have truly an Innovative culture that adopts change quickly so a successful data mesh implementation does require a strong commitment to collaboration and a cultural shift in some cases so if an organization isn't willing to foster a collaborative culture where data is a shared asset it might be challenging for them to implement so I think these are some of the key aspects they could consider when determining if that's a good solution for them it also is determined by the size of the company like if you have a very small engineering team uh supporting a few data sets uh into it it does not make sense to actually create that whole architecture because you're not going to get that kind of benefit of uh and then sometimes in a smaller company having this separation out as and I was mentioning people may not be always open to it because it it is definitely a way to work differently than what you are used to doing it so uh you know an engineer told that okay he's doing this engineering work only and now he's told he's no longer an engineer alone he has to work with the analyst to create the aggregation he has to work with the reporting person to make sure the reports come out so he needs to know other things and that change as Anna was mentioning is very important if people are open to that change then it becomes easier to implement and with the smaller company or even a startup they don't have that bigger team to actually utilize the benefits I'm not saying they cannot do it it's just that they may not get the same kind of benefit that data mesh provides basically sure makes sense so there may be a learning curve involved here so what are the common issues and challenges that companies encounter as they operationalize data mesh and here's really the big question how do you recommend addressing these challenges Michelle what advice do you give so um as I said in my earlier earlier conversation it basically uh you need to um see first that is the data that you are publishing is it you know a you have um lots of data that you let multiple teams want to use it that's that is the first criteria you take that okay this data is going to be used by my five different teams and these are kind of uh solutions they are building on top of it once you know what that use case is on that data and what is the business impact of the data that's where you can decide whether that's where you can decide how to create that domain Centric data now the challenges that they may face initial challenges that they might face is basically um to if first of all have that culture that you want to have a cross-functional team across different business units because a lot of times in very very large companies um different business units don't talk to each other as as a small company in a small company the small knitted uh theme they talk to each other very frequently but in a very large company you will have people uh separated out in time zones people separate out in different functional roles they may not talk to each other or they may not be very much open to it so that's where the culture has to come up right from the top to say that this is the vision that we are going to have and that's where um you know you can use it so those are some of the challenges that you would have to encounter the person will encounter but what I have seen at least from my experience is that if you have the top level executive backing on things that you're doing you can show the vision of what you're trying to achieve effective a lot of people will accept it because they know that this not only will help them in their own area but it will also help the overall organization to come with the common goal and you have you are then you are also part of a stronger team now is are you doing things on your own in your own Silo you're now part of a team where um it your work will be directly responsible against an impact which will get created so that that is something which is is huge you know once you once you overcome that then that's where you get all the benefit I don't know what advice do you give companies to operation operationalize data mesh I was just about to say that the shall I agree with with all the things that you called out um I would probably add a few more and I'm I'm a huge fan of uh bucketing things in groups of three so I'm gonna choose the the three key challenges that um I think are are most prevalent so one being around data governance and standardization uh it's also something we have learned through our journey Capital One and I've heard other companies talk about it as well um which basically uh ensures that there is consistency across the different domains for how governance is established is standardized and it's enforced across the board so to address this challenge I would recommend companies ensure that they have a very well and clear established governance framework that has been aligned on by all the different stakeholders and teams involved through collaboration and also have a regular means of communication through like governance committees or any other shared methodologies of communication or documentation another challenge I think is monitoring and observability right so data flows have always been an issue across the board and it's hard to build really robust data flows and in a data mesh architecture they can become Amplified so if you have multiple data products and the domain teams involved which can lead to that complexity it just keeps adding to the to the challenge so to address that I think sometimes an investment is needed in a centralized monitoring solution that can provide that visibility and those insights into your data product usage into your data pipeline performance into your data quality system Health you name it so I think this should help identify those bottlenecks and other challenges early enough to ensure smooth operations across the data mesh and finally the third I would say would be a data product life cycle management so as the number of products in an organization grows it can naturally be challenging to manage the versioning deprecation retirement keeping up to date all of these different data products so to address this challenge I think companies would need to Foster an ability to for their data teams to operate in in a simple fashion just like devops teams if we're to make a comparison who are used to continuous Improvement continuous development style of like software pipelining just just like in software engineering I find that those types of principles also work very well with product management honor is there answer to the question how long does it take to set up data mesh or is it really depend on the company and their data infrastructure uh it definitely depends on the company in the infrastructure um there's there's no one fixed timeline for every single company yeah a data mesh is not a silver bullet you cannot just buy something from somewhere and start implementing it there's not a tool you cannot buy a tool and say start doing it um unlike other things where you can um but it's like I said that you know the benefits that you're getting if you do implement it are huge if done correctly now people have actually come back saying I have implemented it but I'm not seeing that benefit there could be multiple reasons one of the reason could be they may not have done it in the way it's supposed to be done they may have in order to um Fast Track the piece they may have uh you know cut down certain areas which is prudent for the data mesh to work and that's where things go off and then they say data mesh is not working because whenever you know I've got that experience a lot of people have asked me you know um how much time did it take for your company to do it and I'll tell you we reinvested a significant amount of time initially purely because it was a for us it was a huge undertaking because the amount of data that we uh that we bring in so we we had we took around six to eight months actually to to actually start the initial infrastructure get all the policies and process procedures set up like how we are going to create a data domain how we are going to create a data product how are the cross-function teams going to get created what would be the recurring Cadence what would the data governance committee do what are the policies that are going to be implemented we did a lot of leg work before we started even implementing it so that that basically as I said that you know not many economies will be able to spend that much time and that's where if they cut if they cut route and start doing it then there are chances that it may not work but that does not mean a data mesh itself is not a feasible solution right and in all reality it's it is a concept that it's not you know said and done it's something that will continue to evolve over time as technology evolves as processes evolve as data needs evolve there's a lot of continuous changing across the board in this ecosystem that I don't believe data mesh is a concept that has a specific timeline from start to finish you started but then it continuously evolves in time with your business needs and grows with you to serve your purpose well let's talk about key takeaways what are the key takeaways you want companies looking to implement data mesh to walk away with really the bottom line Vishal what's most important here um if if they have multiple teams using the data to create Solutions see using the same data and creating different solutions it makes sense to use a data mesh architecture so that a they have one way to look at the data in the say because what we had saw one of the things that we had saw which gave us a big benefit was people are looking at the data and creating a report two teams are looking at the same data and creating a report but both are giving different results same data but different result now we cannot say any one of them is wrong because both of them are doing their own kind of logic on the data so technically both of their report is right but to an executive they say okay which number should I trust with the data mesh piece because those teams are all coming together to generate it generate that product which is going to be eventually used on that on the on the dashboard that product will then be having a single result going out with all the information so then it's one one single source of truth of everything that's I think I would say that's the term I would use that data match allows you to get a single source of Truth on your data which is trustworthy honor what are the key takeaways in your review what do companies really need to know I would say that based on um all the different principles and examples and shared stories that we discussed here if I were to summarize um I think companies can can take away three key things to focus on if they're looking to implement data mesh I think first and foremost they have to determine if data mesh is truly the right approach for the organization and we talked about a few ways they can do that earlier in our conversation um if so if they determine it's the right approach for them I would recommend to start early because starting early does reduce some of the complexity and level of effort required to potentially retrofit existing products in this new uh Paradigm in this New Concept and last but not least I would definitely recommend uh using a similar two-prong approach as Capital One did you know where they would build this like Central policy and Central tooling that will then enable a Federated data management because you know data mesh remains just a concept unless these organizations can provide self-service tools and automated workflows to operationalize this this Federated ownership of data that we've been talking about what about if companies want to learn more on it where should they go to learn more online well first and foremost I want to thank you for having us here James this has been really fun and I actually thoroughly enjoyed this conversation today and Vishal thank you to you as well for joining this conversation with me listeners can learn more about data mesh or about all the things that Capital One software is up to at capitalone.com software Michelle Anna thanks for joining us today and I hope you come back and talk with us again sometime thank you so much have a good day thank you

This transcript was generated automatically from the video's captions and may contain errors.