hello hello and thanks for joining me once again as I get some opinions off my chest and today's opinions might surprise you so in the past I've shared my opinions about data mesh I kind of said I don't think it's appropriate for the majority of companies I think it's a fantastic idea that a lot of companies aren't really ready for yet with the whole organizational and social restructure that's required for that level of decentral ownership now some places great they've adopted it they've spent millions and millions and millions of pounds reorganizing and they've adopted their
whole hearted like yes this is it this is us others they're like no we can't afford to change everything we're not mature enough in our date of maturity our data is scattered in so many different places it doesn't make sense to spin up essentially a set of INF for every single domain that's there now for a lot of people that's like cool okay we'll throw away the entire idea of data M and we just we'll we'll treat it all as not for me and that's naive right because with every big architectural idea there are smaller
ideas which have fantastic Merit so today I want to dig into the world of data as a product or data products and there's a lot of Indy confusion about what those two things are there's a lot of people who are like no there's no confusion it's set in stone this is exactly what it is and then there's a plethora of blogs and thought pieces and all that stuff the entirely disagrees so I'm just going to add to the confusion and say this is how we and AA have been using data products this is how we
see it this is how the terminology we have actually you know adopted and used over the past kind of well probably 9 12 months to change how we're delivering things and change how we're thinking about stuff so that's what I want to talk about how we use data products how the bits of data products we see as good and we've adopted the bit of dead products we see as amazing but we'll never ever get anyone to agree to and a bit of that why it's not the same as mesh that's that's the plan that's what
we here to talk about as always don't forget to like And subscribe if you guys to hang around feel free to chck any questions and opinions down in the comments and I know people are going to have opinions about this one so yeah yeah so first and foremost I am not alone in my plan of going data mesh data products yeah sure I'll take that I'll r with it we'll go and use it Gartner for example put out their height curve in around July last year so the Gartner hyp curve kind of upcoming Trends it's
what's going to go well what they thought was really good but actually people a bit disillusioned with it's measuring the EB and flow of hype and buzzwords and Technologies in the data world there's ones for lots of different areas but this is the data management hype curve and you'll see got my mouse to draw today that we got data mesh actually right at the top and as early as two years ago whil data meses it's you know coming up to the biggest thing is being introduced the book was coming out they' already marked it it's
going to be obsolete before it actually plate becomes mainstream they're like yeah it's got some Merit there's a lot of hype but we don't think it's a long-term thing now this year continued hype is almost at its peak but the interesting point is this we've now got data product as a separate thing as a separate whole ethos as a separate pattern it's a separate thing that is now separated out from data mesh that is now rising in hype on its own treated as a separate approach so it's not just me it is not just me
saying actually the two are not the same thing and honestly data products the idea of applying product thinking to data has been around for a long time it was not first introduced by data mesh but it's been really hard to get people to think about it as a product the the businesses have found no impetus no reason to adopt it so a lot of the the terminology a lot of the the revamp excitement about it honestly a lot of the spotlight that mesh brought on to data products is fantastic because it means actually basically people
are interested in governance again it's good it's a great thing for us data professionals so what are we talking about so the best way of seen it articulated which not everyone agrees with is you got the idea of data as a product which is a whole methodology and you got whole approach and they have data products which are the output of using that approach so if I'm applying a data as a product way of thinking I'm going to build and produce data products as a result of the adoption of that thinking I'm like okay that
makes sense to me that is not how everyone articulates it is not how everyone describes it but I'm happy to adopt that and say okay let's take that definition so what is what is product thinking well if I'm delivering a new product if I'm there's a new website I'm going to make it go live and I want to go put it out into the world well there's a few things that I do firstly I do I do some research and find out what problem am I solving why am I trying to sell this website what
do my users actually want I have a product owner who goes through and they they prioritize those features they work out how things should work they design the right user experience I have a launch I do a soft launch to people I do a big launch I try and generate enthusiasm I try and sell my product to people articulate what benefits they get from it and how they should use it but also I have guarantees I give to my my users uh what the SLA is how long it's going to be up what the response
time is what they get what my side of the bargain is if they buy into my product what am I going to do to promise them that it's going to be the product that it's they actually wanted so a lot of those things have come over into this idea of applying product thinking how can we take all those things and apply it to the world of data so all of that is great but then we've been trying to take get the business to take ownership of their own data for decades that's not new but it
was always treated as a bit carrot and stick it was always very much you're the data owner you're held responsible I'm going to get a stick and punish you if your data is no good but you don't get anything in return where's the the benefit of being a data owner no one's ever said hey can can I can I be the data owner for this because it's always seen as as a bit of responsibility and a bit of punishment and an extra job that they didn't really want to do so this whole role of being
a data product owner and saying well actually you're the person who launches it to the business you're the person who gets to decide what features it has and what benefits it's going to do you're the person who when it goes live gets to walk around and wave a flag and go this is all the benefits the thing I have delivered actually delivers now that's that's great that is the carrot end of the stick where previously there was no carrot so this idea of building things and finding someone the champion for that thing that is then
deciding the features and you're giving away some control as the data team you're like no no better ownership this is your baby you can decide what the features are you can prioritize it you can decide what level of quality it needs you can decide what makes it good and what makes it bad but you then have to adhere to those things and you're then responsible for looking after it once it's live it is a given take and previously it's only been take so as an idea of saying that need to be a product owner there
needs to be a data product owner that makes huge amounts of sense to us it's a great way of actually making sure the business is fully engaged and invested in these various different things that we're deploying to do with data so that whole idea is just great so much traction in there really really good way of actually engaging with people um at that point let's clarify some things so I'm talking about data products so we're releasing something to do with data it has an owner in the business who is deciding what it should do and
they're selling it to the rest of the business and they're responsible for launch and they're responsible for maintaining it but what actually is a data product now there is a ton of disagreement out there in Industry obviously every single data visualization kind of tool says a data product is a dashboard a data product is a report any kind of individual thing is a product and should be treated like a product and I complet disagree so for me the whole point of self- serve analytics is that there's things that aren't treated like products I want people
to go and actually do their own bit of dashboarding do their own reporting I want people these days to write a bit of English language and for some large language bold to interpret that and come up with a oneoff dashboard that they then use interpret and throw away the whole idea of visualizations being the the actual data product just doesn't make sense to me they're the roll at rigor and things that go into product thinking don't make sense for me at the visualization level for me it's a step back from that we're talking about essentially
a data model just simplify things let's talk about Medallion architectures ronze silver gold in my gold layer I've got a Kimble style star schema that star schema is domain bound it's talking about particular business process that we have modeled into a star schema and that is used for Analytics that could be used to yeah put straight into a semantic model and put some reports off the back of it it could be used to just do some deep data analysis it could be plugged straight into an operational dashboard ticking away on a monitor somewhere it could
be used to feed a machine learning model that for me is the data product I'm producing a curated data asset with a load of features a load of uptime guarantees a load of quality guarantees and then yeah there'll be some dashboards and reports that are managed centrally that we put out to the user but there'll be a whole plethora of stuff that people build themselves cuz that's the point of self- serve analytics so for me a data product is a curated data set produced around a set domain published out to the business with a set
of specifications and guarantees and parameters around it so that's that's how I Define that's how I draw the line not everyone draws the line that way that's fine you do you so there's a couple of distinguishing things doesn't have to be lots of tables you could have one big wide table if you're going the one big table kind of uh data architecture method it could be a machine learning model because that in itself is something that is providing but there they're essentially the three kind of ideas well two data models of various different kinds that
are curated and ready for business consumption and machine learning models that is what I want to apply data as a product thinking too now this is very specifically everything I'm talking about is around analytics so yeah technically all of your Source systems all of your applications the data that they pump out should be treated like a product they are data products of their own right but that's that's not what I'm thinking about here I'm talking about analytical data products I'm talking about the data products that are built from consuming other data massaging it applying some
business logic applying calculations aggregating stuff together and producing a curated data asset to answer certain business problems that for me is an analytical data product now you might cry out now and go well that just means I've got loads of data products look at all my data products I've got star schemers everywhere and the differentiation is do they have a designated data product owner who is their Champion that is looking after it that is on the business side of the fence do they have agreed specifications and uptime and timeliness and guarantee and are those measured
and published out essentially are you treating it like a product so you might have a data Mar with star schemers are you treating it like a product if not it's not really a data product J you can uplift it and start treating it like a product great Hallelujah please do that but yeah it's about the application of product thinking that's what we're getting to it's not the fact that it's a star scheme so that's a big shift in terms of how people think about things it's not it's not massive we've been trying to do this
kind of thing for a long time but we've sudden found that with a lot of the focus on this kind of stuff over the last year the business is much more receptive to it people are much more receptive saying yeah no no absolutely can I can I be the owner can I can I actually deign all the features and agree what we we need to do with it and then can I get the credit when we launch it absolutely but you're also going to be held accountable for looking after it okay yeah sure so when
we when we design a you know a big data platform these days we we start they going okay cool there's the platform engineering bits there's the do you need separate platforms that belong to maybe different business units or different areas of the business or are you having your infrastructure centralized I don't care there's different business thing do they have the team do they have the resources does it was centralized going to be okay does it need to be split out and honestly technology that are you know off the shelf tech for doing things and having
a Federated system much easier these days data Bri you got Unity catalog you can have a bunch of different workspaces and they just all register their data with unity catalog and then you have a common layer for your analytical data products fabric very much going that same way of having all you've got these different domains people can stand up and whether you're using a lake house or a warehouse or data flows it pumps things out in the same standard and they're published in the same way you can shume it however you want people are heading
towards this anyway this idea of you can mix and match a bit of tech you can have dist distributed things you can have un unified governance across the top whether you do that unified governance against one set of infer or 10 sets of imperor that is just a logistical problem in terms of the skills that you have in your teams saying who owns that data model who's responsible for the design who's responsible for looking after in the upkeep and that being Federated that's the magic that's not pure data mesh that is not what data mesh
is getting at in the majority of stuff data mesh goes too far down the distributed Tech in my opinion so yeah it's it's a big change about how we think about things so when you're next about to publish out a new data model do you have a backlog that is defined and reviewed and owned by the business uh are you publishing some specs behind it saying we're going to measure these bits and call that data quality and we're going to publish a score about how clean that data is we're going to guarantee the timeliness of
this data it's never going to be more than 12 12 hours out a date 24 hours out a date 10 minutes out a date whatever it happens to be guaranteeing all the things that you'd want from a product documentation release notes when it changes change management and dependencies all boring SLA things that are so vital about people having trust in the product that you're trying to sell them works the same way when we try and get some data working internally massive massive shift in terms of actually how we think about the stuff that we're standing
up and talking to the businesses about so many data teams these days are kind of just looked so inwards go no no but we understand the data more than the business do we understand the source systems and so we we do the data models so that we can make the report of the business one it's like but if you're making a data model by looking at the source system that means you're not talking in the business's language that means the business don't really know what's available what's possible they're not thinking about oh if I did
it that way I could achieve a they're not thinking about the value that you get who on Earth designs a product launches a product and then does doesn't measure If the product makes any money and yet we do it constantly as data teams we stand up a new star schemer and then we're like does anyone use it don't really know oh maybe they use the report we can track usage on the report great teams actually look at Telemetry of users did they get value from it why are we building it we built this star schema
so that we could save x amount of money over the next two years so that we could harness a market opportunity and actually make x million in the next three years does anyone actually go back and measure those things in the star schemers that they produce often times data teams don't got no concept of that they've been asked by the business can I this report sure we'll build a data model we'll build a report we'll push it out but they're not saying okay you want that report what's the value can we measure the value you
get from the the report what business problem are you trying to solve that is product thinking that is saying there has to be a point for doing this and if we're doing that great so yeah big big fundamental shift in terms of how we think about things how we talk to the business how we manage things Who's involved that data product owner role it's kind of different to how data owner roles have been in the past it's a combination of a traditional product owner plus a data owner plus I guess some new thing that's kind
of growing it's honestly hard it's a specialist skill you're if you've got someone plucked from the business who deeply understands their business domain they desperately want to drive value are they going to have all the skills and knowledge about actually what's possible with data are they going to be able to talk in the language that your data engineering teams actually understand basically we're going to see a whole role with responsibilities and skills and learnings there's going to be learning paths you go so you want to be a data product owner cool you need to go
through this quick half day training course to actually understand some of the things that we're talking about say half day that's a quick get up to speed kind of thing maybe it's a lot deeper than that so that that is where we're kind of seeing data products going and it's yeah you might be surprised that I am so into it honestly but I've seen such a big change it's made and so much of it is because people looked at mesh came to us and said could you build the data mesh we're like no why what
are you trying to achieve and then talking around the problems that they're trying to do that idea of Federated ownership over the domain made so much sense and could actually bring them so much value but it doesn't have to be hide to all of the other stuff that comes with mesh which is so focused around the software engineering side the decentralized technology the standing up INF for everything side ignore all that pluck out that fantastic nugget which is data products which is again it didn't come from mesh originally but it's been popularized by mesh and
yeah for us it is a great idea and there's lots behind it now that as an impassioned rant about why it's a good thing doesn't really tell you much about how to get started um there is a huge amount in it around restructuring how you go about designing data projects how you go about Gathering requirements and actually the the roles that people have in those projects how you go about launching a new data model the fact that you have a launch if you've just published and got to production with a new star schema tell the
business run a showcase use Delta use data storytelling to actually craft and narra narrative about what someone can do and where the value is to inspire people to use it more people never do that and they should all of these fantastic things take a lot of prep take a lot of thinking take a lot of getting into that heads space but if you put the work in and think about how that can work in your organization if you think about how you can actually start to use those things there is just so much value in
it it is insane so yeah kind of bit that caught us a bit but surprise last year is just how much that exploded I am not at all surprised now that to see data products on the hype curve going up and not at all surprised that it's actually marked on the Gardner hype goers it's going to be uh mainstream within a couple of years you tend to get those categories of it's going to be within two years two to five years or it's going to take a long time before this is mainstream or it's going
to go up delete before it gets to mainstream like meshes marked the idea that data products couple of years and it'll be mainstream everywhere so start thinking about it now start making that decisions if you're about to embark on a massive data prod project or indeed you have a ton of star schemas that you're currently managing start thinking about how you can make those a little bit more like a product start questioning what do you promise to the business around it and how do you keep yourself honest with those things start questioning how you can
take help the business take ownership of it by giving them something in return it's a two-way street yeah so that is all I wanted to rant about just to talk about something that it's just we are talking about constantly with clients never really talked about it on here so I thought i' pop on have a bit of a rant get look to Da products and again I warn you there is a huge amount of literature out there in the world not all of it degrees degrees agrees some people have one definition of data product other
people have a different one what we are calling analytical data products that is one definition that we are using in advancing analytics other people don't necessarily do the thing the same make your own mind up that's you always my advice this is one opinion go find lots of opinions make your own mind up we are seeing huge amounts of value in it I suggest you look into it right that is all I wanted to rant about as always don't forget to like And subscribe and I'll find you next time with something techy and we can
talk about some code somewhere all right cheers is
