foreign [Music] what are data products you may have used them before or maybe you're considering adopting them in the future as with any technology you're probably asking yourself a few questions what exactly are they and what do they have to offer what problems are they designed to solve most importantly of all where do they fit into the traditional ecosystem of big data and how do they do things differently all great questions let's dive in first let's start with a problem familiar with in the Big Data space The Divide between data producers and data consumers in
many organizations responsibility is dispersed between those who produce data and those who consume it this follows the division of labor needed to operate in traditional ETL pipelines alongside the more strategic business functions derived from organizational needs typically data producers are responsible for understanding how to extract transform and load the data this is often known as ETL and has traditionally required a complex set of technical skills for this reason data producers are experts on working with data but not experts on the context of that data this context changes from organization to organization and business case to
business case meanwhile data consumers are responsible for understanding how that data will be used to gather insights and drive value for the business this division of labor has served as a blueprint for the industry for some time and allows different Specialists to operate within the sphere of influence working together towards a common goal but there's a problem the gap between data producers and data consumers requires a lot of time and energy to manage and it doesn't always go smoothly you might think of this as the data divide and managing it costs businesses both time and
money data producers understand the data itself but without context data consumers understand the context but not the data overcoming this requires an ongoing conversation between the two groups causing data to change hands many times this process is time consuming labor-intensive and can lead to confusion and a lack of accountability owing to the complex web of responsibilities across projects in the meantime the business questions that are supposed to be answered may have changed leading to inaccurate and unhelpful insights this is where data products come in they help overcome the gap between data producers and data consumers
by empowering consumers to do some of the work previously done by producers let's look at how they do this in more depth at their heart data products are an Innovative Modern Way of creating packaged data sets to use by Downstream consumers to achieve this they are both curated and designed to create value let's look at how curated data sets allow data products that are demand driven and built for a specific need they are typically made up of three components the data itself the metadata surrounding it and the access patterns needed to access that data data
products create value by presenting data in a way that makes it more useful and more accessible importantly these data sets are the same ones used by traditional methods but they're much less complex to operate this allows non-technical teams typically data consumers to take a more direct role in managing data themselves because these teams know the business context best this eliminates many of the complexities associated with communicating across the data divide in the past data pipelines existed as a complex Patchwork of different Technologies working as a kind of relay system moving data through the different stages
until complete data products bundle all of this together and allow users to interact with their data as discrete independent entities this means that each data product has all of the structural components to do its job as a discrete object access to the data product should give you all the information you need to gain insights there is also a social Dimensions data products they are typically created for others shared widely and used across teams as such the collaborative way in which we create them deploy them and interact with them is one of their defining characteristics taken
together data products are a powerful tool they help overcome the data divide and help address significant organizational bottlenecks perhaps most of all they help businesses derive greater Speed and Agility from the data they use this enhances one of the core reasons that data exists in the first place
