[Music] hi everyone and thank you for joining us today for this webinar on demystifying data ops riding the change my name is priyanka and i work as an event marketing specialist at hebo i will be your moderator for this webinar before we start today's session i would like to remind you of a few things all attendees will be on listen only mode you can type in your questions anytime during the webinar in the q a window that's available on your screen and we will address them at the end of the session so i'm sure most of you have already uh already know about hebo but as we have a lot of new attendees i will give you a quick overview hebo is a no code data pipeline platform that connects and loads your data that is lying across multiple sources into a data warehouse of your choice such as redshift bigquery snowflake and others hebo provides pre-built integrations with over 100 different data sources and also supports advanced data transformation hebo is currently used by thousands of data enthusiasts who are working across multiple verticals and who are based in over 40 countries for more information you can check our website www. hivodata. com i will now introduce you to our speaker for today's session uh christopher berg is the ceo and head chef at data kitchen chris has more than 30 years of research software engineering data analytics and executive management experience at various points in his career he has been a ceo cto vp and director of engineering chris is a recognized expert on data ops he is the co-author of the data ops manifesto the data ops cookbook and recipes for dataops success as well as an instructor for the fundamentals of data ops he is also a speaker on data ops at many industry conferences and with other media outlets chris began his career at the mit's london laboratory and nasa ames research center there he created software and algorithms that provided aircraft arrival optimization at several major airports in the united states chris holds an ms from columbia university and a bs from the university of wisconsin-madison and served as a peace corps volunteer math teacher in botswana africa it's nice to have you here chris thank you so much for joining us so chris let's begin this session uh the virtual stage is all yours all right well thank you for that great introduction and so what we're going to talk about is kind of an introduction to data ops and to do that i'm going to talk a little bit about myself as a way to do it and with that great introduction i think a simple way to say is i was a software guy for 15 years i managed teams and wrote a lot of code and then about 2005 i got the bright idea that i would switch to data and analytics full-time um and i had people who did beta engineering and data science and data visualization working with me and and honestly my life was not that great um you know i had small kids and and things were breaking left and right and you know i had i'd go to soccer games and sit there and pipelines would break i'd get calls or text and i'd have to leave and go fix things or marshall people to fix things it wasn't pleasant and you know i had uh we had lots of customers and it's nothing worse than deliver wrong data um in the form of reports or data sets to 5 000 people and then have the vice president call you up and say you know i hear you know if you do this again you're out it's just not a great feeling to give wrong data um and i had this sort of pit in my stomach every morning like i would come in and i'd be worried that something would go wrong in the production line and i get this email saying you know basically you suck the data is wrong again and i had to brace myself every day for things that were going wrong um and i had a i the company i worked for did analytics for the u.
s healthcare industry and the us healthcare industry has got lots of notoriously complicated data um and as a result it has a whole opportunity to create new insights from that data and the boss that i worked for was a harvard educated doctor and he knew a lot about health care and had a lot of great ideas and he would come to me and say hey chris i've got this idea of how to analyze data and i would sit down with a data engineer and a data scientist and maybe someone who knew tableau and uh we'd think about it we'd say okay we think this is going to take us two weeks and he'd sort of pull his glasses down and say chris i thought this should take two hours and not two weeks and so you know and and lastly we i i hired just a bunch of smart people and and smart people like to use their favorite tool they've got favorite tools to put data together um they've got favorite tools to visualize data favorite tools to do data science and you know i just i had a lot of conversations about you know i want to do elt versus etl i want to do r versus python and so um after years of me complaining about this my wife finally said you know you got to fix this problem um and it's you know i can't take you complaining about work anymore so either quit your job or go ahead and fix the problem and so in some ways um my journey of the last 15 years has been about solving these kinds of problems which are i'm going to talk about next and so i started the company and i'm going to walk through kind of this idea of data ops and what it is and so i think one way to look at it is the the core problem is of perception like we've got a lot of data tasks that we have to do as a team we've got to put data together we've got to put an algorithm on top of it we've got to visualize it we've got to govern it um we've got to judge its quality and this sort of river of work that's right in front of us and that focus on the river of work has sort of blinded us to very large scale problems in data and analytics i. e that most projects and data and analytics fail um that most pipelines have too many errors that most data science models never get into production and so we're really focused on the task at hand but there's these things upstream that are really problematic and if you look at it from up upstream really what that means is about how and not what um and the how is how do you develop new work how do you iterate quickly how do you deploy things to production how do you monitor and check it once it's in production how do you test how do you collaborate and these house are more people and process problems um then they are doing the actual work and so when uh eight years ago we started the company we ended up um getting some customers building a product and five or six years ago we decided to bring it up to the market and and we found that no one really kind of got what we were talking about so we had to put a name on it and uh we wrote the first wikipedia article on manifesto and and we called it data ops because some other people had been using it and our other names were quite poor like agile analytic operations or devops for data science and so if you could think about this is that data ops is it's not about what you do it's not about the model or the transformation it's about how do you actually iterate how do you take things from your mind as a data engineer scientist and get it into the hands of your customers and that cycle time of deployment how fast you can experiment is really about how well you learn and then also you run a factory to deliver analytics and how what's the error rate on that factory are your reports wrong are you late and how much time are you spending in meetings how collaborative is your team and then finally how do you measure and monitor results and so we've taken that uh called the data ops written some books and so that's what this topic is about to try to say what is data ops how it's different than devops how it's different than doing data and analytics um and so i'm gonna go kind of into that right now to sort of talk about um what that is and so let's say we just start off with a really broad brushed picture of people who do data and analytics and so there are data engineers and they maybe use a tool like kivo or other tools to access data and integrate data and put it in a shape for analysis and then there are kind of two groups of people who do the analysis one is a data science team who uh takes the data and um segments it groups it organizes it for uh an algorithm and then there's perhaps a self-service team or a business analytic team who visualizes it and then lastly there's a data governance team who tries to keep track of it and so if you just walk through uh the market there's a 80 billion dollar market and tools for each one of these people um there's a whole there's sort of 50 data engineering tools 50 data science tools 50 uh visualization tools and new ones are being created each day um and so our perspective and if you walk through this is that like let's just walk through an example to give our perspective so first the data engineer they source data and they have a sales is a really simple example they get the sales data with joe and kelly on and then the data science team applies a cool algorithm to it and you know what they segmented and joe is a is a low value in kelly's high value and then the self-service team gets at it and maybe what they do is they mix in some data from alteryx they visualize it and they add in this column and then lastly the poor data governance team tries to find out where it's from you know where uh who added the data where it's uh where it came from and that's called data cataloging or data lineage and so all these people kind of work together and in this idealized model there's different groups of people and oftentimes they they overlap but they generally work in teams and another way to think of the process of doing data and analytics is these teams of people take data from lots of places um and crm erp supply chain but they actually produce insight and and one of the things that we talk about in data ops is the idea of a data factory or every step that you do to integrate data to build value from data so accessing it transforming it running a data science model on it visualizing it governance security all those tools that are used to uh work and assemble the insight from the data think of them as workstations in a toyota factory and you're putting data together you're adding value to it you're putting artifacts like reports on top of it and that is an assembly line and so that assembly line itself if you've ever run data and analytic teams is problematic things break um and so that's one aspect think of what you do as an assembly line and the funny thing about our assembly lines and data and analytics is they're not owned by one team in a lot of organizations there's a central data team who builds a data warehouse or a data lake or has a data enablement there's a data science team who uses that data in different they have different names and different verticals they're an ai team they're a risk team and in lines of business there tends to be people who are focused on visualizing um and giving information to customers and this sort of hub and spoke model um is hard and there's lots of people who are involved in the manufacturing of data and there's this rule of in engineering called conway's law that you know ideally when you build a technical system it should be the interfaces on that system should be based on sort of logical structures of the system um and conway's law says no that actually turns out the way systems are built are actually reflect the organizational structure and i think that's that's true in our our data and analytics world i mean that's not an ideal situation and then lastly uh one of the beliefs here is that i think the core belief in data ops is that you need to deliver value to your customer and that's the measure of success not that you've done a task in a project list is that your customer is has analytics that they're using and getting value from and the second tenant of that in some ways is that you should do it in short iterations that you should be able to give something to your customer have them use it and give you feedback and that cycle that iteration cycle should be as quick as possible and and why does that matter well you don't want to spend six months and giving yourself giving your customer a very complicated system only to find out that you didn't get the right requirements and so the idea of agility the idea of short cycle times is really important in analytics because the other thing i have learned is that analytic customers um oftentimes don't quite know what they want and they have to if you give them something that's working give them something that they can react to you get a better work product at the end and you end up doing less work that's the magic of productivity of iterative agile cycles is that your productivity as a team goes up because uh you're doing less you're giving the ratio of work that you've done that is a valuable customer to unvalue is much higher and and but the challenge here is we've got to do all these things on purpose and and data kitchen we've talked about these two pipelines the value pipeline which is production the innovation pilot is in deployment but you've got to be able to do these things simultaneously you got to run your toyota like factory um produce analytics with low error rates you've got to be able to pick up a piece of that factory and change it um so that you have a fast cycle time to iterate to be agile like a customer value you've got to sort of hop over the fence to talk to each other in the conway's world when your data science team and your development team and your production team and then lastly one of the aspects i think of of agility and certainly lean is a metric driven way and try to measure your team and measure these the cycle time your value pipeline your innovation pipeline your error rate and so all these things kind of go in and talk about sort of what data ops is and sort of where it fits in the world of data analytics so to sum up you know what data ops data ops in some cases is used as a a moniker to cover everything in data and analytics and i i don't think of that you know you do data ops on a data fabric or in a data management tool um and i think data ops is a process-oriented methodology whose goal is to help teams deliver better value to the customer and you do that through faster iteration cycles and focusing on low error rates in production and do that through driving collaboration with your team and measuring the processes that you use to build and run things so if i go to the next case and kind of talk about what i want to do here is kind of give a concrete example sort of talk about what happens with some customers and so to do that i'm going to kind of go through and sort of talk about some pains that people have when they start doing wanting to do data ops and you know it just to be concrete um you know i think all of us who've been in data analytics have heard some of these things like hey your data's wrong you know can you fix it um and or your team is just too slow you're not getting enough done um and or you know i can't prove my team's got value you can't measure your results or can't show your value and so um and they all come in different forms um and so like if we talk about from a people standpoint you know that there's your data is wrong let's talk about what eric feels and eric's the production engineer and then we're going to talk about development data ops which is ahmed who does deployment and then we're going to talk about the manager and we sort of uh bracket those up into sort of production and development and measuring data ops but i think these quotes here are very common in data and analytic teams you know your data is wrong and your heart sinks and you sort of get them uh wait for that email when you get in the morning or your boss looks down your glasses saying you're too slow or you just can't prove that your your team's doing anything right because it's hard to you're not measuring the work output or the deployment output of your team and so let's just talk about this this case here of eric and so eric is a production perfectionist right and everyone has a team where eric exists and so they're in charge of running the factory day in and day out they are the data comes in it goes through a manufacturing process and reports and models and governance are are put out and they want to minimize errors people like eric are not about um they're about getting things right and they want to make sure it's on time and if there is a problem they want to know where the problem is right away and um and to do that they sort of want a single pane of glass over their factory or factories um and so here's an example of an insurance company and so it's a big american insurance company uh they built this fantastic what's called telematics and so what that means is you put a beacon in your car and that beacon sort of tweets out your lat long and your velocity in a time and then a bunch of companies aggregate that data and then that beacon then goes into the insurance company and they look and they say wow this person is you know driving too fast or breaking too fast and that can end up affecting your insurance rates and so you know for some people giving a privacy like that is anathema for other people it's a it's a good way to get a discount so depending upon your political perspective i think um that has one influence on this but they built this really cool system that's event driven uh things happen in kafka it goes into the cloud it goes into data bricks to do etl there's some proprietary scoring and risk models there's databases like snowflake there's a policy system and then there's some applications and so what happened is they built this last couple years they built this really cool thing it runs it does great it brings in from half a dozen different systems it actually does the algorithm to um score it they and the problem is that it runs fine uh the challenge is they don't know whether the data that they're getting in it is good or whether things are running so for instance this is all a containerized application well one container went down one day and they started to get bad data and the people who are using the applications and looking at it were saying it's wrong or sometimes the data providers would suddenly give data and the car would jump a thousand miles in one second and of course that can't happen right or it would be incomplete and so how do you then tell if it's the system is the problem or there's other problems and the other challenge is that this was built kind of across lots of people there were some people doing the event bus some people doing the data science some people doing the data warehouse and so when things go wrong kind of who do you call and so the idea here is that if you could put something on top and kind of check the data but also check the things that are being created from data so as you're applying and interacting with the policy system as you're checking the raw data itself and integrating it can you get low errors and notify people when they're wrong and perhaps even quarantine bad data and so the idea of testing in production or some people call it data observability i think is really important because pushing down low errors and it's errors not only in data but it's also in the data that you've integrated together and in the things that you've done on top of data the reports and the models and and keeping track of it and notifying it and so from a principal standpoint in terms of production data ops one of the things that we believe is that you should test your data in production automatically kind of on top of your favorite tool your favorite etl tool your favorite data science tool your favorite bi tool and if you find a problem in it send alerts and notifications and and to do that you've got to kind of keep track of history and make it easy for people to create tasks and there's a whole bunch of ways to think about checking data like one example is statistical process control and a very simple example is day in and day out you've gotten about a million rows every day from a data supplier well it's tuesday and you got 100 000 rows and so that's something to be noticed and and that's something to be notified and that's sort of a trend break and so the idea here is by focusing on error rates in production you actually get more time to drive innovation and you get more customer data trust because what happens when things go wrong is a lot of smart people in your organization are working nights and weekends to fix it and so yeah errors happen in data but you can build a system that can react to those errors and find them out before your customer sees them and that's just less stress less time to failure and improves your customer data trust so that's the the first part of example i'd like to say um and so i think i see a question here i can't see what it is um we'll have some time for questions at the end i think so but the second is that um if you think of how fast so a tenant of any agile process in uh development or not um and so um is that you can learn quickly from customer feedback so that means driving iterations that means deploying something quickly into production and so a lot of organizations have challenges in deploying because they have lots of environments they have a development environment and a system test environment and a pre-prod environment a qa environment and there's two aspects to it one is sort of the is there a railroad track that moves things from one environment to the other have you made it easy to pick things up and there's deployment technologies like jenkins to do it but then there's the second is can you actually have the signals on those railroad tracks to tell if you can move from one stage to the other without any problems and i think that's that's really interesting and a lot of data and analytics systems you you're doing your work as an individual so i make a change in one piece i make a change into some sql what's the impact of that change on the model or the visualization downstream so how do you actually judge that how do you properly regression test or do impact analysis in a development or a qa environment and that that's really the key here it's not about moving things from place to place because that's that the sort of infrastructure is code and and ci cd processing solves as solved as how do you actually regression test and so um you know the way to do that is actually decorate your whole system use a lot of those same tests that you've done in production and so automatically test don't manually test in in a development or a qa environment and do what software engineers say pull the pain forward if you see as an individual contributor you've made a change to a data transformation or a model you should be able to see at your desktop whether there's a problem and whether that works or not and to be able to judge the impact across all your tools and all the subsequent work and the idea here is that code-based unit tests like i make up some fake test data i run it through works they're good but they're insufficient and so what this really means is you've got to test data to test your code and so in development or in qa or pre-fraud you've got to pour in good test data into your data and analytics work and reuse a lot of those same production tests for what we think of as regressional or regression or functional tests and by putting impact analysis in a qa environment or i. e do checking regression tests you actually improve the velocity of deployment and you re you allow people to sort of remove their individual blinders and it lowers the risk of change and less customers eye roll because the team you broke something um and so i think this is this is amenable to automation and amenable to being able to have automated testing and a good way to do data ops and so the third example i want to give is measuring data ops so and this is really how does a team leader understand how their team's doing how do they look at errors across all their pipelines how do you assure you make your customers a success and prove your team's value and in their in in short they're sort of think of it as process analytics there's a way to measure and get analytics about what you're doing and so from that perspective there's um production metrics like how many pipelines are in production are they on time are they coming up there's data source metrics being able to look at different data providers and see if they're good um and then there's project metrics like how many tests are are run and how many have i met my sla how many deploys am i doing and so these metrics about the processes are actually really important and i think of them from two what two ways one is because if you've got these metrics it's a way to show that your team is awesome as a manager and then number two you can use these metrics to help in troublesome circumstances so for instance that chart on the upper left we call a tornado report it actually shows problems by data source and then you can use that to give leverage saying hey over the last month you've given us crappy data sources 15 times and it's cost us this many hours of work um that's leverage or it's also being able to say that to people who um are less than uh are skeptical of this data ops idea um saying that yeah you can go fast you can deploy fast and you cannot have a whole lot of errors and by showing the relationship of tests to deploys to slas i think you can do that and lastly there's a lot of more automation you can do with data ops you can automate your the build of your environment right to actually script it you can automate the build uh you can actually handle something called hub and spoke pipelines to be able to have handle your self-service you can handle different types of deployment pipelines i'm a fan of um more functional programming and functional deployments and then just production pipelines there's a lot of automation that you can do you know when these a couple examples are really focused on automated testing and then automated monitoring and but there is a whole lot of more automation you can do in data ops and and that's needed because like there is just a lot of tools out there like if you look at azure or aws or gcp in the last five years they've assembled everything that you can do in data analytics visualizations data science models et cetera and they've got databases and bi tools and and devops tools and and so there's no the complexity of all these tools is actually hard to do and so um and to do that i just want to talk about a little bit before i finish the role of the data ops engineer before i finish up and i think this idea of automation is what the data ops engineer owns and you think of it at this the data ops engineer owns the assembly line um and they're trying to help people who do data engineering or data science or bi be successful that's their customer and in a lot of cases um you know does it mean that a data engineer takes their sequel and throws it over in a pod you know what does done mean um how do you help these people get out of sort of willful blindness how do you help them focus on their customer how do you help them stop doing manual checks and do automation and so for a lot of cases what we think of the role of the data ops engineer is to build a system that takes these nuggets of work etl code data science code visualization code put it into through pipelines create tests create deployments run the factory automate work across people measure success and so in a lot of ways what the role of the data ops engineer is to very simply take what's invisible and make it visible and that has the net effect of helping productivity of your team and so i'm going to skip these last ones and say think of the last thing of the amount of work your team does in terms of data ops engineering if you look at the average software engineering team twenty eight percent of their time actually this is like 23 of their time is doing devops and that's sort of making what's visible visible working on the system working on the how and not the what and i think we should think about this idea of greater data ops engineering spending more time up to 15 percent of the time and so in the the last few minutes here i promised i'd go for 30 minutes i just want to talk about the what what this all means and what the net effect is to end up so imagine that you've got a graphic equalizer here on the left and your cycle time of deployment from development to production is measured in weeks or months you know every every other day you're having some significant error um you're you're spending a lot of time in meetings um and you don't really have an idea of what's going on and and the net effect of that we think is that team productivity is low and your customers are unhappy with what your team is doing and we think if you go and do data ops you can actually push up this graphic equalizer all at the same time you can deploy faster and you can have less errors in production at the same time they're not opposite you can actually improve the collaboration and improve your process measurement and if you do all those things your team is actually more productive um and your customers are less unhappy um and and i think what that means is that you actually are giving more insight to your customers and that's the um the net goal here and so in my final two slides i want to say you know we are data kitchen is a software company we have a software platform that helps people do data ops um and helps people accomplish these goals of decreasing cycle time lowering error rates and production and improving collaboration and measurement um and so we have a great software product i'd love to show it to you but one of the things that we do is we've had to write a lot about what data ops is so we've got two books that you can download for free we wrote a manifesto we've got a data op certification video that you can go and so if you go to our website on datakitchen.
