The Root Causes of Product Failure by Marty Cagan at Mind the Product San Francisco

[Music] [Applause] you know this is a really a unique opportunity to get in front of all of the product people is just well it's rare first of all and I don't want to waste even a minute of it which is why it would be good to see some slides there we go so there we go I wanted to talk I actually feel very strongly about what I want to talk about today and I'm gonna sort of break all my rules you know like Mina just finished up with the three key points I added them up

and there's like seventy four key points I want to make it in the next minutes so I but I'm gonna throw a lot at you but I so I spend almost all my time working with companies mostly startups but someone that some companies get to the IPO stage growth stage scale there is a tremendous difference I find constantly between how most companies work and how the best companies work most product teams the best product teams that drives me nuts I think as an industry we don't do a good enough job sharing and this is one

of the reasons I'm so happy to have this opportunity in front of you is to try to share now I'm going to try to shine a light on what I believe are the root causes of why so many product efforts fail and we shouldn't kid ourselves the vast majority of product efforts fail for every uber for every Airbnb there's thousands of failed companies and now even within an uber or a Facebook or a Google but you know we heard you know for every AdWords there's many Google Glasses or things that struggle and have a hard

time so even within a great company there's a lot of challenges and I I this there's me I think we can have a much better track record but I will say I'm gonna I'm gonna try to be pretty blunt in this I want to what my plan is to do this it's to start by shine a light on how I think despite whatever nomenclature you might have despite whatever sort of a gillece does you might have in your company that have a you know very strong on what they think you're really you're doing I want

to try to describe what I would bet money the vast majority in this room are actually doing okay on how you actually work so I want to start by doing that and I'll try to not editorialize I'll try to just sum it up quick and then I'm gonna go through the ten biggest reasons why that way of working is so fundamentally flawed and I'm gonna that's where I'm gonna spend the heart of the time because to me what matters here the big opportunity is to have you connect the dots about why the way you work

has so many fundamental problems and then and then just to not leave you completely depressed at the end of today I'm gonna give you a glimpse into how the best companies work best teams work now I actually know a good number of the teams in the room and I can I can tell you for a fact that there are several companies several teams in the room or people in the room that are working not like I'm about to subscribe they're working like the way I'll finish up with as a great team but the vast majority

I'll lay money and you guys can you can let me know or you can decide for yourself how close this is to how most people how your team works now the other thing I'd say is even if you're a company that doesn't have these sort of flawed ways of working if you grow if you kind of get that product market fit and you kind of gets get growing is a really common problem that the company sort of back slides into this way of working as the company grows it happens all the time we'll Noah great

startup and I bet you guys have heard this I hear it all the time I'm introduced to a CEO usually from a VC they'll say you know and we got when we just were going early we were we were experimenting constantly we were learning fast we were growing and then we kind of reached this point and it's like I feel like we can't innovate anymore I hear that all the time I can't it we can't innovate anymore and the other thing they say is and it takes us forever to get an idea life you guys

have all heard this and this is because the way they're working now is not like the way they worked when they were an actual startup there's some legitimate reasons for that there's also a whole lot of bogus reasons for that and I want to really highlight so even if you're not one of those teams in the majority I think you should you'll want to know so that you don't fall into these traps I can that's just skip the time here and go into this the let me just again I'm gonna summarize how I think most

teams work all right first point everything starts with ideas it is true that a lot of companies like to pretend things start with something like a lean canvas but the truth is it starts with ideas and then we sort of reverse-engineer the canvas but the idea is we start with ideas now there's two there's two main sources in most companies of ideas there's the executives the business owners the stakeholders whatever you'd like to call that group of highly paid people and then there's the customers if it's a b2b company it's usually big customers if it's

a b2c company it might be a little bit from customers might be advertisers or other kinds of partners those are the two main sources and okay two main sources of ideas now most companies most companies their big goal is to make a roadmap they usually do them on a quarterly basis sometimes monthly sometimes annual but one way or another they want to come up with a roadmap which of course there's lots of flavors of roadmaps but basically we're talking a prioritized list of features and projects what are you going to do this quarter now in

order to do that roadmap they need to do some kind of business case now I have it in quotes because most companies today don't do a formal business case they're informal you guys have all seen the sessions that many of you probably spends a significant portion of your time managing the product roadmap for your product but basically it's not crazy why people want these roadmaps there's really two big reasons the first one is they want to know that you're working on the highest value things first you've all heard the concept of business value and then

the other reason usually even the bigger reason is they want to have some level of predictability in the business they want to know what's gonna happen this quarter so they can run the business and those are not insane reasons now in order to know what you should put on that roadmap and what the priority is they do some form a business case now business cases really are based on two things they're not actually even if you do a formal one they're not complicated there's two main inputs how much money you think you're gonna make or

how much value do you think you're gonna create and how much is it gonna cost or level of effort whatever term you guys use so anyway they'll do that business case for the four of them they don't do it for every you know little feature or a little bug certainly but they do it for typically features and bigger projects and initiatives and anyway they come up with a roadmap next eventually something makes it to the top of the roadmap it's time to work on something and this is where you usually enter the picture some product

manager will be assigned or it's for the team so the product manager will picking these world words carefully gather up the requirements and document them for the engineers or if you have a UX team which we hope you do but then giving this sort of description of what needs to be built and may be as simple as user stories to the designers interaction designers visual designers giving them to the UX team that UX team of course will take that and come up with a design they'll come up with an experience and they'll usually that's sort

of the other half of what has to be described and eventually that will make it to some developers and the developers will of course build that out now that's usually where agile comes into the picture so this is showing that yeah maybe it takes three Sprint's or two Sprint's to do something fine you might have depending on your organization maybe it's maybe it's the developers are doing all the QA as well but most teams there's sort of a few people that are dedicated to test automation or even manual testing and they'll make sure that basically

it works as advertised and then hopefully finally we deploy okay nothing I said should sound well my bet is what I just described is pretty close actually looking at your nodding heads looks like you recognize this so this is what I see in the vast majority of companies most of you may recognize this is waterfall right does everybody realize hopefully if you're in this room you recognize this is waterfall the the agile is you know this much agile we're talking about but we'll get to that but this is if this is how you work and

you know the truth is some companies will do bits and pieces of things that aren't quite as I'm gonna sort of paint a picture that's pretty binary but it's it's a bit of a spectrum but my guess is this is pretty close to how you work and frankly this is how teams have worked for a long time before I mean for 30 years teams have basically worked this way it's not like it's insane but it's I'm gonna absolutely argue it's incredibly ineffective and this is the reason why so many of our efforts fail so what

I want to do now is get into the heart of this and talk about why is this so bad so ten key points I sort of I honestly I'm not exaggerating can talk all day about why this is so bad but I've just picked the ten biggest reasons why this is so bad so the first one it's a source of the ideas themselves so I described it in the typical process the two main sources are the company executives or business owners stakeholders or customers now the real issue is not that those two are terrible sources

of ideas I will say those are generally speaking our two least valuable sources of ideas okay that's the truth the biggest sources of ideas are not actually at the party this is the big this is the bigger issue the lost opportunity the biggest consistently the best single source of ideas it's your engineers and they're not even in the picture they're typically not ever in this room where are these things are even debated now the engineers why there's actually a really basic reason why the engineers are so often the best source of ideas it's because they're

working with the technology every day and they're in the best possible position to see what's just now possible which is really with product what we're trying to do is match real customer need or pain with what's something that's just now technically possible and that's sort of when the magic happens and so to not include the engineers at the ideation at the very earliest is just a huge lost opportunity and again one of those really big differences between the best teams and most teams that's just one source of ideas that's great another favorite source of ideas

for me is data now people talk about being data-driven and stuff but the data is typically not in the hands of the people doing the ideation in this old way it's not the executives maybe you know they see date a third hand but they're not inside the data every day customers certainly are not of course customers remember their customers are awesome we can talk a lot about how to really engage with customers but what they're not good for is is coming up with our product ideas they for two fundamental reasons number one they don't know

what's possible and this is not their business their businesses whatever they do it's not to know what's technically possible and number two and this applies actually to all of us not just our customers but know what they want until after they see it this is fundamental truth about tech products so anyway the source of ideas in this way of working you know you might occasionally include an engineer or some idea might get passed up but it's not institutionalized it's almost the opposite is institutionalized so the source of ideas is a fundamental problem in this way

of working second way second big problem is this idea of business cases the fallacy of business cases I was trying to remember Tom Qi had a great phrase that I wrote down but of course I don't have it in front of me but it was speaking to this point I Marc Andreessen he has the phrase that's always stuck in my head about this which is in technology products most important thing is to know what you can't know and I'll tell you you can't know this is the two things you can remember I said business case

is based on two basic inputs the first one is how much money you're gonna make and the second one is how much will it cost you have no clue how much money you're gonna make you can make this up and I've done that game for years - I know how to play it and I know how to get my stuff I can like Linda said I could tell a good story and you know selectively use the data and we can all do business cases of course we always get nervous that somebody's going to ask us

six months from now how it worked out but thankfully they don't want to know either they already know it's bad that's true so but here look there there's no way let's just go through those one at a time the first one how much money gonna make you have no clue because it completely depends on how good your solution is we don't even have a solution yet so how do we know how good it's gonna be all we have is an idea somebody decided it's time to integrate with PayPal and we can say well we're gonna

make 25 million more a year because we're gonna integrate with PayPal there's no way to know that until after you've actually integrated with PayPal and decided or at least until you've run some smart tests to figure out how much that PayPal integration can actually get for you of course the thing that people just like to skip over is there's a whole spectrum on how good a job you can do with any of these things on your roadmap any of them if you do a really crappy job it will probably make nothing and that's not a

hyperbole I mean literally in an a/b test we can quantify we can attribute how much this thing helps or hurts as the case may be and very often it is literally zero literally zero it's one of the reasons I'm so pessimistic about features today it's because so often especially in a mobile context don't assume a feature will actually contribute anything positive it may just as well be a negative but anyway we don't know how much money it's gonna make similarly we don't know how much it's gonna cost to build you can go to your engineers

and most companies that use this silly process do they go to their engineers and say hey we want to integrate PayPal or whatever I shouldn't and they say and the engineers say well what does that mean you know I mean there's lots of ways there's and and are we talking about on all devices and are we talking about in-app payments and like what are we talking about they're like we don't know that yet we're just trying to decide if we should actually do it and so they'll have this back and forth and that's actually where

t-shirt sizing came from you guys have all heard the the phrase right it just means well since you can't really tell us is it small medium large or extra-large we'll just compromise on that anyway it's it's under the category I call corporate silliness alright so a lot of big companies do stuff like this it's usually driven by some CFO that thinks they're they're bringing a level of sort of rationality to the process but to be honest in the terms of pick your battles don't worry about it because that's not the real problem anyway because no

matter what if you're doing a road map you've got to have to have your items in some sort of priority this is probably just as good a way of prioritizing as any other the real issue is this road map road maps and they are number one on my list of fundamental reasons why teams fail now again they come from you know two pretty reasonable requests they one of the leaders of the company want to know what's are you working on the things that are most valuable or at least that they think is the most valuable

and then they want some level of predictability in the business here's the problem and I've long referred to this as the two inconvenient truths about product and I honestly wish this was not the case but I promise you this is the case the first invasion Inconvenient Truth about product is that at least half the items on your roadmap are never going to work with customers now lots of possible reasons for that and of course what's hard about this is I can't tell you which items on the roadmap loan we don't know until we kind of

go through this long process but the in this way of working we don't know but most common reasons are we are all excited about it and then we push it live and our customers aren't so excited you're over and you you're you think you've got a great feature for drivers and you push it live in the app and the drivers could care less all you know is they're not using it sometimes they try to use it but then it's so complicated and maybe they're driving along and they don't really actually want to go read a

user manual for how to do this thing and so they don't use it they choose not to use it it's too complicated for them so it's the same problem sometimes actually never even gets that far because what happens is yeah early on them the developers were sort of pressed they said you know shouldn't be more than a month to build and then we get down into building and the engineers actually get into this and they realize okay this is a much more complicated thing than we thought you know which is really by the way not

their fault again at this high level in this process they were given almost nothing and if they just come back and say everything's extra large then nobody really goes along with that so they might have said it was medium and they learn out later that it was probably that with maybe it was large and now instead of being a month it's closer to three months and you say or your executives say no way is this justified three months forget it well we've got plenty of other things we can do so that's the first inconvenient truth

that at least half the ideas on our roadmap are not going to deliver the value we had hoped the second inconvenient truth which is sort of doubly depressing here is that it's now for the half that can that could really have the potential to work those are going to take several iterations usually three four five iterations before it works well enough that it makes money well enough that it makes money and that is that is frustrating in a lot of companies because this is the process and look at how long it takes just to cycle

through once if you need to be able to do that three or four times I mean in a startup they usually at a time and and money one or the other they run out of runway so is sort of that's their big issue in a larger company they don't usually run out of money or time they run out of management patience so then they say look guys we've talked about this six months ago you've already done you know you finally got it out people aren't using it yeah and so many product teams complain to me

that it's you know they got one shot it's something one shot and if they don't get it right you know and they're on one hand their pressure to treat it like an MVP get out real fast on the other hand it's like you've got one shot and so yeah and this is how long it takes so we have a so much orphaned software now some of that orphaned software was never going to be any good and some of that orphaned software could have been good if they would have iterated we say what matters is not

time to market in our business it's time to money and and we don't always charge for everything it may just be a new feature that people aren't using but the point is time until its value till it delivers that value so those are the two inconvenient truths about product the first one we don't know at least half are not gonna work as we had hoped with our customers and the other is that it's going to take several iterations now because of that roadmaps are a serious problem at Amazon Jeff Bezos has one of my favorite

quotes which is be stubborn on your vision but flexible on your details roadmaps are the details not the vision and the problem with a room to be honest the real problem with a roadmap if all it was was a list of ideas that you happen to think are in priority order fine that's not the issue but that's not what a roadmap isn't just about every company in almost every company if you put anything on a document labeled with the string roadmap it's a commitment that's the issue and if it's a commitment we're kind of locked

in to delivering on these things even though we know most of these are not going to pan out this is why roadmaps really so fundamentally at the heart of serious problems or product failed now we have alternatives to all of these things meanness talked just now she hinted at the mean alternative to roadmap so I'll go through that in a bit but I want you to understand I want you to realize why these roadmaps are such a problem and in in most of your companies at least those of you more than say a hundred people

in your company it's really compounded because you've got all these different business owners all kind of fighting over for their share of the roadmap and it's a big negotiated mess this is what's happening ok but let's keep going because that's just three fundamental problems how about you guys you know the role of product manager has changed dramatically well I should say it's in the process still of changing one of the reasons it's hindering that is that the people literally you guys are ahead of your organization's so you can have a product person that's extremely strong

knows what to do but they're in an organization working this way it really hamstrings the whole movement but let's talk about it in this model the role of products what is it I remember I intentionally use the phrase gather requirements this is how literally the job is described in many companies product manager gathers requirements from customers from stakeholders from different parts of the company and documents them for engineering now thank goodness we don't do a lot of PR DS anymore or functional specs but basically it just moved to JIRA moved to use cases and move

to use with what did I say they're just some form of annotations on your stories and that's that's not the job that is not the job of product in fact I would argue this role in this process is pretty much the antithesis of the modern role of product so you're just kind of we're often just backed into that you know one of the common problems we suffer in this in the product management community is so many product managers tell me they're essentially project managers they are like the vast majority of the week is doing project

management and of course it's kind of what this process is asking for that's what it's you're sort of shepherding something through in fact if it wasn't a product company that would literally be called project manager doing these things they would gather up the requirements and make sure they're documented and getting in and move it through the process of design and build and test so just suffice it to say here this is not the role of product this is not what we you know we heard about today was describing a very different role for product and

by the way for design which is the next topic but in this process of way of working it's very difficult to actually apply the things we've been talking about today all right so that's number four number five is this issue the role of design I mean we heard today from different speakers that coming from the design community especially and unfortunately in this process which again I'm arguing the vast majority of companies are using it's considered the lipstick on the pig model of design right at this point it's just trying to you know it's not a

very first of all it's not a very fun job being a designer in this model so those of you that work with a designer it might be a designer it's tough because you're essentially you're starting to work on something where a lot of bad decisions have already been made they really are Frank the product managers tell me the same thing in this model they say okay what am i another one of my favorite quotes this is from John doar he's he's for those that don't know he's a major venture capitalist he was the earliest VC

and behind Netscape and he likes to say we need teams of missionaries not teams of mercenaries this is really fundamental because this process is a mercenary process which is why I was just about to say so many product managers come to me and say yeah this roadmap it's ridiculous these are not the right things I know they're not the right things but it's like above my paygrade the executives went off in a room or they went to a retreat once they do once a corner and they come back and like here's the roadmap aren't you

happy you know like and they're like you know what it really is we gave 25% to this business unit and 20% to this business unit and it's just a joke so the product managers know it's ridiculous they still have to kind of write the stories for it the designers know it's silly they still have to kind of do the designs and of course the engineers at this point they're so cynical they're like you guys have no clue what you're doing we had the right a course about this but it's it really is a mercenary process

and that is not what we want I could go into more another little maybe this is a little digression I'm only sharing this because so much of the content today was actually about design more than product and one of the reasons I'm sharing it and that's my I mean my one of my big passions is design I love that but I will say in this process because product managers aren't really set up to do the job of product management what do they do they play interaction designer they create wireframes I wonder I won't embarrass people

how many of you create wireframes and hand them to your designer but I bet way too many of you would raise your hand and that is of course now you imagine your designer and your product manager is giving you wireframes at this point you're probably to be honest and that set up you're probably a visual designer because an interaction designer is probably not going to want to work there but and that of course leads to other problems so let me just say if you're spending your time doing wireframes well you might be a truly gifted

sort of unicorn product manager interaction designer and I love that and contact me afterwards because I know so many places that would love to hire but probably not you probably are no better at interaction design as your engineers would be and so you probably have any and if we look at your product it's probably clear because that's very very easy to tell when it's not a trained interaction designer so anyway that's a more of a digression so that's the role a design is a problem this is not why we hire our own UX team to

be honest if this is the way your company insists on working you might as well save the money and just use visual design agency might as well because the reason we bring it in-house is so we don't work this way all right how about now the role of engineering I've already shared some of the key points here but because the engineers are not meant we we can't go along with just sort of keeping them as mercenaries sort of in the basement there to code we say if you're just using your developers to code you're only

getting about half their value this is a very big deal now it doesn't mean every engineer wants to spend time face to face with customers and engage with the product issues but some them better or you're in big trouble so we need some of them at least to engage heavily now we can get in there there's a whole there's whole other issues around how test automation is handled but the in terms of a higher order issue the problem here is developers are coming way too late and they aren't so they're not they don't even know

really the reasons they don't have that understanding of the full context and we are not getting the benefit of their understanding of the technology and what's possible not now let's talk about the role of agile because I if I had asked how many of you in the audience were in an agile team my bet is almost every hand would have come up and while sort of that's true this what I've just described this is not agile I mean this is a little this is a July in delivery a bit but to be honest in terms

of agile this is like maybe a 10% of the value of agile and yet this is the vast majority of what agile so-called agile organizations are actually doing and to me that's a huge shame because agile is way more important than that it should be used much more we can't well again that that Amazon quotes stubborn on vision but flexible on details this process is not at all flexible on those details we need it to be flexible so this is not what we in a product organization mean by agile all right totally different point but

a huge one and actually Mina was hinting at this in her talk or alluding to this this is a project centric process that's actually where it came from that in in those big bureaucratic companies especially in the old sort of IT world where they with and projects they would fund and staff projects literally we'd say we are gonna do an Android version of this and you have to make a business case and then you round up some engineers and designers and a product and you go through this whole thing and you launch it that's output

it's not outcome so then what we want and frankly it's not hard to get output we know how to get output what's hard is to get results this is the outcome point so for I forget the exact example that mean it was giving I think it was in an uber context but for example if our job is let's say today it takes 30 days to take a new customer and get them live the onboarding process many of your companies have some flavor of onboarding process let's say it takes 30 days and that's not a problem

really for an early start up where you can handhold a small number but let's say you know you're - that's not a good long-term thing and maybe that needs to get down to 30 minutes which is totally not I mean if you automate a process like that it's gonna be on the order of 30 minutes let's say so your job in the product organization then your job is to go from 30 days to 30 minutes that might be well it's almost certainly not one release of one feature it's probably a whole set of things we

are asked to solve that problem get onboarding down to 30 minutes that's our job we don't with if we're a modern team using things like continuous discovery and continuous deployment which I'm going to talk about then you know that might be 50 iterations to do that over the course of maybe three weeks but that's what matters we don't get points for output we get points for outcome and that's a huge problem with this process it's all about getting these projects launched that's why it leads to these orphaned efforts another huge issue and this has always

been really the big nail in the coffin for waterfall process the valid the actual feedback from customers happens at the very end it's way too late all the money all the time you know one of the things that I just can't even keep with a straight face anymore I mean a lot of startups and almost all of them love to brag to me about how they're following Lean Startup practices and then I'd say okay that's great but let's start you show me how you work they basically draw this and I'm like you know this is

pretty much the opposite of Lean Startup this is I mean this is lean startups all about what's the fastest cheapest way to validate our ideas this is I this is the slowest most expensive way we know how to validate our ideas so how is this Lean Startup I know idea this this is know nothing about that it's way too little way too late you never have to do this you can always do much better than this all right thinking the times going too fast the biggest issue of all though is this notion of opportunity cost

because what's going on here is that this way of working takes a lot of time a lot of money and in in most companies you can't get that back it's gone and opportunity cost really speaks to what you could have been doing instead of messing around like this what you should have been doing you know I there's a whole other dimension to this discussion that I haven't brought up you know I don't want to use the word bubble I don't want to jinx us really cuz it's been a pretty good run most of you would

probably agree but let's just agree it's frauding right it is a little wild right now but one of the problems big I mean this is not a little problem with almost every team I work with cannot hire enough people they literally can't staff all their teams they have more money than job candidates this is why of course you're seeing the changes to the salaries it's it's and so it's really created it's a royal pain right now is you know there are some of the big players in this industry are just paying outrageous prices to get

people and for those of you that are not Google or Facebook it makes it really hard to staff your teams so I mention all that you not that this is news to any of you if you know this but I mentioned that because it's caused a lot of teams to have to take a hard look at how can we how can we do more with less people and the first thing I tell them will stop working like this because this is costing you a huge amount of wasted opportunity okay so those that I would argue

those are the ten biggest problems with that way of working now I just have a few minutes but I want to give you a taste so the best teams first of all the best teams do not work anything like what I just described and and maybe you can come up with a better explanation that I can about why it's so different but it is different and what I want to do is give you a taste of what that means now there's a lot of ways I can mention there's I've written lots of articles and blog

posts s of others about how this other way of working but I want to give you a taste of what's important so first of all there are still ideas but now we have many more sources of ideas starting with developers and the idea let me just show more of this here the idea it goes by different names I refer to as continuous discovery and continuous delivery but the idea here is that discovery is really what all product people do it's what Lean Startup techniques are about it's about what customer development techniques are about it's how

do we come up with good - bill that's if you're in this room that's what you're responsible for coming up with good stuff to build discovery though says look building it QA launching it that is the worst way to do this way too slow is too expensive we have to do better than that I absolutely love Tom cheese talked this morning if any of you missed that talk you've got to watch it online his talk because that's an example he was describing the techniques that he uses for product discovery and and by the way if

you didn't catch this he was describing for Hardware saw hardware software kinds of products which are even harder to do discovery on than software only which is most of us and but if the idea is we use those techniques to quickly separate the good ideas from the bad and to quickly iterate to a good solution and then we deliver deliver means building production quality software so you heard that theme come out earlier today - between where one discussion was around how do we figure out the right product and the other was about scale that's the

same concept just sort of showing a little different way now just in terms of language there are so many ideas of course and I will tell you good teams try out many more ideas than than most teams in other words if you're a typical team this completely depends on the size of your organization but let's say you're doing a hundred ideas a quarter on on a roadmap and getting those launched a quarter you should be doing on the order of 500 ideas and discovery and still only delivering that hundred but now you're delivering the hundred

where you have evidence it's going to work okay so we use vision and we use one system is OK our system objective and key results this is the ok our system is really the alternative to roadmaps now it's not the only art alternative it is my favorite alternative it's what Google and Facebook and LinkedIn have kind of run their business on for a decade but that's if you want to know well what's the alternative to roadmaps start there and anyway we use the vision which says of course where we're trying to go over the next

several years the okrs which are telling us what to work on this quarter what problems to solve not the solution and then our job product teams and I should say discovery is product managers designers UX designers that's interaction visual and engineers we're working side by side in a truly collaborative fashion to come up with good solutions to these problems and once we do once we have that evidence now let me just make a couple other points the main way we do that is we run these MVP tests this is really what an MVP is a

lot of people go sideways on this because they take MVP too literally it's not a product a product is something you can scale and run a business on this is an experiment it's a test so we run these MVP tests and discovery and the idea is to get the product market fit again that's another those are the two most important concepts in product MVP testing and product market fit now when we're doing discovery I hope it's ok Martin I'm gonna go a few seconds over but when we're in product discovery we there's really four well

we need to first of all make sure whatever we're doing is valuable if it's not valuable it's not useable to anybody it's not important for anybody it also needs to be usable so valuable is like would they buy it or would they choose to use it usable is could they use it feasible which is really two things feasible is can we build it and can our stakeholders support it these are the three key areas we're trying to answer in Discovery I'm gonna take the opportunity because it's a rare moment to be able to get in

front of you guys this group and I'm gonna say this and I hope I don't come across as sanctimonious I really don't want to but I will tell you something I care a lot about and with the teams that I work with the truth is and you heard a little people were alluding to this through the day a lot of teams now are quite good and they can well you heard this notion of we can actually create a pretty addictive solution application mobile app service and so the point I'd like to raise is I'd like

to see I'm trying to encourage more teams to add to add one more consideration to the discussion which is really should we build it it's really the ethical question now I don't want to sound like I'm preaching and I know that there are there are definitely a lot of CEOs in this valley that have absolutely zero interest in that fourth question but I think it's important and the truth is I think this group most everybody I know in product is truly passionate about products and so you do care about this you really are trying to

change the world in a good way and I am suggesting that in Discovery we also consider ethics that's maybe a whole other discussion okay but that's maybe a taste good teams are working this way in Discovery they're actually trying between 10 20 iterations per per week okay that's not one iteration every two week which is typical scrum in delivery but this is many iterations a day and each iteration is like Tom Qi was describing it as learning or insights they're all about learning we have quantitative techniques in discovery and qualitative techniques and discovery but it's

all about quickly getting that evidence separating the good ideas from the bad and making sure our developers are spending most of their time building production building the stuff we have evidence is going to move the needle that's the difference all right so I think I probably threw enough at you there I am I know I threw a lot at you there but I just wanted to take advantage of the opportunity I really do hope this was useful I my emails right there anybody who if you have a question after feel free to contact me thank

you very much for your time I appreciate it [Applause] [Music]