let's talk about trade-off questions basically how would you decide between variation a and variation b or let's say we're seeing this product go up and then this product goes down what do we do so trade-off questions are some of the harder execution metrics question you'll get in a product interview and a bunch of you have told me that you're really confused by the all different types of trade-off questions that could be asked and how to tackle them because some of them sound like debugging questions some of them it's not even clear that it's a trade-off
question so today we're going to talk about the three types of trade-off questions that I've observed and tell you how to tackle each one differently so there are three different types of trade-off questions so the three we're going to cover today number one is similar products cannibalizing each other the second is same product different variations and number three is different products but same surface so let's start off with similar products cannibalizing each other and this one causes the most confusion because people think it's a debugging question because it sounds something like oh we're seeing Marketplace
messages go up and then shops transactions go down what do we do how do we figure it out and people like oh my gosh okay well let me think what is the root cause analysis I mean there's a portion of that in there but the main focus is for you to figure out like what is the trade-off here and are we okay with it other example questions I covered in another video which I'll link above is comments and reactions basically we're seeing comments go down but reactions go up after the reactions launch like is this
okay what do we do and others are we're seeing Facebook live videos come out versus you see videos and YouTube videos maybe are getting less activity what do we do so how do you identify them well usually they are products that are similar in the same product category but different products next how do you tackle them the first key part is understand for this product category what are the unifying metrics or common denominators that this product category is trying to optimize for so for example for a Marketplace versus shops the product category they're in is
Commerce and hence the probably organizationally in the Commerce org all under the same org and what they're trying to optimize for is probably like the number of transactions gmv the number of buyers so those are the metrics that you want to identify and bring up top to help you do the comparison because these unifying metrics or common denominator is going to be what helps you compare apples to apples versus orange just apples so we gave the example of marketplace and shops but a couple other examples I'll throw at you if you're talking about comments and
reactions again the common thing that they're trying to achieve here is basically engagement on posts if we're talking about Facebook live versus new TV video probably things like video views or video engagement so look for the common unifying metric that both products are trying to optimize within this one product category the second part of what to do here is to figure out the hypothesis of why we think this is happening so for example people if we talked about Marketplace messages going down and shops are going up shops transactions are going up like why do we
think people are flocking from from Marketplace to Facebook shops because by unpacking the why it helps us understand strategically like this Behavior do we want to see this type of behavior on the platform a common mistake that I see here is people listing out metrics separately for one product and another product and trying to compare them which is going to create a lot of confusion for you because then you're going to be comparing apples and oranges and that's a thing that you want to avoid and lastly for this type of question there is no a
b testing to be done oftentimes people will tackle any trade-off question and they'll throw in an A B test there and it just like doesn't make sense and it makes me realize they were listening to the question or just applying a common framework for any trade-off question so here there is no a b test because they're ready to telling you the results of a launch the second type of trade-off question is when you have the same product but different variations so questions like we covered again I'll link it above is for Instagram stories you have
24 or 48 hours which one would you prioritize or for video ads would you place them in the beginning of the ad or in the middle of the ad so how do you identify these type of questions well one they're going to be talking about one single product and they're going to be giving you different ways that the product can be shaped to be experienced differently so it's mostly a product and they're going to be talking about features that are different and what do you do here so this is one of the most basic type
of trade-off questions that we go through every day when we build products and thinking through like product design and here the key thing you want to do is understand what are the key metrics for the product you're talking about so because you have one product here it's easier to assess for variation a versus variation B how it's going to affect the key metric for that product so in this case if we're talking about Instagram stories that is the product and we're trying to figure out for Instagram stories like the key metrics would be the number
of stories viewed or the number of stories that are produced things like that and I would try to figure out for 24 hours how does affect those metrics and 48 Hours what are their hypotheses on how it affects those metrics and does this require an A B test yes this is a great opportunity to show your experimentation chops which they're going to be evaluating because in a lot of these cases your evaluation is going to help you realize there's not such a straightforward answer that sometimes the 24 hours might actually lead your main metrics to
go up or down and it's not obvious if you just kind of theoretically think through it but sometimes you just need to put it out into the wild and see how people are actually reacting to it and what the total sum of all this little little actions are leading to the third type of question is when we talk about different products but the same surface so an example is let's say you are looking at Facebook news feed and you're trying to decide should I show an ad or people you might know or maybe for notifications
should I show a fundraiser from a friend or an event from a friend so how do you identify it usually when you're talking about the two different products they're living on a common surface so figure out what that common surface is a lot of times the interviewer will be explicit about it so you don't have to guess and what do you do in these instances so the key thing here after you identify what is the common surface is to figure out for that surface what are the key metrics they're trying to optimize for so for
example on news feed things are going to care about is engagement with posts interactions on the news feed through comments likes Etc and then you're asking yourself and even for news feed like revenue is going to be a key metric there because news feed is probably one of the big Revenue generators for Facebook so from there you're then assessing for these different products in this case people you might know versus ads how is it going to affect these list of metrics for this common surface this again helps you compare apples to apples versus oranges to
Apples a common mistake people make here similar to the first one above is people will come up with metrics for people you might know and then come up with metrics for ads and then they try to compare apples to oranges which again creates confusion because you're not really assessing like what is the trade-off on the common thing I mean a lot of people for that question is like oh you're trading off as revenue for then like friend Connection and they're like yes that is the trade-off off but how do you put it on a common
comparison so you know how to Value them equally and here would you run a Navy test maybe I mean for the ones we talked about it's like fairly straightforward to run an AP test that either show people people you might know unit or an ads unit to find out like what to do you have these three different type of questions but they're a common set of things three things actually that are going to help you tackle all these different type of questions if you keep them in mind and it's pretty simple the first one always
understand the product again if you have two different products talk high level about what the different products do then the second thing is find that unifying metric what are they trying to optimize for if it's a product and two variations or two different products on the same surface or under the same product category and then lastly is your hypotheses of the different variations or different products on your key metrics so to successfully answer these trade-off questions there are a couple of other things that you can't miss that I didn't cover in detail here so check
out these two other videos where I actually use examples to cover how to tackle trade-off questions of these different three variations and I will see you guys in the next video
