Curriculum - How would you set metrics for Facebook Marketplace?

how would you set metrics for facebook marketplace hey everyone i'm super excited to be here today with neri to do a product management mock interview in today's interview we're going to be focusing on a question related to facebook marketplace but before we get into the pmock interview and eric could you share a little bit more about yourself yeah definitely thanks for having me steven i'm just steven and i know each other as classmates at stanford gsb um before the gsb i worked in consulting and this past summer i just finished a product management internship at

apple um and after graduation i'll be joining facebook as a product manager and the exponent program was incredibly helpful for me as i was prepping both the videos and the course component itself so um yeah it's great to be back full circle totally um and it's always awesome at exponent we believe strongly and community and uh bringing community to the interview prep space so it's always awesome to have people who've been through the experience come back to help others um so thanks so much neri for taking the time and congrats again on your successes in

getting the facebook offer um well without further ado i'd love to jump into the actual interview question today um and so the interview question i'd love to ask you is how would you set metrics for facebook marketplace so i've used marketplace before as a buyer and seller but just want to make sure that i fully understand the product before i get into cutting metrics and so marketplace is a marketplace it's two-sided there's buyers and sellers and essentially it's a platform for buyers to be able to discover and browse through items and then for sellers like

an everyday person to be able to list items and actually attract buyers and and and sell the items does that have a good understanding of it yep that's right okay and um i think the one of the kind of key components of marketplace is that it's you know facebook has a large commerce platform there's buy sell groups there's now facebook and instagram shops and so i think what makes marketplace different in terms of why this product like this is it allows anyone to be a seller so you don't need to have actually a storefront it's

you know if i'm moving and i want to sell all the furniture in my house i'm able to kind of quickly create a listing and then attract an audience um and then also i think it like ties to your local community so it's like area like geography based um is there anything else that i'm missing that you think would be important to call out for studying metrics that seems like a good list for now and we can always revisit it and come back to it if we need to okay great um so just to kind

of share my thought process of how i'm going to think through this problem first i just want to thank overall why do we have marketplace where does it kind of operate within facebook's broader goals um and then i'll kind of start thinking about the different users and what are the types of things that we should be measuring um and then from there we can kind of talk about what are the metrics we should be prioritizing because obviously there's thousands of things we can measure but there's probably a few metrics that can help drive our team

towards a success so i want to really focus on what those key driving metrics are does that sound like a good approach yeah let's do it okay great so um so first just kind of facebook's mission overall is to give people the power to create community and bring the world closer together and marketplace really does that by i think the technology really kind of leans into that community aspect and it gives people that power with free and accessible tools and technology to to to discover items to buy items and sell um so i think there's

a lot of alignment with with the mission and i think the fact that it's accessible to everyone um is kind of a key component that really stays true to the mission um so i think that the main goal of marketplace is to allow people um to empower buyers and sellers with technology that allows them to uh buy and sell products does that um so i think and and we can obviously hone in what does it actually mean and be a little bit more tactical but as kind of an overarching goal does that kind of sound

like the direction that captures uh what the product is really about totally and i love um for folks watching like an area starting with the mission here which is really really effective and kind of like bubbling down from that and i yeah that sounds great let's let's jump into it we can talk more about um how we'd go about that awesome and then in terms of like the focus of the metrics i really want to focus on engagement i'm just given the maturity of marketplace i think it's been around for i've i've been aware of

it for the last three years but i think it's been around for um at least a few years now and so um i think that right now the goal isn't really about getting more people to use it but it's more about what what are the ways in which they do use it and how are they actually getting value out of it um is there anything else i should keep in mind for the goal um i think that sounds good i'd love to at the end of this conversation have a discussion around trade-offs or how we

think about this goal as it relates to some of our other goals like secondary goals and maybe how they may conflict or how we might navigate that awesome okay great yeah so i will um just be clear focus really on engagement now but then as we talk about trade-offs i'm really gonna tie in other aspects of what might we be missing when we focus only on engagement and on other aspects sounds good okay great so i'm gonna just take um a few minutes to structure my thoughts and just to kind of as i'm structuring it

the main questions i'm going to be thinking about are when our product is wildly successful like what does that actually look like because we want to set metrics that help us drive towards that and then also like how are users actually getting um value through and so i'm going to kind of use those as filters to help me think through what are the main metrics we should be tracking great okay great so um there's i came up with a few metrics i want to just kind of talk through them and there's a few buckets of

the metrics the first are really just kind of those core engagement metrics that would be important to track and um we'll just talk a lot through my thoughts on that and then the second bucket are just kind of thinking about the ecosystem at large um how marketplace kind of plays in with the other commerce initiatives at facebook and then kind of the third is maybe what are some of those quality metrics that we need to look at make sure that we're not providing a bad experience for our buyers and sellers um so i'm going to

start with the first one um and really focus the metrics around the buyers and the sellers these are two main user groups um so the first metric that i am thinking about is some sort of active user metric um and so i want to kind of think about what does active mean and what is like the cadence and so i think that with with buyers and sellers a weekly cadence is um what is probably most helpful to measure i think that there would be a lot of fluctuation if we look at like a daily active

user metric because maybe people shop more on weekends or list items more on weekends um and so i think and i think it's just not commerce is an activity that people do on a daily basis so i think a weekly active user metric um would be really helpful for both buyers and sellers got it um go ahead yeah i think with that the other thing i just want to make sure that we really get right is what is what does active mean and what is like the type of behavior we want to be tracking and

i think for example on the buyer side there's a lot of things we can track such as are they just viewing are they spending 30 seconds on marketplace are they clicking on for post or are they clicking on four posts and then pressing message to the seller and so i really want to work with our data scientists team to understand what are the types of behaviors that really drive uh like long-term engagement um but i think that right now i'm leading towards maybe active is you've put um on at least one post um so you're

viewing marketplace and you click like details on at least one post for the buyer's side so weekly active users got it so to summarize a weekly active user is someone who's clicked on a post at least one post within a week's time span yeah and then i think weekly active sellers as well um and to be an active seller maybe you've actually posted a listing um and yeah with kind of the caveat would want to really flush out if that activity maybe if 90 of people actually do that maybe that's not actually a metric that's

helpful to track and maybe it's the second step of actually pressing message seller um because you're interested in that item totally that makes a lot of sense great um so that's kind of the first area the second one is um just i actually have a few clarifying questions of how marketplace works so i understand that there's a lot of items that are listed if you're interested in it you can message seller do we actually know if the transaction occurs or how do we track that yeah that's a great question um so there are some ways

that we can know that or guess at that information based on message uh history and things like that but we can never know for certain since it's an in-person interaction right yeah so i think that makes it really tricky but when i think about that question of how do users get value it really comes when that transaction happens when the sale happens because i can look through like five or six items per day and you know that might show i'm an active user but if i don't actually prove just that sofa and especially the seller

if they're not actually selling that sofa they don't get value from marketplace um so i do think it would be important to try and understand how do we estimate um those uh those sales or those transactions and so i think a potential proxy metric is you know maybe percentage of or maybe listings that have at least six back and forth messages um where there's at least you know one message on each side and so kind of the way that i imagine it's going is i'm interested in the sofa i message is this still available you

respond yes maybe there's some bargaining of price and then you give me the address and pick up instructions and so i think six is just kind of a number i'll throw out for now but maybe if there's six messages we're really we're highly confident that a sale has actually taken place got it that makes sense um just to push you a little further are there other ways on the seller side that we might be able to know if a sale has been made or other guesses that we can make when we come up with this

model yeah definitely i think um if the seller removes the item um or marks it is sold we have good confidence that it's been sold especially if we ask them has this been sold i think one risk with that is if they're removing items because they might have sold it on craigslist or on ebay or they might just be frustrated that they're not getting enough interest or traction and so i think that maybe having a few of these different ways to track it and maybe benchmark them against each other and see how accurate is our

like prediction and being able to tell whether an item has been on awesome yeah that's a great point well um i think that's the thing so i mean we have a good sense of how many people are on our platform whether the transaction happens but i want to know how concentrated it is is that you know one buyer that's selling thousands of things per day and then every you know every other person is is not selling anything so i'd want to maybe look at like a a distribution of a buyers and sellers to see like

number of purchases per buyer per week a number of sales per seller per week um and potentially even get like a like a p25 to 50 to 90 so understanding you know the bottom 25 percent of users what's that average at the median and then like our super super active buyers and sellers what is their average um and understand that distribution totally that's a great uh thoughtful point like what are you worried about in that case can you tell me a little bit more about why you're looking at the distribution yeah i think what like

the main motivation for this metric is to make sure that we actually have it it is like kind of hits our mission that it's in every a tool for everyday buyers and everyday sellers to use and my concern is that you have you know professional shops maybe using this as like selling like new furniture and then like someone like me who's you know moving out of my house and wants to sell my furniture just i'm kind of like the buyer that really loses in this and i think we really want to make sure that we

have like a pretty like even distribution throughout and i think that shows that our product is hitting our mission and that we are attracting the right types of buyers and sellers and i also think it would help like later on with like cohort analysis to understand where um where our buyers and sellers fall and maybe we can target these buyers and sellers um for different like advertising or different kind of new product releases based off of the cadence of their purchases got it fine awesome i think the other metric that comes to mind for engagement

and i know for a lot of our other products that facebook we use um is is time spent on the platform so like time spent per user um i think that it it kind of shows the value that we are bringing into the product if people are interacting with it a lot and spending a lot of time but i think the potential risk is maybe they're spending a lot of time and they're not actually finding anything so they're like searching for that perfect lamp and they're like spending a lot of time and changing their area

a lot but um can't actually find it so i think it's like an interesting thing to monitor but maybe it's not a metric we want to maximize but just really want to have it so that we are understanding if people are actually using our platform and for how long cool sounds good so i think those are the main engagement ones i think that the other kind of two retention ones that might be important are maybe just like a 28 day retention for sellers and buyers to understand kind of the average time between purchases um for

each listing um and then maybe some sort of like stickiness that also falls under retention so potentially weekly active users over monthly active users for both buyers and sellers to just get a sense that the content is really like evergreen that people kind of keep coming back for more content and they're finding more and more value um i think that would give us a good sense is like is this kind of hitting that one-time need or are we actually cultivating um like repeat behavior from our buyers and sellers got it yeah that's that's really helpful

um well it sounds like we've come to a set of metrics that we feel sort of good about i would love to go back to some of that earlier question around you know we're focusing a lot of engagement retention what are we concerned about or what are the risks here with the metrics that we've picked yeah i think the the risk is is we're focusing on people who are using our product and how they use it but we're not thinking about people who aren't using our product and um my hypothesis is that marketplace might do

very well in certain areas so if you have like a lot of people around like a city like you know geographically dense area um but maybe that there's some regions or some areas that it's not that it's not as popular and so we're not really capturing that with these metrics um so in that case i would think about um some awareness um some acquisition uh metrics um adoption and then i think that's kind of earlier in the funnel and then kind of later in the final because this is a product that is really mature it

might be good to also kind of track some monetization metrics um to make sure that we are kind of utilizing the user base to be able to like you know target really relevant um ads and how that behavior uh works totally um and you know one more follow-up question sort of towards the end of the interview and feel free to add anything that you wanted to share that we may have missed too but why why didn't we include monetization metrics as part of this set like what was your reasoning and not including those yeah i

think um my rationale is that if we really can get engagement right which means that we're showing that not only are users using our product but that they're finding value in it but i think monetization comes after and so if we can really create this like really engaging experience for them then we're able to kind of think about how we bring ads into the mix or how we can potentially maybe charge a fee for the purchases um but if that engagement part is broken then there's no point to get into monetization so that's kind of

the first rationale um for that totally it makes a lot of sense um an area this is awesome like is there anything else we want you want to add here in the interviewee mode before we talk a little bit about how this interview was and feedback and things like that yeah i think just the um the two things i'd want to touch on those are kind of the core metrics but um first is like the ecosystem so you know marketplace plays a role within facebook's overall commerce objectives and so there are shops and um you

know buy sell groups and a lot of different places even through ads where people can buy products and so i want to make sure that you know we're not cannibalizing it farther than um are kind of aligned with the company's overall goal so i want to kind of look at a few like cohort metrics for cannibalization um and then the second thing is i'd also want to track like when it's a super super bad experience and so things like you know the seller is searching something and there's no results found or looking at items that

are like blocked or you know sellers that are reported because of like quality issues or like you know threats or harassment so um i think there's a few buckets that you know if i were the pm for this would kind of spend more time kind of thinking through but um just want to share that that is also as important to measure as it is kind of those good value experiences totally totally um awesome well an area this is so fantastic um it was so great to see your interview skills like to everyone and honestly you

you crushed it um would love to hear your own i wrote a bunch of notes about what i thought you did really effectively but would love to hear your own thoughts like what do you think went well with this interview or any self-reflection about it yeah um cool well thank you i i thought that it went pretty well i think something i've been working on was like less is more sometimes and it's probably more interesting to have a discussion on like five or six metrics than to list out 20 or 30 metrics um so that's

something that i tried to do but would love to get your feedback on how that works versus being more comprehensive and just listing out there's a ton of things you can list out in the funnel um i think one thing i maybe could have done better has been more explicit on like fire versus seller they are the kind of the two users and been more explicit about what are the actions they take and how what are the metrics we should measure for each um and then i think maybe like being more clear about that north

star metric i think the one i would have decided on is that proxy for a number of transactions of like the number of listings that have at least six back and forth messages um and maybe just giving more like rationale why should we be focusing on that i think it like focuses on both the buyer and seller it kind of measures a ton of things but i think just being more clear and explicit with um the rationale for that um rather than just like laying out six or seven would have been helpful totally um those

are all great points let me first jump into what i thought you did really well and i can talk a little bit about what you also just mentioned um first of all i thought you you did an excellent job of checking in with me throughout the interview like i was engaged as an interviewer one of the biggest mistakes people make is they think their interviewer is always paying attention which is unfortunately not always the case and so you did a great job of engaging me and making me a part of it um you as i

mentioned during the interview you brought in the mission really early on this is so important and you know whether it's facebook or any company i recommend everyone read about the mission and watch a keynote from the company before you interview so you can bring that into your answers so that was really really effective i thought um i liked the the discussion on the engagement was really smart like you had a lot of thoughts and you know there were points where you called out i'm going to decide weekly because of this or i'm going to explain

and you kind of explain your thought processes throughout which is really really effective and i thought we came to a set of metrics that did really make sense um the one that felt the hardest to me was the time spent one just because there's like this ongoing trend of facebook you know sucking up all our time versus you know that clearly is a useful metric sometimes but marketplace maybe is a more action-oriented or intent-oriented experience and so it might not be good that people are spending too much time searching for a product or experience um

and you know i if we had a little bit more time i might have pushed you more on that and we might have had a really thoughtful discussion um but i did what i did really like that you did is you called it out during bringing it up you you sort of said this is my metric you know i'm a little unsure about some of this or you know this is maybe something you want to watch out for and it did help me ease my nerves and that's almost why i didn't ask that follow-up question

is because i was like okay an area already seems like she kind of has a sense of what's going on here um i i thought the buyer versus seller comment that you made about yourself would be effective you know segmenting those out and thinking about those this is just such a clear marketplace thing that it's nice to kind of like segment them um i think you know i loved the focus like i definitely don't think you should have mentioned more metrics and i actually liked that you mentioned more but you're like you know i'd focus

on these i think focus for the metrics questions are always so important because it is really easy to just talk about all the metrics in the world you know like you can just keep talking about different cool metrics that you can measure so i really like that i think um if i yeah and i agree generally it does help to pick one north star metric or one or two north star metrics we had four which are a little bit pushing it like um it was good overall and i got a sense from you about which

ones are important but um yeah like for those who are watching north star metrics are kind of like your primary metric or the metric that you would focus on um so overall but overall an area this is awesome you crushed it like this is such a great crisp example of an interview for the metrics based questions which can be very intimidating to a lot of product managers or aspiring product managers because it you have to kind of figure out like how you think about these metrics how you talk about them um and it really helps

to have a lot of practice and it it shows that you did a lot of practice so that's awesome i want to leave some space an area if you had any reactions thoughts comments or any tips for folks watching before we wrap the video awesome yeah thanks steven that was really helpful i do think that um and i think i can be more clear with articulating the risks and preempting it so i think i like touched on it but being more clear with maybe even like i think it's an interesting and engaging conversation to be

like broader do we even want to have a metric that is going to suck up people people's time and what's happening right now in society of people kind of questioning that time i think that would have been an interesting discussion so maybe opening the door for some of those um and i think uh yeah so i mean i think the feedback it all makes a lot of sense um and definitely honing on one north star metric i i highly recommend and and think that that's really important um i think in terms of general tips i

think that there there becomes a point when you're kind of like studying where you like you kind of understand the concepts um like you like you know that there's all these concepts that are available and then when you really understand them and truly understand how they relate and so um what's been really helpful for me is really just like look at any product that you interact with whether it be digital or physical and really think about like what would an awareness metric look like for this what would an acquisition metric look like what are the

engagement metrics um and really when you can take those like textbook concepts that exponent does a really great job of explaining and then bring it to your real world um i think that it starts to like sink in um and then you kind of get this like new level of understanding so that is my awesome everyone yeah well thanks denary for the kind words um folks watching uh definitely leave comments below if you want to have any questions about this interview or for me ordinary we're here to help um and thank you again so much

and area for being on the show um for folks watching good luck on your upcoming interview good luck bye