how you determine success for instagram reels [Music] hey everyone i'm kevin way i'm a product manager at ibm data and ai and today we have a mock pm interview with dan and before we jump into today's question dan can you tell us a little bit about yourself yeah for sure so my name is denzel i'm a product manager at home base home base is an app that helps hourly employees be able to do scheduling and time sheets so it's a really cool app if you ever had a job at a coffee shop or a cafe
you know how painful it is to do your schedule to be a manager homebase is an app that helps with that and i'm primarily working on the hiring product and some of the hr products that home base offers awesome well we're super excited to have you and today we're doing an execution type question so for this question i'd like you to tell me how you determine success for instagram reels okay cool uh do you mind if i just take a couple minutes just to kind of jump down my thoughts and then we can go for
it yeah go ahead all right so before i get started i just want to clarify what instagram's reals is and uh just to make sure that i'm understanding the problem uh correctly so instagram reels as i understand it is a product where uh creators can create these short videos um and then they can share it with their friends and to the broader network and you can just watch these videos as part of the instagram app as a consumer is that is that accurate am i missing something about this product yeah that's pretty accurate and it
came through us looking at tick tock as a bigger and bigger competitor got it okay yeah so the way i would think about this is um i'll start by thinking about uh the facebook mission so i want to start there and think about how instagram reels fits into the broader facebook mission second part of it i'll probably want to go into the uh the face of the product so focusing on whether we want to uh grow the product or focus on things like monetization the next part i want to talk about is the actual metrics
themselves i want to get into how do we want to measure the success of this and then lastly i'll follow up with the goals so setting up the goals for this app how do we wanna think about um growing this app in the future yeah that framework sounds good sounds good okay so the first thing i wanna do is start off with the facebook mission um you know facebook is a place where um you know we want to build tools to connect people from all over the world i think that instagram reels personally is one
of my favorite apps within the instagram ecosystem and the reason is that it just allows people to share their creative work right it's it's a tool that really lets people create these awesomely entertaining videos and i'm seeing videos that not just from my friends and family but from creators that i wouldn't otherwise be introduced to outside of this product so i think it really fits into the facebook mission because it is connecting people and communities uh in in ways that wasn't possible before so in in one line how would you describe instagram rio's mission if
if we're gonna say facebook's mission is to connect people and build this community what would our one liner be for reals yeah i think the wine liner would be uh instagram reels is a set of tools that help content creators be able to reach out to their broader community and connect with people that's not just friends and family okay and it's it's more for short form video just so we're on the same page right right got it all right so the next part i want to think about is what is the phase of instagram reels
um at this at this point so my understanding of instagram reels is that it's this is a fairly new product and the reason why the face of the product matters is i'm thinking about what is the most important part to making instagram real successful um if it's a new product things that i want might want to focus on are adoption and engagement so i'm going to make sure that users are knows about the product and knows like how to use the different features within within the app um if it's a more mature product i might
focus on things like retention and monetization so because i know that instagram reels is a fairly new app um for me when i'm setting goals and thinking about metrics i'm not going to be focusing on things like monetization rather i want to think about how to go the app in terms of engagement and and retention does that make sense uh yeah yeah so let's focus on engagement slash retention yeah okay okay now let me go through and sort of think about um what this product is and what type of users might we want to measure
uh what what to think about when we come up with metrics so with something like instagram reels there's really two sides to the equation there is the creators and there's also the viewers right so when we think about metrics we kind of have to address both sides um and so i would like to take a couple minutes just to think about both the creators and the viewers and then i'll come up with a framework for the metrics that we want to talk about for each okay all right um so i'm going to first talk about
the viewer side um so this is the consumer so people who are actually watching the reveals um so the three uh set of metrics i want to focus on are adoption engagement and retention so for adoption when i think about viewers um i want to track a pretty high level adoption metric here and what this adoption metric will tell us is basically how many people are uh basically discovering this feature um instagram reels is a feature within the whole instagram platform so we want to know just of all active instagram users how many of them
are actually discovering wheels so i want to keep it pretty high level here um and then we can kind of go into more of the details so for adoption one of the things i want to measure is the number of uh users who are actually doing reels and so what we expect to see with this metric is that this metric is something that will grow over time and i know that this is a very high level metric but then we can kind of break down this metric further to understand what type of users are adopting
what are the type of users that are not adopting but i think uh it's very important to keep that very high level metric in mind when we think about goals so number of users who are viewing rios so does this mean if i view reals today but i never view it again for 30 days does that count or how are we defining these people who are viewing reals yeah i think for adoption um i just want to keep it very high level so if you view reels even once we'll count as someone who's discovered that
feature and when we then look at retention we can then break out to see if you actively retain on this feature okay so this is like a cumulative metric yeah exactly great um then i want to talk about engagement so engagement is then like how um so you might have viewed uh reels once but what is your engagement with that um so for that there's two metrics that i want to use to measure engagement the first is number of reels viewed per user per day so that's a metric that i want to understand in terms
of like the intensity of the engagement right so if you're someone who just viewed one reel you might have adopt you might have had some uh awareness and adoption that feature but you're not using that very intensely so i want to sort of understand the intensity of usage there another metric that i want to understand with engagement is the number of minutes that a user spends uh viewing reels this is a really important metric for us because uh i want to understand how people are spending time in the app once we track start tracking this
metric we can also compare the number of times spent uh doing other stuff on instagram or on facebook versus on reels and sort of understand how people are splitting their time um on the facebook platforms and lastly for viewers uh here is where i want to understand the retention so for this one i might track there's two ways i'm thinking about it i'm not sure exactly how i want to track it the first way is just using wa use so weekly active users for the viewers and so this gets back to your question which is
if someone has viewed it once does that mean that they're retained so i want to track this on a weekly basis so if someone viewed reels on monday and then they viewed rios again next monday that will be that will be counted as a weekly active user uh for this feature cool cool so back to in the beginning we talked about how we wanted to focus on engagement and retention so that rios can really be the place where people come to share their creative work that's in short form video and so so you've given me
some pretty good metrics here and i'm curious if if you have a north star that you would pick for instagram reels yeah that's a good question i think if i had to pick one northstar metric it is the amount of time that people spent on the app watching reels and the reason why i think this metrics is important because it really levels up to the facebook mission and um about the broader company goal which is to um one uh help people stay connected um and also i think the amount of time that people spend on
the app is an important metric for us to measure not just for this feature alone but just think about um does does time spent on reels cannibalize other parts of the facebook product um so one thing that we could look at right with this metric is if people are spending let's say two hours on average on reels does that mean that they're spending less time on facebook or in other parts of the product um that's that's a really important uh factor for us to understand to determine the success of this feature yeah that's really interesting
and i like that point that you brought up where instagram rios could potentially cannibalize not even just other parts of facebook even parts within instagram maybe people are viewing fewer stories as a result or maybe people are scrolling um at a lesser depth on the the home page um so i'm curious if you were a pm at facebook working on reels what would you do if rios was launched is massively successful because people are using it tick tock is also massively successful so there's a lot more minutes um being spent on here but people are
viewing let's say uh stories the the minutes spent on stories is going down so rios is up stories is down what would you do i think the first thing i want to understand here is um how are the two products different and how do they serve different types of audiences um to me from my understanding of the the products here stories is all about connecting with your close friends and family right so that is a really important function of stories um if the reels i think it's more about the broader community like i'm seeing a
lot of reels from people that i don't know so it's outside of my own friend group so if you think about the um i think this is where we have to weigh the different um types of uh values right that people want um i think this is something that i would want to look into the data and understand a little bit more so i think there's definitely a set of users who definitely favor the friend and family content more and some people who maybe don't have too that many close friends on instagram who might actually
prefer reels so this might be something where i would look at existing data it exists to try to understand these two different groups of people and understanding like what the impact of that is i would i could also set up an experiment um i can think of an experiment where we expose different user groups to more reals content versus stories content and see what that impact is to long-term retention of the whole instagram platform and engagement the platform themselves um like to your point we could see this impact of if people spent too much time
on reels over the long term they lose interest and connection with their close friends and family that results in like poor engagement of the instagram platform that will be a negative impact right of the whole facebook ecosystem so i really want to understand that relationship before making a decision about that these trade-offs cool yeah that's a pretty good point since the mission of instagram is to bring us closer to our friends in our community and there's a chance that if reels takes off and story views go down there's a chance that we will stray a
little bit away from that mission um so i guess it's pretty interesting that you said you might run an experiment on this and figuring out what sort of or i guess i can let you tell me what sort of experiment you would run but you can maybe here now you can pretend that i'm a data scientist on the team and i'm curious what sort of questions you might ask me to get the kind of data you need yeah so one of the questions that i might ask is um [Music] i i wouldn't understand like the
distribution of uh the i guess the broader level question i want to answer would be um what is the distribution of time that people spend on stories versus reels today and whether that kind of breaks out in different types of demographics um so one thing i want to understand would be like let's say we have people between like 18 and 35 how are they spending time on their app and i also want to understand uh like like i think backing out from that point i want to understand um the relationship between the number of close
friends that you have on instagram and the amount of time that you spent between stories and instagram reels it makes sense yeah so it sounds like you you want to cohort these users into if they spend time with their close friends or not and if they don't then there's a chance that we could expose them to reals more otherwise we might want to keep them on stories since that's and that would that would be a better investment for them right so i guess the way i think about it is um i maybe will break it
up into three buckets of users the first bucket will have a few um i'll call it like few um close friends on instagram medium number of friends on instagram high number of friends on instagram right um from these three buckets i want to look at what is the percentage of time spent on stories versus reals and then i want to look at how that impacts the long-term engagement and retention within the app interesting cool cool um all right this is great so i i guess um i want to just backtrack back to the original question
which is the success of reels so you you've given me the north star which is the minutes you've given me some great um additional signals that we should look at and i'm curious if there's anything else you wanted to add here yeah so i think the one thing that i do want to understand and talk about is the creator side of things so we've only talked about it from the consumer side but we have to remember that in order for realtor exists there has to be a creator side of things um i think that um
whether that's something that we want to prioritize really depends on uh basically like the inventory of reels that exists within instagram today the way i think about this is are people finding content that they like or not if we see that um you know people are really engaged with reels but after maybe 10 minutes of browsing they really like there's no more interesting content left we might want to invest in the creator side of the equation which means building more tools maybe simplifying the experience so that more people are than creating reels because this is
a two-sided marketplace right you can't have just people consuming without optimizing for the creator side as well yeah so maybe just real quick what's one northstar metric you would choose to make sure that we're having and maybe not even just an abundance but maybe just enough content there that would be useful so that rios is a useful product yeah i think for that i want to think i would the way i would think about it it would be tracking the number of reels that people consume per session um and the reason why i think that's
a great metric is that i think people abandon the app for many reasons but one of the reasons is the lack of content so that's something that we could look at on a cohort level so what i would imagine would happen is that when reels first launches the number of reels that people watch per session is probably quite low because there's just not enough content out there they could be pretty relevant to what you're interested in but over time i expect that later cohorts would have more reels watch per session because there's a deeper inventory
of content and more engaging content as well yeah that's um that's that's actually a pretty pretty good way to track not only creators but also viewers since if the if the the number goes down if not enough uh creators are creating content it sounds like we would expect this number to go down as well and that'd be a bad signal right exactly that makes sense cool um all right awesome well i think that that wraps up for this interview um unless there's anything else you wanted to add uh no that's uh that's all i have
cool okay so taking off my interviewer hat um we can just talk now so thanks for that awesome uh deep dive i know we street a little bit into another experiment that we talked about but you know that happens in interviews sometimes where you might say something that peaks the interviewers interest um it's always good to have some some sort of conversation there where it helps the interviewer see what it would be like to be a co-worker of yours and so before i give my feedback looking back would love to hear your self-reflection on how
you thought this mock interview went yeah i think these metric questions are always pretty difficult because it involves you understanding the business a little bit right from an outsider's perspective so i think i could have done a better job setting up the initial problem i think i could talk a little bit more about the goals we're trying to achieve um i think tying it back to the facebook mission was good but um it would be good to mention sort of maybe the business objectives or the um like what what are we really like what is
the broader goal of optimizing for instagram reels that way when i talk about metrics like to tie it back to say like the engagement piece is the most important piece that should be the north star because as a business we want to obviously we sell ads right and so we want that engagement on our app in order to hit that goal or something like that i think leveling up to a business goal when we talk about the metrics could be could be something that's helpful absolutely yeah you you also did a lot of things in
the interview very well so i like how your framework was laid out in the beginning how you laid out how you talk about the mission and then the different types of goals you might consider and then talked about metrics and i really liked how i remember there was a point in the interview where you connected it back to the mission and that was really good i think if we did it just one more time at the end and really hit it home that that that would have been even better i also liked how we talked
about the creator side as well as the viewer side we talked about how we have to consider both parts of the ecosystem one thing that i would have done is in the beginning i would talk more about tick tock since that's really the driving force as to why instagram created reels it's kind of like how if we thought thought about how when instagram first came out with stories i think a an answer to that would kind of be incomplete if we didn't really mention why instagram made stories so um if and also i think that
if uh if you were given this question and if you had any specific insights about gen z that would be also interesting to bring up since they're a core user group of tick tock and they might you know i'm i'm not sure but they might be a core target segment that we want to target for reels yeah so that's that's a really great point yeah i think talking about the core demographic here and whether we're really addressing the needs of the core demo i think that's really important right i think the metrics should at least
break down into such a way where we can understand how different type of users are engaging with this feature yeah and i also liked how you were able to pause throughout the interview to allow me to jump in and ask questions and ask follow-up questions so that really helped me find places to come in and also i like how you uh you remain calm even when we kind of strayed down another just tangential path about talking about experiments that we would run and i think overall this is a pretty good mock pretty good mock um
is there anything else you wanted to add or any tips that you would give viewers yeah i mean i think um it's important to i always i think for me the most helpful thing is actually putting myself in the shoes of a pm actually working on this product i think when you mentioned that oh i was uh you know anything i could answer uh as a like a data scientist on your team i think that really helps me because then i can think about in my everyday work how would i actually interface with the data
scientists like what questions would i want to ask um that really helped me frame um these responses so i think from the very early on i should have gotten into that mindset more that would have made this interview more that's like an interview and more like just working with a colleague absolutely well thanks again for being on today's show it was super valuable and for the viewers if you have a different approach to this question leave a note in the comments would love to hear what you would have to say thanks for tuning in and
good luck on your upcoming pm interview
