How to Get Ideas & Measure - Stewart Butterfield of Slack & Adam D'Angelo of Quora - Stanford CS183F

>> Good afternoon. Today we have two different speakers. We have Stewart the founder and CEO of Slack and then we have Adam, the Founder and CEO of Quora. I'm going to interview Stewart. We're going to talk about ideas and how to find good ones, and then Adam is going to talk about metrics why they are important, and what to measure. Stewart, thank you for coming and spending some time with us. Stewart is the founder of Flickr and Slack, and what I didn't know until very recently is that he tried to be the founder twice of

massively multiplayer online games, and they turned into these two iconic companies. So I wanted to talk about just the process of coming up with the idea for the game and then the ideas for Flickr, and Slack. >> So yes, twice we set out trying to build a web-based massively multiplayer game and the origin of that was all the way back in 1992, I arrive at college and I got my account on the school's Unix machine and was exposed to the internet for the first time and it was kind of a mind blowing experience. So, this

is about a year before the web became popular. So it was like IRC, HughesNet, talk, email and I was really fascinated with the idea that people could find their community irrespective of what that community was. So if you were a breast cancer survivor or a model train enthusiast or whatever, if you were interested in some as I was obscure 19th century biologists, then you can find all that stuff. And the idea of a game that lived online like that and had the game like aspects or play as a pretext for social interactions seemed really interesting.

And I'm not going to go into to long of an explanation, because it turns out it's like a terrible, terrible idea at least as a commercial business. Like it might have been fun as a hobby, I think it could have been the kind of business that sustained the number of employees required to operate it. So, like a restaurant or something like that. But when in 2009, we started the company that ended up making Slack, we raised $17.5 million in extra capital and it's just never going to become the kind of business that would justify that

investment. The first time was a little bit different. So, 2002 was after the Dot.com crash. It was after WorldCom and Enron accounting scandals. It was after 9/11 and the NASDAQ was down 80%. S&P 500 was down, 65% from its peak. No one wanted to invest in internet facing stuff. So it was very hard to tell whether this was a good idea or not, because there was no signal from the market that said, this is an investable idea, because nothing was investable at that time. We raised a little bit of friends and family money. We spent

about a year or so building a prototype. The prototype was well-received, but it was going to take us another year and a half to finish it and we couldn't raise any money at all. We got to the point where the one person on the team who had kids against was the only person that's getting paid and then we decided to try to find something else we could do that would make use of the technology that we already developed, but that will be quicker to get to market. And that thing was Flicker, but it was also

a bad idea at first. So because we had developed this game client and we had a messaging server on the back-end, we just re-purposed the game client as a photo sharing client. So in the game you had a bunch of slots along the bottom of the screen for inventory, those became slots into which you can fit photos. And because the game was all real time interaction, the photo sharing was all real-time interaction which meant that you have to be online at the same time as someone you cared about in order to use it. And while

it demoed really well, it was amazing technology, especially for at the time we started and that maybe like late 2003. It was a terrible product, because you didn't care about any of the people who were actually online. That morphed really quickly and the experience making Slack was a little bit different. We had all of the resources we could ever ask for. I mean, we had more than enough money, because we had money leftover and we decided to abandon the idea and we had AWS which we didn't have before. Hardware was effectively free internet connections were

much better. There were many more people online. Hardware was better. So, none of like those external conditions was the reason this wasn't. This massively multiplayer game on the web wasn't a good idea. It just wasn't a good idea, because there wasn't a big enough market for it. >> And why was Slack a good idea and how did you decide that Slack was a thing to build? >> Well, we started developing the system for internal communication that was base IRC and those of you who had IRC kind of, and also use Slack can recognize that Slack

is IRC with all of things that were missing when it was originally designed in 1989. I think this is, I don't know, it's like it's post facto analysis. So it's hard to know if this is really true or not, but I think that the reason this lock has been so successful is because we had a 3.5-year long design process where we weren't conscious that we were designing it. So, we started using IRC. IRC has many deficiencies. Among them, you can't store and forward the messages. So if I wanted to send you a message and you're

not connected to the server that the moment I want to send it, I just can't send you a message. So, the first thing we built was about to log all the messages and put them in our archive. And once we had that, we had messages at a database and we thought search would be useful, so we built search for it and this was 2009. There were no good IRC clients for iPhones. And so, we built an HTML5 front-end for the archives and Safari. And once we had that, we wanted to be able to post and

just like on and on and on. But in the normal process of software development, there's a huge amount of ego and speculation. So when you're arguing with someone about whether this feature is a good idea or not and if you can get to the point of enlightenment and a Buddha sense, then maybe you can do that without any ego. But the practical reality is people are very ego laden in those discussions and there's a lot of speculation like, I think this could be really valuable for someone. Whereas when we were building the system that was

kind of the proto Slack, that was the inspiration for Slack, it was whatever problem is the most irritating. Like something that we just absolutely can't stand, we would address that and then with a minimum amount of effort and then go back to what we're supposed to be doing and then kind of let cook for several months. So we had 45 people working on it, working on the game. So, we had a lot of real world feedback or when there was a problem. I mean, when an opportunity that was so obvious that we couldn't help, but

take advantage of it, we would spend the minimum number of minutes on that and go back to it. So the end of that process, we had this fully designed product that we were using that was a terrible implementation not what you would do if you started from scratch, but it was obvious that this was something that had enormous value. Everyone who worked at the company agreed they would never work without something like this again. And so just on that basis, it seemed like it was a great idea and we had years of evidence. >> Were

there other things you considered pivoting to or was it just like there's this thing we're going to do it, it's working. >> I wish that I had better access to the proposals that I personally had written, ciz I had a whole bunch of **** terrible, terrible ideas and this was the one that had the broader consensus among the team as being something that would be valuable. What's interesting at the time though, so we had $5 million left in the bank. None of our investors wanted their money back. They want us to just go try something

else and Jason Horowitz is one of our investors, so we went down and did the partner meeting and kind of presented what our big idea was. And at that time, this is before we actually started development of Slack. But when we had this proto version, we said that one day, this could be like a hundred million dollar a year revenue business. It could be valued at a billion dollars on that basis and we thought that, that was like a decade out and that would be the culmination. That would be We have 100% of the ten

and we went past that in year two, kind of blew past it. We'll do 200 million in gap revenue this year and still be growing at a very fast rate. And now we think that number is more like 10 billion in revenue. Like a hundred billion dollar company. So we didn't realize how good of an idea it was, but it's also there's an interesting story about deciding to shut it down. Because we still had money, because we had a relatively big user base who were very engaged, but we had a really like a leaky bucket,

it was very difficult to add new users to the game that we're working on, which is called Glitch. I felt very stuck in this position, because on the one hand you have narrative of good entrepeneurs are resilient. When everyone else thinks it's a bad idea, it's probably a really good idea. You have to dig in, prove people wrong. Keep going, even when times are dark. One the other hand, there's a morning when I woke up and it's just like, I can't do this anymore, I'm the CEO, I'm the chairman of the board, I'm sure that

this is not going to work. And it's actually hard to convince everyone else. >> I think that's one of the hardest things about evaluating ideas. Is saying when this one's not going to work. So, do you remember what it was about that morning when you woke up and said, you know what, it's time to move on? >> I don't. I mean, I think people have very little interest in access to their own cognitive processes to begin with. So anything say would be kind of making it up. But I think it was I had exhausted all

the ideas we had to fix problems, so we focused on new user experience. And we really wanted to teach people what this was and kind of what the possibilities were and get them excited about it. Because we were okay at attracting people into the top of the funnel but had a hard time retaining them. And. >> That's always when I tell people to give up an idea when you're out of ideas about how to make it work. >> Yeah exactly, so we'd have an idea, try it didn't work, have another idea and at first you

can fill up post it notes, like those giant boards and you can fill up white boards with ideas and you argue about which ones are the highest value. And then one by one you eliminate them. Months go by, a year goes by, another six months goes by. And it's like ****, these are all of the ideas we thought were the worst ones to begin with and we only have three left so this is probably not going to work. >> You mentioned at one point you had a bunch of proposals, how do you come up with

ideas? Or how did you even think that you know, slack is maybe this thing is a really good idea. How do you come up with new ideas and how do you evaluate them? >> Coming up with new ideas is easy. Like I just I feel like it's hard almost to not come up with ideas. I mean, they're not necessarily good ideas so the recognition of which ones are good is actually much harder. I recently finally got around to reading thinking fast and Slow and there's this story there of a psychologist named Herbert Simon who studied

in Chess Grand Masters because Chess Grand Masters are one of the category of people like really experienced doctors. That people regard as sort of miraculous in their ability to intuit a situation, like glance at a board and say, white checkmate in three moves. And people thought that this was like some super well developed intuition. And Professor Simon argues that it's just a recognition in the same way that like a 18 month old can look at a dog and say doggie. Or within the first 500 milliseconds of someone speaking you can tell if they're angry at

you, like you just develop the ability to recognize situation and ques which. Are hard to disambiguate, they're hard to articulate and enumerate in your mind and say exactly why you think this way. But over time, you get better and better doing it. I've been making software professionally for about 20 years, so the specific micro-ideas, or sub-ideas, about feature implementation, or the order of actions in the user experience. Or whether a certain feature is more important than another feature. Those ones, I feel like I have an intuitive sense. Which is actually just the ability to recognize,

over time. So it's really just a process of filtering. Like you just have constant ideas. Everyone has ideas. We at Slack have developed a lot of organizational design and practices to try to absorb as many as we can. So we get about, somewhere between 10 and 20,000 tweets at us a month. And all of those are responded to by a actual human being. We get a similar number of customer support tickets, and we have Processes for trying to filter up and abstract all of the feedback that we're getting. But even down to the level of

all the tweets at Slack the company are pumped into our Slack instance and people check them, and they can add an emoji reaction and that filters it to another channel. For the ones that seem above some threshold of this seems like a good idea, and it also seems practical. I could imagine that we could do it. And then there's some debate at that level, but, I mean, every Well, because almost every person I know in the technology industry also use Slack. Every time I talk to any of them they have ideas for me. I mean

they have complains or proposals >> So the big ideas or the little ideas I get the intuition on. What about the big idea, did you spend a lot of time thinking before you decided to go ahead with Slack, did you spend a lot of time thinking about this would be really strategically valuable and there would be a deep network effect? >> No not at all. I mean so we thought that this would be valuable for individual teams and we could see that value demonstrated in our own experience. We definitely didn't see how applicable it would

be. Because right now, we have 38,000 paid customers around the world, so that's a lot. We have a sales team but we've never talked to 90 plus percent of our customers because we just couldn't talk to that many people. So again, this is already farther than I thought we would get at the very end. And so we under recognize it. Those big ideas are, I don't know, we talked briefly about this topic before and one of my favorite lines is, I have a great idea for a novel all I need is for someone to write

it for me. Because obviously preposterous. Right? Like no one would really think, I mean only a really deluded person would think that that's true. But people definitely have ideas for apps all the time, and those can be harder to assess. And it's so rare that I hear one where someone proposes an idea where it just, it's like prima facie that is a good idea as opposed to [SOUND] I don't know, I mean it really depends on the execution. Actually just last night, I was watching something on YouTube, and the ad that came up before was

for an app called MileIQ, and it's you turn it on, ask for location permissions, and then every time you go on a car ride, it just looks to see the speed that you're traveling. And then at the point you stop, it stops it. So kind of like an automatic Strava. Like it just records all your trips. And then you can swipe right to say that this is a trip that my company's supposed to reimburse me for and swipe left, not. Anyway, it's a very simple idea. It's actually a really great idea because people have trouble

recording those, they have trouble in audits with the IRS to prove that they used the card for this purpose and stuff like that. And that's one that actually seems like. Hm, the execution doesn't seem very difficult. I think that would be a good idea. If you could market it effectively, it would probably be a successful business. But like 95% of the ideas I hear, I have no idea whether it'd be successful. >> Where are you generally on the spectrum of ideas matter and it's worth the effort to think of a good one and I think

it's just all that execution, and ideas don't matter at all. And you just have to keep trying things until someday one of your internal tools turns into slack. >> So there's, I'm There's a lot of people in this class and then there's a lot of people at Stanford, and there's a lot of people in South Bay and there's a lot of people in the Bay area and many many people want to start tech companies and all hope to be successful. So to a certain extent there is like 100 million monkeys banging on typewriters and someone's

going to start a Shakespeare sonnet so, there is a little bit of that. Like a little bit of survivorship bias and other cognitive biases that come into play. On the other hand, I don't think either end of the spectrums is right. I would definitely lean more towards execution. There's a good Steve Jobs elaboration of John Sculley got hired and started hiring all these people at Apple who thought that only the ideas matter. The idea absolutely matters, but the idea when you start hitting execution has to accommodate so many things. In Steve Jobs case, it's about

what it's possible to get glass to do and how small you can make batteries and what kind of tensile strength plastics have, and stuff like that. because your idea has to stand up to all these things and it'll transmogrify and alter and may even be unrecognizable at the end, but if you're really good at execution and the core of the idea can make it through all those steps. So I absolutely favor execution, but you can execute a lot on a terrible idea and then not get anywhere so you kind of need both. >> If you

were starting a new company today, you know April 2017, where would you look for ideas? What sorts of things would you be thinking about? >> I don't know if this works for other people, but I would definitely look at my own experiences as a consumer, generally. Because it's very easy for me to see things that are frustrating, I complain about stuff a lot like, especially given how good my life is. My life is really good and yet every time I have to fly somewhere I have at least ten significant complaints and 50 minor complaints. And

every one of those is an opportunity, right? Every one of those things that don't go as well as you would like are opportunities for improvement and some of those seem like someone else should see them and it should be really obvious. But they're often not. There's this story I tell in our internal on boarding process of me and our head of product design going for a walk in Vancouver. And our office in Vancouver is in a neighborhood that has really narrow sidewalks. The sidewalks have sandwich boards from vendors out so it's very crowded so you

kind of wind around. And it starts raining while we're on this walk. And maybe two thirds of the people had umbrellas with them and we didn't and people are walking towards us on the sidewalk. And almost no one would move their umbrella out of the way so that the pokey things wouldn't get us in the eye. And like I said, the sidewalks are very narrow, so we had to keep on ducking. And I can come up with many explanations for why this would be. But don't explain with malice that which can be explained by ignorance.

So it probably wasn't just that these people have few avenues for exercising power in their lives, and they were choosing like this moment to exercise some power. But then there's only really two explanations. One is, they just walk through the world and they don't see that we're going like this to get away from their umbrella, even though inevitably they've had that experience themselves. Or they see that this is happening and they're like, I just can't think of anything I can do to ameliorate the situation, despite the fact that it's this tiny, like a hundredth of

a calorie, worth of effort and a tiny amount of consideration. And the point of telling that story is that that's the way, and this is a sad way of looking at the world, but that's the way that most people go through the world. They're oblivious to the problems that other people have and If they notice the problems, they're unable to come up with any kind of solution. So like two thirds of the people just didn't tilt their umbrellas. Which means that if you're the kind of person who's willing to tilt your umbrella, there's a whole

world of opportunity out there. >> Do you have any tips on how to notice that you should tilt your umbrella? >> Pay attention. I mean, that is the biggest thing. But pay attention especially to other people. So you can pay attention to your own signals and maybe if you get practiced- >> This has actually worked at the [INAUDIBLE] history when people have needed to come up with a new idea on the fly, it doesn't usually work, but when they really just commit to paying attention to other people and sort of what seems broken, that has

worked. And then so if you do it this way you come up with like thousands of ideas. Anything you've noticed as a good way to filter those besides just building up the experience of seeing a lot and the recognition? >> Yeah there is the kind of dialectic between execution and idea because there's definitely something to be said when things are above some threshold of good to sticking with the same idea. Not so much because that idea's the best one, but because the wild and erratic chasing of the next idea when you have something that has

a little bit of traction can be really dangerous. So, Slack is a good example of that. We conceived of Slack as a tool for business. We developed it that way, we priced it that way, it was successful. But, there's also this huge amount of social usage, like now, maybe a third of the Slack teams that get created are for some kind of social usage. And a lot of people had the idea that we should pursue that, we should make Slack for groups, or even just pivot to being a consumer company, like kind of a private

version of Twitter or something like that. And we've been pretty steadfast in not doing that because you can only do so many things well. And there's a lot of shiny objects like that, that are very distracted. >> Are you the one that says no to most of those new ideas or you have you had to build that into the company? When something's going as well as Slack, there's so many new ideas. >> I'm chief no sayer but there's definitely a whole bunch of people who are equally committed to no. >> Great. We can probably take

one question for Stewart if someone has a really good one before- >> Only a really, really good one. >> Maybe that was so much pressure now no one's going to ask. >> [LAUGH] >> So you already had a successful company before Slack. I was wondering what the differences between as a first time founder and a second time founder? >> What are the differences as a second time founder? >> So Everyone of the external factors was much easier. So the first time I mentioned that we ended up making Flickr because we couldn't raise any money and

we're desperate to try to finish something. Fast forward seven years, and we could raise as much money as we wanted. So that was easy. We didn't even make a DAC, we just said, we would like some money and then people gave us money. So a lot of those things were easier, it's always easier to get press, it's easier get attention, easier to recruit, easier to raise money. It's not any easier to be successful certainly because we started the company that made Slack in 2009 and failed pretty hard for three and a half years. Kind of

like pushing into the wind of something that wasn't ultimately going to work. And we could have just stopped there. It was great for investors that we didn't recap and we just kept the same company and so that company has now been successful. We could've also just walked away from that company and incorporated a new company. And then it would've been my second company was a failure instead of my second company was a great pivot. I'm not sure which things get harder, I'm not sure than any do other than the degree of self criticism and like

awareness of your own limitations. Great, thank you very much. >> No problem. >> Appreciate you coming in. >> [APPLAUSE] >> Now, we'll have Adam talk about metrics. >> Great, so thanks for having me. I'm happy to be here. I'm going to talk about metrics and measurement and this whole group of issues that come up as you're trying to start a company. And, one of the things, I think is really interesting about this topic that is so much of what you have to do in a startup is specific to the startup. You're going to have to

get to be an expert at some domain. You're going to have to get to be better than anyone else at that domain. And a lot of what it takes to do that is, hard to teach in a class because it's just different for every startup. But, metrics is this domain that's kind of general, general enough to be useful to almost all startups. So, I'm going to start off with this example. So, this is the first version of amazon.com from the early 90s. [COUGH] And, I really like looking at these. You can look at the first

versions of lots of different products. It's a good reminder that it's much easier to get these things started than you'd think. Even these companies that are massive today, if you just go and look at how they started, this is a good way to keep yourself in touch with the reality of how companies get started. So, does anyone know why Amazon decided to start with books? Yeah? >> Choice. >> Choice, yeah, yeah, yeah, so that's actually, I think what you mean is that you have a lot of, you've got to choose out from a lot of

different books. Yeah, yeah, so it wasn't just that they chose books, it was that- >> [LAUGH] >> It was books give you, so, Jeff Bezos wanted to a start a store and one of the most important things when you're getting started is differentiation. So, you don't have to make a product that's going to appeal to everyone immediately, but, you need to make a product that is going to appeal to some people more than anything else. So, you want to find some way to differentiate, and usually, you want to do that by doing something that's going

to be easy for you but that's really hard for all the existing companies. And, the Internet enabled this in Amazon's case because brick and mortar stores have limited space to hold their inventory. And so, you realize that as on the Internet, you can have a big warehouse somewhere, you can have items shipped from all over. This is something that enables you to offer much wider selection than a normal store. So then, you go and run through the thought experiment. You say, well, if what we can do is provide better selection, then which category, what kind

of store is going to benefit the most from a greater selection? And it turns out that books is a category where you have the most, there are more books than there are other products, right? So if you look at like movies, it's very expensive to make a movie. That's not a industry that would benefit immediately from the Internet in the way that books would. So, I think there's a general point here that measurement can turn a vague idea into a good idea. And, if you do a lot of these kind of like back of the

envelope calculations, it can really helps sharpen your ideas and it can help you iterate and move through the space of ideas a lot faster than if you have to go and like, build the prototype. You can measure what Amazon started and they started to focus on movies and after a few years, they realized that it wasn't working that well, then they're set back by a few years, now then there's competition. These kind of metrics on, even in the realm of ideas can really help you get started. And the one other thing I'd say about this

is that, I think using measurement here is very good for helping you figure out what products are going to be possible or where you're going to be able to provide a lot of value to a user? I actually don't think metrics works that well for stuff like market size. So you just saw this example that Stewart, they thought that the market size for chat was 100 million a year in revenue. It turns out to be 100 times bigger. It's just very hard to figure those things out. I wouldn't spend a lot of time on that.

I think you want to know that whatever you're doing, someday there's some path for it to eventually get really big. But, you want to really apply your metrics to how can you differentiate your product, and what's going to be possible that wasn't previously possible. So, okay so next, now you're building your product and what should you measure? Just in any guesses, if you guys have products, what are you measuring, what would you recommend, yeah? >> [INAUDIBLE] >> The amount of users, yeah, yeah, so what exactly would you mean when you say that? >> [INAUDIBLE] >>

Yeah, so how many people use a total and how often they use it and go ahead. >> I think [INAUDIBLE] people share it but it's viral, [INAUDIBLE] >> Like whether it's growing or whether they do something so that it would grow, yeah? >> Total addressable market like when you're getting started. >> Total addressable market, yep. Yeah? >> User attention. >> Yeah, user attention. So these are all great ideas. And, there's lots of things that you can measure. As a startup, you really need to focus. So the core concept I would try to focus on is,

users that are getting value today. So, there's different indicators of this. You might say, you want to focus on active users. People are going to be active, if they're getting value. You might want to focus on revenue. That's a sign that people are getting value. In the case of a marketplace, where you have things like buyers and sellers, that's usually a case where it would make sense to measure transactions. Any time you have these like different groups, if you say, okay, we're going to measure buyers and then we're going to measure sellers then, it's actually

lot of stuff you can do that will make things better for the buyers and worse for the sellers and some of those things are good. And so measuring transactions is a way to unify them and align your work on things that are going to benefit both sides. So, if you have a two-sided market or anything like that then, you want to measure transactions or transaction values. So things like eBay, they'll measure total value of all the goods that are sold on eBay. So to start, this is kind of a quick thing. And you go a

long way just by measuring this. If you're not measuring anything, you're susceptible to all these cognitive biases that Stewart talked about. There's this all these kinds of problems, you're just not going to know what's going on. So next was most important thing to think about is retention. So, first, I want to make sure everyone understands what a cohort is. So, a cohort is all the users who first use your product in a certain time window. So for a startup, you might make the time window a week. So you could say, there's a cohort of all

the users who first use your product in the first week that it launched, and you can track those users over time. There's a cohort of all the users that first use the product a week later. And, you measure all these cohorts and you can track them and this tells you what's happening to users over time. And this is a really important way to balance, you're just measuring the total value that all the users are getting so This is a graph of total of total cohort usage for a sort of relevantly old Internet product. So the

blue line is users who joined this product in 2004. And that's as you can see, and so this the y axis here is what percent of them are still using it. And the x axis is time. So overtime you just follow the blue line. You can see usually over month there's fewer and fewer people that joined the product in 2004 that are still using it. And this is usually the same story for all these different Cohort lines. So what do you think is going to happen to this product in ten. This is about, let's say

this is about six years. The scale of this graph. What do you think is going to happen six years later? Any guesses? >> So kind of the opposite. >> It's going to turn around. >> Yeah. >> Yeah, it could. I think something would need, something different would have to happen to change that. >> No one using. >> No one will be using it. Yeah, so this is, it's hard to know what would happen. I think probably everyone is going to stop using their product. I think some companies will manage to turn these around. But it's

very hard too because you get fewer and fewer, you get less and less usage from your old users. It's also usually the old users are what caused you to get new users. And so if you have this problem it's pretty bad. This is a pretty scary place to be. Even if you can have a ton of users you can have total usage growing. But if usage from every cohort is declining then you're going to run into big problems. So there's this idea called the ring of fire which is a way to kind of visualize this.

So unless there's some kind of lower bound you're going to burn out all your users. And so here's the analogy, imagine there's a big field of let's say of dead grass and someone lights a match in the middle. You're going to get a fire and the fire's going to grow and grow. But then in the middle of the fire you're going to start to run out of fuel and it's going to burn out. And so you get this ring. And the ring is going to get bigger and bigger. And in the center you're going to

burn out more and more of the fuel. And so the fire will get bigger and bigger but eventually you're going to run out of the whole field and the fires going to stop. You don't want this to happen. Here's an animation I made to try to illustrate that. >> [LAUGH] >> So here's what you notice. So it looks great at first. There's more fire every frame than the previous frame. But in the end there's nothing, right. So you don't want this to happen. And it's really important to internalize this concept. Because if you don't you

can fool yourself and you can do a lot of things that are going to cause you to get more and more usage over time. But at the expense of the existing users and if you don't do this eventually your company is going to die. I think some recent examples are I think Groupon was a good example of this. They got a huge tons of usage, all this merchants but it wasn't a good experience for the merchants in the long term. And so they all stopped using it. And then I think more recently Pokeman GO is

another example. It got really big, lots of users but it just kind of faded away. And so you don't want to build a fad. You want to build something that's going to last. And so you need to make sure that you're measuring the usage from the existing users. It's not just that you're getting more users. So you need to make sure the existing users keep using your product. Actually if this can go in reverse you're in an extremely powerful position. There's a small number of products where the cohort usage actually will increase over time. One

example is Whatsapp. This is I think most messaging apps, as you get more of your friends on the app there is more people to send messages to. And so they are able to get this increase in cohorts usage overtime. Uber on the rider side. This is another good example as people get used to Uber they use it more. As the pickup times come down they use it more. As prices come down they use it more. And so Uber's in a really strong position there. It's not true, actually on the driver side. So Uber has problems

on driver side because they don't have this property there. And then Facebook you can see the numbers they report publicly. They keep on their total usage is growing faster than their number of users. So you know that they must have this increase in usage per user overtime. So if you get to this, this is very hard to get to this. But if you get to this point then you're in a really strong position and one way to actually think an idea is to whether. As the product gets bigger as you get more users. As you

develop the market is it the kind of product where people would use it more over time. I think that's another lens you can apply as you're considering ideas. Just one more data point, this venture capitalist named Tom Tunguz did a study on what predicts valuations that startups raise money at. And it's just a simple correlation study. And these are some different factors in how much they correlate to the valuation that a startup's going to get. First is growth of revenue, and that's 0.18 correlation. Second is just how much revenue they have in total. That actually

Carl is a little bit more strongly than growth. But by far the strongest correlation is account expansion which means people spending more overtime. It's the same thing as retention increasing overtime. So and this correlation is extremely strong. So for a complex number like what the valuation of a company is going to be. The fact that you can explain point five four out of one of the variation in that number by the single variable. About whether the company is increasing revenue per user over time. That's a sign, it's almost like there's just two classes of companies.

One class where there's increasing revenue over time and another class that gets totally different evaluations where this is not happening. So the main thing I just want you to remember from today is you should measure your retention. That's going to lead you to do a lot of things that will help you in all kinds of ways. It's going to help you, it's going to help you grow. It's going to help you retain your users. It's going to help you build a product that users really love. So next I just want to talk about exponential growth.

So retention is the most important thing but it's not sufficient. You need to have a product that's going to retain your users. But that's not necessarily enough to make sure that you're going to be able to build a big product and reach everyone. You also need to make sure that you are going to grow and by far the most powerful way to grow is through exponential growth. This is just a few functions spotted the red line is linear growth. And blue line is I think is cubic growth and then the green line is exponential growth.

And the linear growth path this you could do these with few things like continually doing PR. You can get more and more users. Eventually, there's a limit to how big you're going to be able to get. It gets expensive to run advertising, it get's expensive to do these kinds of things that attract a fixed number of users. But if you can do things that cause you to get users in proportion to the number of users you already have, then that's going to get you to exponential growth. So there are other places to learn about what

to do to grow faster. But as you're measuring your growth, you want to make sure that the way you measure it is aligned with what you want, and if what you want is exponential growth, then what you need to measure is the rate of growth. So a good way to measure this is just the percent increase per week in how much usage you have, or how much revenue you have. This is a tweet from Paul Graham that I especially agreed with. He said, a suggestion for really tough founders is instead of graphing your revenue, graph

the percent growth per week. I don't actually know why he said that this is for really tough founders. I actually think this is what's going to make you really tough if you measure this. I think the causality can go the other way. So this is a very important thing to measure. The reason he says you need to be tough is that, it's going to be hard to keep up exponential growth, and it might not feel good to just see this line that doesn't increase. But I mean you can also measure your total revenue. There's no

problem with doing that, but just make sure that you're measuring the percent growth per week. Because a lot of the time, there's going to be things you can do that don't cause exponential growth, that would actually help you maybe more in the short term. And you want to focus on the stuff that's going to cause the product to grow, in the short-term and in the long term. Which is basically things like viral growth, getting users to invite more of their friends, getting word of mouth spreading. Getting some kind of dynamic where the bigger your product

gets, the easier it is to attract the next user to it. So next I want to talk about iteration. So one of the most important things, one of the most important advantages you have as a start-up, is that you can iterate faster than a big company. There's all kinds of things that cause it to be hard for companies to change direction, and you just don't have these forces when you're a small company. And so if you can iterate much faster than other companies, then you can go through ideas faster. You can go through changes to

your product faster. You can go through changes to improve retention faster. You can go through changes to increase growth faster. It's going to add up to a really, really strong position. But at the same time you have to be iterating in the right direction. And so one of the reasons metrics, even just simple metrics, are so important, is that they let you know whether you're going in the right direction. They let you know whether what you're doing is working, and they let you out of these different ways you can go they kind of help you

stay aligned. One other point I want to make, I think this isn't mentioned too often, but you can measure how fast you're iterating, and you can measure all these components of iteration. So whether it's fully quantified or not, you should have in your head, okay, how long did this cycle take us, and what could we do to make the next cycle faster? At Quora, one of the metrics we track is, how long does it take from an engineer hitting commits on their code, to us being able to run the unit test, and then and then

get that code out into production. And we try to keep that in like between 10 and 15 minutes, from a change in the code base to it being totally out to all the users, because that allows people to iterate just much much faster. And if we didn't track that metric, there's just a lot of forces that naturally would cause it to get slower. I think a lot of companies end up, they maybe start out like that, then they end up releasing daily, or they go to weekly, or even monthly. I've heard of teams that are

on like every three months they do a release. And you just imagine, how could you possibly iterate if your releases are only going monthly, or even weekly? So next, there's a spectrum of how seriously to take metrics, that I see a lot of companies stumble on. So on the one hand, there's not taking your metrics seriously enough. And this might mean things like, you don't have metrics. It might mean that you have metrics, they're on some page, but no one ever looks at that. It might mean that you have metrics and one person in the

company looks at it. But no one else looks at it, and the issues that happen on this end of the spectrum are, you kind of like, you can fool yourself, you go around in circles, you just like lose sense of what the right things to do are. And then there's another end of the spectrum, where you can actually take metrics too seriously. So you've gotta remember that metrics are an abstraction. There is where you're actually care about underneath is, let's say the value to the users, and maybe the long-term value to the users. And so

a good metric will be strongly correlated with that, but it won't be perfect. And so you don't want to forget that there's a reality under the metric that's not exactly the metric itself. And so if you focus too much on the metrics, then you start to get into things like, there's these ways to game the metrics. And we can make the metrics go up, but actually we're doing some things that are going to hurt us long-term, and those things are going to outweigh the short-term benefits. So I would really make sure to be careful about

that. I'd say, personally, I think the ideal is to be somewhere in the middle, and I'd say you want to be very close to taking the metrics extremely seriously. Not too seriously, but I think most companies bias toward not being serious enough about their metrics. If you're in the right, if you're middle in the right spot then you're learning, you're making good decisions, and you're informed. But you're still aware that there is a product underneath, and there is a strategy and there's more to the world than just these metrics. I see a lot of times

companies will oscillate between these, like they'll say, they'll get really focused on metrics and then that causes them to do some things that they think are bad. And they're like these metrics are terrible, and we're going to throw away our metrics, not focus on them at all. And then you get into this world where you don't actually know what you're doing, and there's no accountability. And so oscillation is not very productive, you want to just make sure you find the right point and try to stick there. So finally, I want to just talk about some

of these psychological aspects that come with metrics. I think one of the big things that come when you start measuring things, is you just get closer to being in touch with reality. And that can be painful, because sometimes the metrics are going to be worse than you thought they were. So sometimes they can be better than you thought, and that's great, and everyone feels great. But a lot of times if you weren't measuring something chances are you weren't even aware of it, it was hard for you to make it good, and so it won't be

good. And so that means that as you set up metrics, there's going to be some pain, and that can cause problems for companies, I even think at the leadership level, this can get even worse. Actually, I think the majority of leaders out there in the world rely on some amount of delusion they're creating among their followers. To get them excited, to have them working on this project in the first place, and metrics can get in the way of that. And so people get into this, then you get into this question of like, okay, well, what

should you do as a leader? And this is where I come out, after thinking about this for a long time. I think you want to be learning from the past, so you want everyone fully aware of what has happened, what has worked, what has not worked. You want to be totally in touch with the reality of where you are today. So if things are good, you should know they're good. If things are not good, you should know they're not good, you should know exactly where there are problems, and where there are not problems. But then,

those things are not in tension with being optimistic about the future, you should be optimistic about the future. Because there's so much potential, and the reason you're doing a startup is because of this potential. And if you don't feel confident that the path you're on is able to get you to a place that you feel optimistic about, then that's a good sign that you might want to change the path. And you should do something about that, because it's totally in your control. So there's not really a tension between acknowledging that things are not in a

great spot, and having people not be optimistic about the future. So as long as you just make sure that the reality is that, and I don't think you should make people optimistic about the future in a way that you don't believe. I think you should do the things you need to do, to get yourself to be confident, and optimistic about the future. And then it's easy to get people to follow through that. So that's all I have, and happy to take questions. >> [INAUDIBLE] the question with the IT's desk >> [LAUGH] >> Thank you very

much. >> [APPLAUSE] >> All right, see you on Thursday.