How Google sets goals: OKRs / Startup Lab Workshop

Good morning. Uh my name is Rick Klau. I'm a partner at Google Ventures for those of you I haven't met before. Thank you for coming this morning. Uh thank you to those of you watching online. Uh we're here to talk about objectives and key results. This is one of the critical elements of how Google managed its early uh execution and really grew the company and remains pretty critical element of Its of its DNA today. Uh I came to Google uh little over 5 years ago. I was part of a company as some of you know

called FeedBurner. FeedBurner was acquired in June of '07. Um The very first day uh that I joined Google was the first day of a quarter, which meant that Google was already about a week and a half 2 weeks into that quarter's OKR process. I didn't know what OKRs meant. Um I was asked to draft my OKRs within my new team. Um I I really didn't have any idea what was expected of me. Um certainly at startups prior to Google had never had a formal process of setting my objectives um or measuring goals. Um but in

the last 5 years I've set OKRs as a biz dev lead, as a product manager on a couple of uh different products. And then more recently in in my time as A as a partner at Google Ventures. Um process had a real clarifying effect for me personally and for the teams that I was a part of. Helped me understand what mattered. Um it just as importantly helped me understand what I wasn't going to be working on. Um today we're going to talk about what are OKRs, how do we implement them at Google. Um How to

adopt this process at your own company. I think one of the things Startups often struggle with uh and one of the things I spend a lot of time working with you all on is fighting the urge to just say, "Well, that's Google, right? You guys you're you're different. We're not Google." Well, as you'll see in this this deck, um Google wasn't Google when this was first adopted at Google. Google was under a year old uh when John Doerr came to Google and pitched Larry, Sergey, Marissa, and others on how to adopt this Process uh at

a very young and uh an ambitious company. Um I'm going to apologize in advance cuz I've committed uh at least two cardinal sins of uh of of in this case Keynote, not PowerPoint. Um so much so it wouldn't surprise me if somebody from Cupertino uh drove over and wrestled uh control of this remote from me. Not only have I uh used slides that are almost entirely text, um which is bad enough on its own, But in addition I've also embedded an all text presentation inside of an all text presentation, which uh is is a new

low for me as a presenter, but I think as you'll see there's there's a reason for having done that. Quick show of hands. Um how many have read the book uh In the Plex by by Steven Levy? It's a book about uh Google's origins and growth up until about 2 years ago. Uh okay, um maybe 20% or so. Um It really is a a great uh book. Um Steven was given uh really incredibly uh direct access to not only our founders and many early employees, uh but a number of people across Google that gave Steven

a very broad sense of who we are, how we worked. And he talked about um how in 1999 uh John Doerr, who uh is a partner as some of you may know at Kleiner Perkins, uh remains a board member for Google to This day, and he came to Google to talk about a process that he'd witnessed at at Intel uh called OKRs, objectives and key results. And Steven in his book talks about and I know that those of you on the livestream can't see this. I won't read the whole uh pages here, but talks about

how how Doerr came into Google and and OKRs were an immediate hit. Um that unlike uh what you might expect where this might be seen as some kind of Dilbertization of of of a tech company, instead um it was immediately apparent to uh Googlers that this was data, right? That that um what this gave the the company access to was a way to quantify instead of just qualifying um what it was they were working on and how well they were doing towards those goals and periodically reviewing um Did they meet expectations for what it was

that they were trying to do. Um and a highlighted uh element here in a couple of things that are are critical to how we think about OKRs and and I think you'll see themes resonate throughout this discussion. Um it was essential that OKRs be measurable. Um an employee didn't say, "I'll make Gmail a success," but "I'll Launch Gmail in September and have a million users by November." Real critical there are two different points, right? In the key result, launch by September and a million users by November, those are both measurable. Right? The key here is

that not only is the objective clear, but the means by which you would grade whether or not you had met those uh the that that objective um is is really what's key here. Um and Marissa Goes on to say it's not a a key result unless it has a number. Um The OKR embodied ambition, it sanctions the ability to take risks, says Doerr. Um And then it goes on to talk a little bit more about um uh what else makes OKRs key and and it's important to note that everyone's OKRs, from Larry and Sergey on

down, are public within the company. Everyone. It's part of your directory listing. Uh when when what's called Moma Steven references it by name here in the book. If you go into Moma, you look up an employee right next to their cell phone, their email address, their title, is a link to their OKRs. That means that not only can you see what they're working on this quarter, but you can see what they were working on last quarter, quarter before. You can See their grades. And we'll talk more about the the process of how to grade them

and and why um different approaches may may be beneficial in your companies. So, I got a hold of John Doerr's actual deck. This is the embedded all text presentation uh I was talking about uh a few minutes ago. Um But I actually got the original deck From 1999. And I think it it it helps I I'm not including the entire deck. Um I'm I I I tried to take what I felt were the most critical elements that illustrated uh both how we think about them at Google and how I think you should put them into

context for understanding how they might apply in in your company. Um So, objective of this deck. Imagine that I'm John Doerr for a moment Um presenting this to Larry, Sergey, Marissa. He He started his deck with his OKR for that meeting. And the objective was I'm going to develop a workable model for planning as measured by and then it established three key results. Note how each of these three are inherently measurable. One, finishing the presentation on time. Now, I haven't told any of you how long this workshop is going to take, so I think I

Have a pretty good chance of finishing on time since only I know what on time is. Um Completing a sample set of 3 months objectives and key results. So, at the end of the meeting they were going to have a set of their OKRs. Three, having management agree to institute a trial system for a 3-month period. Pretty straightforward. So, John goes on To give examples of um an OKR in the context of a football team. Now, I'm a big 49er fan, so I'm going to use the 49ers as my illustration. Um I think the key

here is is one of the things that when I've talked with many of you individually, you've struggled with, which is how to connect the individual objectives with the team objectives with the company objectives. And I think John's uh example of the football team actually goes a long way to helping clarify what that what that looks like. So, general manager, I'll I'll say the team owner Jed York, uh his his job is um I he's going to win the Super Bowl um and fill the stands to 88%. Now, 49ers will tell you we're going to have

sellouts all season, but these are measurable key results with an objective That in this case is defined as the GM making money for the owners. You can can get a sense of what that looks like. Now, filter that down. The head coach, notice that he doesn't care about filling the stands. All he cares about is winning the game, right? Winning the Super Bowl. And the way he's going to win the Super Bowl is he's going to have a passing attack. Now, in the case of the 49ers, Uh it's a bit of a crapshoot if Alex

Smith is going to throw for 200 yards in a game, but maybe it's more important that we get Frank Gore the ball. And so we might say that, you know, we're going to we're going to we're going to cumulatively run for more than 200 yards. Or number three in defensive stats, given the 49ers defense, you might say things like we'll have a a net um positive takeaway Uh differential where, you know, we'll have more interceptions or fumble recoveries than the other team. Right? These are measurable things that at a high level, in this case the

head coach, is focused on. Now, these roll up into the prior comment, which was, you know, the GM is going to win the Super Bowl fill the stands, but obviously there are some things that the GM is not worried about that the head coach is intensely focused on. Uh pivot from the head coach to public relations, their responsibility is they own filling the stands. Right? We we're going to hire colorful players. I'm not sure that if I were in charge of PR for the 49ers that that would be my focus. Uh but if I think

back to '99, I think that's when they had hired Deion Sanders, so maybe that exactly was the strategy back at the time. Um get media coverage, highlight key Players. Obviously, this is a way that you generate enthusiasm amongst your fans. That may be a byproduct of what the coach focuses on, but it is absolutely 100% of what the PR uh function is focused on here. Now, filter that further down. Um defensive coordinator, all the defensive coordinator is focused on is obviously the defensive stats, holding the other team under 100 yards passing, Uh intercepting at least

twice in the game, uh recording at least three sacks, whatever you want to uh extend that analogy, whatever might be important given the players that that defensive coordinator has at his disposal. Uh offense on the other hand, of course, doesn't care about interceptions except for not throwing them. Um But they're going to set a a a an objective of 75% completion rate. Now, I should note here, um for those of you that follow football, 75% completion rate would be pretty astounding, particularly in a in a Super Bowl. Um part of the goal of an objective,

we'll talk about this again as we get into the mechanics of how you define OKRs, Is that it be a little uncomfortable. You don't want to always get on a grading scale of zero to one. You don't always want to get a one. In fact, the target ought to be a .6 .7. So, you should always have goals that are slightly out of reach, that feel a little uncomfortable, um so that you are always striving to do more, to do better than perhaps you did The quarter before or you think is reasonable um in the

in the context of the current quarter. Um Think about now, we've talked about defense, offense, there is a special teams unit. Right? They're not focused on interceptions. They're not focused on how how well the quarterback does. They're focused specifically on running Back kicks, trying to block the punt, trying to block the the field goal. Um They have very specific goals that are completely distinct from the other guys. Same with the news staff, same with the scouts. Right? They may not even be on the field. They may be off watching spring practice for a bunch of

colleges. The idea here and what what uh John Doerr was trying to reinforce to uh the the the Google team at the time was that if you roll all of the things up from this foundation level, in their objectives and key results are reflected the ultimate company's priorities. But not every single one of them is going to be considered a company priority. Right? Is it um Uh here the news staff has a key result of three Sunday features articles or it might be a, you know, an interview on ESPN or it might be something along

those lines. Is that that's not going to be a company OKR. Right? That's not going to be the the the focus of what the 49ers are are thinking about. The the GM or the owner of the team isn't going to be waking up every day saying, you know, how are we Doing on those Sunday features? But you better believe the head of the news team, the PR team within 49ers organization is absolutely thinking only about those things. Now, what does that further to the company objective? Well, ties back up into filling the stands. Right? Um

so you can start to see how you can pretty clearly connect these dots even if not all of the foundational objectives and key results ultimately show up in the company uh OKRs. Now, um want to talk about why you use objectives and key results. Now, this was John trying to convince Google. Google now has lived these uh this this approach for 13 years. Um for those of you that have not yet made the commitment to embracing this method of measuring uh both your Uh objectives looking forward and grading yourselves looking backwards, let me talk about

some of the things that led to John's bullets here where he was trying to make the case. Let me tell you about, as a Googler who's now lived this for five of Google's 13 years using OKRs, um what is it what is it do? It it is absolutely amazing for imposing discipline on the organization. It helps me individually Understand what I'm working on and why. It also helps me make conscious decisions about what I'm not working on, which is often just as important. Um Make sure that by making these OKRs public, everyone on my team,

anybody in the company that cares, can see exactly what I'm working on, can see what my priorities are. Why is that helpful? Well, to give you one example, um when I get dinged by a A team at Google who wants, let's say when I was the PM for YouTube's homepage, somebody pings me and says, "Hey, we would love to promo this new product on the YouTube homepage." Well, everyone would love to promo things on the YouTube homepage. The person who's going to make that pitch, it might help them to know what my priorities were for

the quarter in which we were working, And they can immediately know, even before they talk to me, what my my answer is likely to be. If the thing that they're promoting is somehow connected to the things that I care about for that quarter, they've got a very easy case to make. They're helping me deliver on my objectives. If on the other hand, it's a distraction, they at least know going into the conversation they're unlikely To get much traction in that quarter. Now, it might help to plant the seed with me, let's say in late October,

so that as I start thinking about Q1's OKRs coming up, that I have them on my radar. By writing objectives and key results, specifically the key results that are inherently measurable, you're establishing how you're going to measure what it is You're working on. And in the process of drafting those OKRs, and we'll talk about how um how particularly individuals set their OKRs, um you come up with really consensus from the individual to the manager, manager to the to the team, to the company, depending on how big you are. That might be individual to CEO. Um

where everyone understands at a pretty basic level what is it we're trying to do and more importantly, how to measure whether we're there or not uh and how close we are. Um And then finally, focusing effort, I think we'll talk more about that in a little bit. Um Thing I really like about um How this process ends up getting embraced is that there's a really virtuous cycle that happens. Um company objectives are generally set by the CEO, your board, the leadership team, who should have a pretty strong sense in any given quarter of what the

things are that matter most to the company. Right? Take my FeedBurner example back Back in the day. Dick Costolo was our CEO. Uh somewhat comical way of establishing it, but the the the mission was get all the feeds. Right? If we get all the feeds, syndicating all these RSS feeds uh through FeedBurner's platform where we could both measure all the traffic that was happening and give publishers a way to monetize that traffic. It was a very clear statement of what it Was we were trying to do. That meant that when I would go off to

New York and come back and say, "Well, such and such company wants us to do X." The question always that that always followed was does that get us closer to getting all the feeds? And if it was a distraction, might be an interesting revenue opportunity, but it was not delivering on the main core Focus of what it was we were trying to achieve, then the answer was no. So, these corporate objectives are set and generally communicated down throughout the organization. Individual objectives, uh by uh contrast, are often what the individual wants to work on and

what the Manager wants that person working on. Instead of this being an all top-down discussion, it often ends up being a bit of a cycle. We know generally what the company wants to be accomplishing, as an individual, I want to work on important things, so I'm going to find the things that are within my sphere that I can do that are most beneficial to the company. And I'll come up with a list and then communicate that to my manager or the CEO or whomever it is I report to and we Can have a bit of

a give and take. Now, what's interesting in this process is that occasionally, things that individuals want to be working on might end up opening up new discussions for what the company should be doing. Good example of that at Google, engineer called named Paul Buchheit, um was really annoyed with uh the state of mail clients in in uh in Linux. Um so he started he he thought maybe I could make Mail searchable as as sort of the core interface and he started what became Gmail. Well, over time, Gmail became its own practice group at Google effectively

and became one of the core strategic goals at Google, which then had a way of creating its own objectives and key results for the people who were working on on Gmail. So, let's talk about the communication. One-on-one meetings where, as an individual contributor, I talk to my manager, my boss, the CEO, whomever it is, about what it is I want to be working on and what I think is is the best use of my time. Uh the one-on-ones at the end of the quarter and the very beginning of the quarter, best to John talks here

about negotiating what those key results might be. That's a I think actually a pretty Good term for it because you are ultimately offering what it is I want to work on and what I'm hearing back is what we want you to be working on and somewhere in the middle we find the best combination of the two. At a company-wide meeting, the um company OKRs are communicated and the team owners uh identify what it is they're going to be uh working on. There is also a grading process that happens here that we'll talk about in a

little Bit. So, a typical cycle. I'm not going to dwell on this timeline only because I drafted one specifically for the time frame we're in about the end of October, um that I think will help clarify um what the general rhythm of um you know, when you should be thinking about which steps in the process. Um so we'll we'll come back to come back to that. Some basic hygiene. Um You don't want to have 12 objectives. It's too much for any one person to be working on. Um really, if you think about a a a

quarter being effectively 12 weeks, um if you had in total only a week to give to any given objective, it's unlikely that you could move the needle much on any of them. Um so, you really want to Limit the number of objectives you have and you want to make sure that you're very clear about what the key results from that objective are. Don't, in the interest of limiting your objectives, don't stuff every objective with 12 key results. Um you want to be very clear. Fine to be ambitious, but I'll tell you personally, there was one

quarter when I had seven objectives, I have never been more uh exhausted Um and and just mentally drained from the amount of of intellectual juggling it took to keep all of those different objectives sort of clear in my head and keeping track on where I was with each and who I was relying on and what I needed to to coordinate. It was just it it was too much and and it uh in the end um uh it ultimately took its toll. I mean, it was it was clear at the end of that Quarter that I

had been stretched a little too thin. Um John here has a number. I don't know that the number here matters as much as uh the the the overall intent, which is 60% of the objectives ought to come from the bottom up. Um I think I would phrase this a little differently. I'd say more than half of the company's um objectives um Uh uh need to be coming from the the individuals up through the organization. If there is too much top-down dictation, um it's going to be hard to inspire a lot of those people to be

working on things because they're going to be told what to do instead of telling you what they think is the best use of their time and and talent. There's a comment here about everybody must mutually agree, no dictating. Um I think that's really just another way of saying what I what I said a minute ago. Um Interesting that they say one page best. Uh we do all of this electronically now, so there's no uh issue of of page length. Um Here's a really important point. This surprised a few of you when I talked ahead of

um giving this workshop. OKRs are not a performance evaluation Tool. What that means is, in the annual performance review cycle that Google goes through, OKRs are not a standard element of that evaluation. Let me be clear. When I was reviewed each of the last five years, when I would sit down and you know, write up my summary of what I did in the year prior, um what I think That means for the company, whether I wanted to go up for promotion or not, at no point in time were my quarterly OKR bullets factored in into

that evaluation. Now, there it it it's always possible to just include your OKRs as part of that packet. I found individually, they were always tremendously useful because it it gave me really at a summary glance What I'd been up to over the last year. I could look at the last four quarters of OKRs. I could look at the grades. Again, we'll talk about the grading process in a little bit. And I knew really within a few minutes um what were the biggest things I'd worked on, what were the most dramatic results as a as a

byproduct of that work, and I could then summarize uh my contribution To the company, my impact, um in a way that previously, when I thought about, you know, at the prior startups, prior to coming to Google, it was much harder to recall back, what did I do six, nine, 12 months ago? Um I mentioned this in passing at the outset. Your target should be a .6 to .7 uh on a zero to one scale. If you consistently get ones, you're not Crushing it, you're sandbagging. Um Anything below 40% um is is definitely in the red.

That that means you missed the key result entirely. Um and that should be a a a a forcing function for reevaluating whether or not that is an objective you want to continue to work on. Okay. Last slide of the embedded slide. Promise we'll we'll we'll we'll move on after this. Um John here talks about um this has an alignment function in that it it it ensures everybody in the organizations working towards the same set of results. I agree with that. That was certainly my experience at Google. It was very easy to see amongst my team

members where we were working on complementary things, where we were working on dependent things, where there was something I was doing that I needed Three other people to deliver something and so by keeping track of where they were in their progress helped me understand how likely I was to deliver on on my OKRs. Um It it fairly organically ensures the organization's tuned in to what everybody is up to. Um and it that virtuous cycle I talked about a Few slides ago, um there's a a it's a wonderful I used the word organic a minute ago.

It's the best I can think of here. Um that just ensures that you're working on things that support what the company is focused on as do the people sitting next to you and across from you and down the hall. And there is a confidence and a candidly a Comfort that comes from knowing that you're not treading water. You're not walking through molasses. You know that everyone shares those uh commitments. Um Bizarrely, uh John Evan says here in the last bullet that OKRs are fun. Um That I think is open to some interpretation. Um but but

I do think that there are elements of um how to make this part of the team process that that I'll talk about in a little bit. Okay. Um Keys to good OKRs. Um just in terms of rhythm and process. Um OKRs are both quarterly and annually set. Um Annual OKRs are not written in stone. If you set them in December and in May you've assessed and you say, you know, things that we thought were true in November, December just aren't proving to be true. Don't obviously feel beholden to see them out. Revise them. They are

they are Directional aids, not um not blinders. Um As I've said and I'll continue to say, I think this is one of the the critical elements of OKRs. They are measurable. The fact that I know whether or not we shipped in September like I said we would well, that's measurable. If we shipped in September great. Give it a give it a give it a one if if in fact that was the entirety of the key result. Um Talk a little bit more, but important to note here that there are OKRs for each individual. There are

OKRs at the team level that summarize what the collection of people are working on. And then there are OKRs up at the company level which tend to reflect the most important three, four no more than five things that the company cares about. And of course, OKRs are publicly available to the entire company. These cannot be siloed. It is critical for Everyone in the in the organization to know what everyone else is working on. That's true from the CEO on down. Grading um is also fundamental to this exercise. Okay. Uh before diving into um uh my

I I pulled some of my personal OKRs from my days when I was PM on Blogger. Um I've got two more comments here. Um two more slides. Uh I want to talk about um Differentiating between the objective and the key results. And then talk a little bit about differentiating personal versus team versus company. Um Objectives should feel uncomfortable. Going into a quarter, you should be uncertain as to whether or not you can deliver on the thing that you've just said you will deliver. If you are certain you will nail it you're not pushing hard enough.

You're not thinking broadly enough. You're not Thinking big. The key results are the things that will ensure that you deliver on the objective. They have to be quantifiable. They have to lead to you being able to objectively grade them. Right? Don't set a an OKR that says, I will make Gmail better because in your opinion you've made it better because you made it prettier. But if that degrades usage, it leads to user attrition if it means people send fewer messages per minute, hour, whatever the metric is then you have not made Gmail better. And it's

impossible to objectively assess that. Don't say make it better. Make it say that, you know, you'll be able to uh help users respond to messages more quickly. Right? Well, that's easy. You can and And in fact, the better way to to say that is, I will uh reduce the average time to response sent from whatever it is now by 30%. Okay, personal versus team versus company. Um Personal OKRs pretty straightforward. What are you working on? What is most important to you? What are the things that at the end of the quarter you'll be able to

show you did. Team OKRs are the priorities for the team not just the bucket of all of the personal OKRs. Think about our example from John Doerr's slides of the 49ers. Personal OKRs that's Alex Smith, Frank Gore David Akers, Vernon Davis, Crabtree, Moss, you name it. It is each individual player has their own OKRs. And obviously, those are going to be quite a bit different. The team OKRs offensive coordinator, defensive coordinator PR head ticket sales, the New stadium person who's coordinating, you know, the the season tickets those are the team OKRs. Um And then the

company OKRs, big picture top level focus. Right? Win the Super Bowl. Um might have a few others there. Secure home field advantage through the playoffs might be another company level OKR. Things that help clarify for each individual what the most critical things are to the organization as a whole. All right. As I said, we're going to look at a couple of my OKRs from my Blogger days. Um some helpful context before we dive into this. At the time that I uh took uh took on Blogger um Blogger was the eighth largest web property in the

world. Still, incidentally, the top in the top 10 worldwide. Um So, that's a good trivia uh question. Next time you're at a bar and somebody Asks, you know, what is Google's third largest web property by page view traffic, not by visitors anymore. Um you know, first is obviously Google. Second, obviously YouTube. Almost no one knows that Blogger's the third. I take a little bit of pride in this as you can tell. Um but again, for context Blogger was a fairly ignored product internally, fairly forgotten at least within the certainly the US Tech echo chamber um

even though it remained this amazing uh huge engine of traffic worldwide. It was the middle of the recession. Um Google was looking at different products that it owned and trying to figure out whether any of them were revenue generating or could be revenue generating. And Blogger was making a little bit of money at the time, but not a lot. So, I was asked to come in and turn Blogger into more of a business. So in these personal OKRs that I'm going to share I've removed some of the specific details because for illustrative purposes, they're not

relevant and I don't want to put you in possession of any material information about some of our products. Um but my first objective was always accelerate revenue growth. Every quarter that I managed Blogger my first Objective the thing that I needed to think about every week that I was running that product make more money. Key results um first element. So, this was the first quarter that I was on Blogger. Uh at the time it was I think I'd counted 14 clicks for any Blogger owner who wanted to put AdSense on their site. Which is ridiculous.

Google owned Blogger. Google owned AdSense. I wanted to launch a monetize tab that would make that a one-click operation. So, first key result launch. Launch a tab. And and by the way, I had other uh uh key results as part of this that that quantified the number of publishers that we would bring into AdSense as a result of this. Um another very specifically uh written Key result. Implement AdSense host channel placement targeting. That's a bunch of words, I'll explain in a moment. To increase RPMs by XX%. Note here this key result is written in a

way that the core goal is increasing RPMs by a set percentage. If I had implemented and launched this host channel placement targeting, which Was just a way of bundling a bunch of uh similarly themed blogs into a targetable channel for advertisers. If I had launched that and had no impact on increasing revenue on those blogs then I'd get a zero or maybe a point two on this key result. The key for nailing this key result is ensuring that it had measurable uplift impact on revenues. And then finally, this was something that that uh really was

disciplined from Joe Kraus who hired me to run Blogger. He's who brought me over to Google Ventures. Um Launch three revenue specific experiments to learn what drives revenue growth. In late '08, early '09, there was a lot and we didn't know what we didn't know about making money on Blogger. So, Joe's thing was don't assume you have the answers. Do lots of experiments. Get into a normal rhythm of launching something to test Whether it has an impact. And if it does great. Do more of that. If it doesn't, well, file that away. Know that it

doesn't have an impact and don't do more like that. And then finally, this was a couple of quarters after I'd been on Blogger um finalize the PRD, product requirements doc for the Blogger ad network. Secure engine allocation to build it out the following quarter. So, this is again a measurable OKR or Key result rather in that it showed preparation work that I'd be doing in this case in Q4 so that something would get launched in Q1. Now, if I wrote the PRD entirely under my own control whether I did or I didn't but failed to

ensure that the engineers would be on on the hook for building it out in Q1 you know, at best I might get a 0.5 on this OKR. Um I would argue that far more Important to get the engine allocation than it is to just write the PRD. So, again in grading this um you know, that'll that'll come into play when we when we look at grading these and I will be grading them in just a sec. Next objective is to grow Blogger traffic by a certain percentage over our organic growth. Now, Blogger was this wonderful

gift in that It just it had hundreds of millions of visitors every month and would continue to have hundreds of millions of visitors a month whether I got out of bed or not. And they would continue to read blog posts and they would continue to visit through search and they would just it it just was this engine. At that point, it was turning 10 years old that year. It was inevitable that it would continue To get traffic and that as the internet continued to grow and more people came online almost certain the traffic would go

up over what it had been the year before. So, my objective here was to not sandbag and say, "Hey, grow grow traffic." But to ensure that I was the one moving the lever up. So, the key here is growing traffic over And above what it would do normally. So, key results to define this launch three features that will have a measurable impact on Blogger traffic. Improve our 404 handling. Very tactical. But if you think about a site that gets hundreds of millions of visitors, not all of them land on pages that still exist. People delete

posts, blogs go away um if we had wanted to create a worse 404 experience for users I don't think it would have been possible. At the time uh we were you know, blogs had their own style, had their own themes. Uh we were serving an error page that was not using the the default style. So, if you were familiar with a blog but landed on a 404, it looked like a system error page, not a a a an error page served up by that blog. We weren't giving you access to archives on that Blog. We

weren't giving you a search box. We weren't even telling you that this post belonged to a blog. It wasn't even a link to the blog on which the 404 was being served. So, we had looked at our our logs and we realized like, "Good lord, we're dumb as a box of rocks." Like, that's a a key area where we could dramatically improve the user experience and probably increase our traffic uh while we're at it. Uh improving Blogger's reputation was another objective I had. Um again, Blogger was about to turn 10 years old that year um

largely forgotten in the US uh even though it continued to have a lot of use um and we had not been proactive in communicating with press, with users, with partners and I saw part of my job as reestablishing Blogger as a as a a member of the community that we were Committed to to being engaged. Um so, that meant going out and speaking at events. Getting Blogger back out in front of people who pay attention to these things. Um again, measurable key result, I said I'd speak at three industry events. Um coordinate Blogger's 10th birthday

PR efforts. Um figured out that Blogger would turn 10 in August of that year, so it was Important to say, "All right, what you know, what are we going to do to draw attention to the fact that Blogger has reached this milestone and how can we how can we use that to further our other objectives? Increasing traffic, increasing revenue. So, some of the partnerships that I launched in conjunction with this 10th birthday uh were done specifically to drive revenue. Others were done Specifically to drive traffic. We partnered with a company that had built an iPhone

app um so that our mobile usage would grow. Each of these ultimately served a couple of different objectives. I pulled a list of the top end users as measured by traffic and personally sent each of them an email. Many of them had been on the platform For years and had never heard from anybody on our team. And I felt like it was critical that you know, in some cases we were making considerable money as they were. That was one of the really great things about working on that product. Any amount of money I made, they

were making more uh because Google would get a percentage of of whatever ads were running on their site uh if they were using AdSense and they would get the Majority. But I gave them my email address, my cell phone number, uh my Twitter account so that they knew how to get in touch if they needed help. And they no longer felt like they were um you know, the the using this nameless, faceless uh product anymore. Um very tactical issue for those that understand in US law the DMCA Um as a host of content, we are

legally obligated to take content down if it is alleged to be infringing the rights of another. Well, a lot of music blogs on Blogger uh the RIAA and the uh and its its uh international counterpart um are are fans of sending DMCA notices and the DMCA process if you were a Blogger user was pretty broken. And I felt this was hurting our Reputation with some of our uh our core constituents. Um not to mention it was just it was it was a degraded user experience overall whether you were the blogger, whether you were the reader

who wanted to read the post. Um so, I had a as a key result fixing this process and eliminating mis- should be another word in here mistaken Uh takedowns of entire blogs when the goal was just to take down a post. And then finally, improving our reputation getting us active on Twitter. Um that was where a lot of our core demographic were actively chatting with their readers, with each other um with the world and if Blogger had a problem, they would often rant on Twitter about us and you know, it was sort of into a

Void. So, I wanted to get us active on on Twitter as @blogger. It's convenient when the founders of Blogger are the founders of Twitter they can help get you started on that path and uh and and so we did that. So, let's look at the grades. Um as I said, accelerating revenue growth was always a key for me. Um and these grades are Written to be measurable. Launching the monetize tab, well, we did. We launched it was it was fairly significant change to the overall UI. Um so, that that got a one. Um here, you

know, as I said earlier, the the the goal is a 0.6 or a 0.7. But if there's a very tangible binary key result You don't give yourself a 0.7 because you never get a 1.0. In this case I did what I said we were going to do. It was considered critical. So, you get a one. Um implement AdSense host channel placement targeting to increase RPMs by X percent. Well, um we implemented a version of this and it didn't have much of an impact on RPMs. So, this this one was was was a Fail. Um launched

three revenue specific experiments. We launched three experiments um only two of them really told us anything about driving revenue growth. That was largely because the third experiment that we launched that quarter that we thought would tell us something interesting about revenue really in the end didn't. Um so, that was as much our Our error, my error in not crafting the experiment more effectively. Um and then finally, the the the PRD for the ad network. Wrote the PRD um was still at the end of this quarter doing some horse trading with my engine manager on uh

which engineers would be building it out but we were fairly agreed at the point at which we were grading these that uh that it would get built. Um there was still a few open Questions, so I I didn't consider this a one um but overall it was a 0.8. Now, quick note on scoring. You'll note that the 0.7 in white at the top here next to the objective is just an average of the four key results below. I've seen some get really cute in how they grade their OKRs by trying to determine by weight each

of the key results and say, "Oh, well, you Know, that that's twice as important as this other one. So, I'm going to give it a 0.7. Or of the objectives, this is the most important objective. So, that's that's 50 It's ridiculous. The goal of this is to provide feedback not only to the individual who owns the OKR, but to the group and the company ultimately. It is far less critical to get these down to the second, third, fourth decimal point. Um I always felt, continue to feel, grades don't matter except as directional indicators of how

you're doing. If you're spending more than a few minutes at the end of a quarter summarizing your grades, you're doing something wrong. Uh the work should go into doing um a delivering on the OKRs, not not grading them. Um On growing traffic, we said we'd launch three features that have a measurable impact on Blogger traffic. Um we launched two, each of which had a measurable impact. Uh so, that's good, but we said we were going to do three. And candidly, to be totally honest, we probably should have been able to do four or five. Um

So, so this this could have been quite a bit better. Um On improving Blogger's four four Handling, by the end of this quarter in which this this one was graded, um we had not proceeded past the mock stage. Was pretty sure we knew how we were going to do it. I just hadn't prioritized getting engineers to build it out. Um so, that was that was a big miss. On improving Blogger's reputation, um not least of which because so much of This was solely on my shoulders, um this was a much easier one to uh to

deliver on. So, let's talk a little bit more about grading. Um as I alluded to uh on the prior slide, simplicity is key. Um grade each key result, then average them up to give yourself an objective uh grade. And then average your objective grades to give yourself an overall grade for the quarter, if you like. Um I don't I ultimately think that much matters, but it is helpful to get a sense of, you know, where you are. I think in my best quarter, I hit a point It was north of 0.7. I don't think I

ever had 0.8 averaged across all of my objectives. Um Pretty sure I'm pretty sure that's right. I think it might have been a 0.78, somewhere thereabouts. That was a Phenomenal quarter, and the grade reflected that. Um I had a bunch of quarters between a 0.6 and 0.7. Um not because it was overly engineered to get there. That just was I was in the you know, sort of sweet spot of I was delivering on more than I was missing, but there was still a lot of room for improvement. And there were probably two or three quarters

where I was below a 0.6 where I was absolutely not delivering to A standard I felt was was good. Um High scores um are are possible, right? 0.8 to 1.0, again, if it's binary, and you did it, you get a one. Um I High scores are possible, they're unlikely. You know, I was I was going to say they're as unlikely as Mike Arrington rejoining TechCrunch, but then he did last night. So, sort of a uh uh uh uh uh maybe a bad example, I guess. Um When it happens, it's exceptional. Be proud of the accomplishment.

Be absolutely eager to share with your colleagues, your manager, your team, what have you. Um But I think the converse is probably more important to focus on, which is the low scores, right? 0.4 below. Um It's not failure. Right? One of the reasons when when Steven Levy talked about how this became so embraced so Early at Google was that this was data. The 0.4, the 0.3, the 0.2, the zero. Um It's not a failure, it's figure out what to stop doing. Or what did you learn? What did you not know when you said you would

do that thing that prevented you from delivering it? Well, do you have any control over removing that obstacle in the quarter ahead? Or is it immovable? If it's immovable, Go around it. Figure out either don't focus on that objective anymore or figure out different ways of achieving it. The scores ultimately benefit everyone in the process more by what they help you know what not to do, what to change, what to continue to do more of. Um That that is ultimately where they're most helpful. Um Company-wide scoring reinforces the commitment here. Um You Your commitment individually,

your commitment as a CEO to repeatedly grade at the company level, at the individual level, helps everyone understand that this really does matter. And as you get into that rhythm, it's just a byproduct of doing your job. It doesn't take a ton of time, but over The the quarters and years, they become extremely useful. So, the company-wide process. You're not if unless the company is only four or five people, you're not going to be grading everyone's OKRs in front of the entire company. That's just that's impractical. But it is practical that at least once every

quarter, sometime after the beginning of the next quarter, that the company's going to get Together and they're going to discuss how they did. And the owners of those individual OKRs, sorry, of the team OKRs. So, in a larger company, that's the SVP. Um you know, in in in a in a small growing company, it might be the you've got the the head of product, the head of engineering, and head of head of Marketing maybe or sales. Um Are each going to get up and say, "Okay, this was the this was the product OKR for last

quarter. Here's how we did. We got a 0.6. Here's why we got a 0.6." Um It's important to explain to everyone in the company why you got the grade you did and what you're going to do differently in the quarter ahead. This calibration Ends up being extremely useful for keeping everybody honest, ensuring the company's as transparent as it can be, but then also saying, "Okay, what did what did we learn?" For me, that was always the most important part. And then at this company meeting, the last part is setting the the upcoming quarter's OKRs. Now,

often there's continuity from the prior Quarter to the next quarter. Sometimes it's you know, we put that project to bed. We either finished it, shipped it, and it's done. Or we learned that that's not what we want to be focused on moving forward. Now, we're we're shifting our attention to this other thing. Let's talk about how to implement for a moment. At Google, We've we eventually built our own system. Uh it's not a complex system. It's a bunch of text fields with text boxes next to the text fields where you can enter in your grades.

And then the most sophisticated that it gets is there's a button you can push that says average my scores. And it does an immediate roll up of your key results and gives that objective a score. And if you want to get even fancier, then you can average your objective scores to get Your overall score for the quarter. Uh that's not something we make available to the public, so that's obviously not a tool you have at your disposal, but occurs to me that I think some relatively low-tech options are are just as valuable. Google Sites. You

can create a page a a a Sites page for the quarter, and each person can add themselves to that page uh where their name becomes the uh Their quarter's objectives and key results, just a bulleted list, very simple. And then at the end of the quarter, they can go back, they can edit their page, and they can add their grades next to it. Pretty low-tech. Um slightly more flexible might be a wiki. Um you could use Google Docs or Google Spreadsheets. I think that starts to get a little unwieldy, particularly as the The company grows.

Um but it is potentially a a way in which you could do it. The the one benefit of using something like Google Spreadsheets, of course, is that if you've got a lot of people, um the ability to calculations across all of them to get sort of directional uh indications of by team, by by department, what have you, might be really useful. Um At least as important as the tool you use is the communication you do about the commitment itself. What I mean by that is this needs to not be a half-hearted effort. It needs to

be clear to everyone in the company that for at least a 3-month trial period, ideally longer, that everyone in the company is behind this idea. At Google, it was always clear, not only because you could see Larry and Sergey's OKRs, it was always clear that from senior management all the way down, OKRs mattered. When I was in the product team, I I to tell you how much of an impact this this next uh email had, um I can tell you that when I was putting these slides together over the last week or so, I knew

when I got to this point in the presentation, I was going to have a screenshot of this email because over 2 years later, I still remembered the email. And the email is one from Jonathan Rosenberg, who used to be the the head of product at Google, still very close to Google and advises uh a bunch of folks in the company on a number of different things. But every quarter, he would send out a a version of this email. And this was the one that was particularly bad, which is why I remembered it. But the email

that he would send out every quarter within the first few weeks was a public shaming of whoever on his team had not updated their OKRs for the quarter. And this particular quarter, he didn't send it out until the beginning of November. So, it was already about 5 weeks into The quarter. And I'll I'll just read to you um a little bit. Our performance on OKRs this quarter was poor. As of today, nine PMs have still not posted their Q4 OKRs uh out of several hundred. That's a full month into the new quarter. The offenders are

listed below. And then right below, you can see right here, he has he has publicly shamed my colleagues. Two friends of mine are on this list as not having taken the time to tell the company what it was they were working on. He goes on to say, "I don't understand why it's so hard for so many of you to take just a few minutes and list your priorities for the quarter." Um So, last paragraph, "For now, I recommend we all torture the offenders below by Asking them in the halls why the heck they're late on

their OKRs." "For those of you who find the previous statement offensive, I'll be happy for you to chastise me as long as your OKRs are posted and current." So, the key here is when you're on a team and you receive an email like this from the person ultimately responsible for your in this case the product team, but this could easily have come from the CEO, this Could have easily come from, you know, if you're an engineer, the the head of engineering. I know this matters. Right? And and as an individual, I never wanted to be

on that list. I never wanted the person who was responsible for my team to identify me as as as holding the rest of the group back. Pretty powerful and simple. So, here's a timeline. Um We're coming into the beginning of November. Couple of thoughts on things that are really critical, I think. Um In the next month, I think there are two things that matter most for a company thinking about embracing OKRs for a 3-month trial. First is you got to be thinking about what your Q1 objectives are going to be. You probably also want to

be thinking about annual OKRs, but but tactically, thinking about what are the what are the immediate things that, you know, we're we're going to be we're going to be worried about. If I'm the CEO, I'm thinking just about what are the three or four most important things for 2013. The other thing that's really critical to be working on right now is deciding how you are going to capture What the OKRs are and how they're going to be shareable inside the company. This process will fail unless everyone in the company has a way to see what

everybody else is working on. So, I think it's critical over the next 4 or 5 weeks that you figure out whether you're going to use a wiki, whether you're going to use Google Sites, whether some engineer's going to Build a really simple Rails app that's just going to have a bunch of text fields and, you know, connect to their username and whatever, right? Keep it simple. Um don't over-engineer it initially. It's always easy to to migrate to something more more powerful down the road. I think the key here is just know what it's going to

be, so that if there's any kinks to work out, you get them well they worked out Well ahead of the end of the year. Now, into December, I think the company-wide uh objectives have got to be communicated for Q1 and for the year. That way, everyone else in the company you also, by the way, have to communicate what the hell OKRs are, so that there's a general understanding within the company. And by the way, this video Is being recorded, is available for anyone in your company to watch. So, if anybody has questions about what it

is, you could just as easily share this video with them. Communicate annual and quarterly Q1 objectives for the company. Now, it's up to the individuals in the company at the end of December, by the very beginning of January, to have their personal OKRs drafted. And that should be at least one Draft with a meeting with the manager and at least an iteration to get to consensus. This is that negotiation that John Doerr was talking about. Company-wide meeting, probably no later than about the second week of January, where the company-wide objectives are presented, and if appropriate,

depending on the size of your team, the team objectives within that company are presented. Now, throughout the rest of Q1, important that there's individual meetings happening between individual contributors and their manager at least once in a quarter. It doesn't need to be overdone, doesn't need to be weekly check-ins, but at least once or twice, so that there's a a general point in the process where everyone in the company has gone through that exercise of saying, "You know, this is what I said at the beginning of the Quarter I would be doing. By February 15th, am

I in the ballpark? If I am, fantastic, keep doing what I'm doing. If I'm not, I need to now recalibrate. Do I need to stop doing certain things so that I can deliver on the objectives I set? Or it it where am I going to miss and what can I stop focusing on entirely so that I can at least deliver on a few that are really important? And then finally, you repeat this whole process so that towards the end of March, now it's up to each individual to start thinking about grading their OKRs, team owners

uh start start uh coming up with the team scores, uh grades, and then the CEO leadership team are going to grade the company's OKRs. And in the beginning of April, There's going to be a company-wide meeting where you grade Q1's OKRs, explain why you missed or hit, and what you've learned about what's going to happen in Q2. And this just this process then just happens on a rolling uh quarterly basis. All right. Believe it or not, we are at The end of my formal um presentation here. So, I am going to pull up uh the

Q&A that have come in from uh from you all online uh and in the room. So, thank you for that. Um Let's see. What kind of tools do you use, Luke asked, for managing and communicating OKRs? Well, we talked a Little bit about that. Um I've seen it go through at least three revs at Google, right? Today, we have a a very simple tool um that is really no no more than I log in with my uh username, R Claw, and I click on the quarter, and I I add type in an objective, and then

under each objective, I have a button to add a key result. And I add add each individual key result. I'd show it to you, but again, it's internal, and it's Proprietary, so it's not the sort of thing that we would share in a workshop here. But you get the get the idea of of what it is. Very, very simple. At the end of the quarter, I go in and I click the score button. And that's when I uh I I am now just assigning a number in the text box next to each key result. And

as I mentioned earlier, then I have the ability to average them. Really sophisticated math going on there. Um I think in in most of the cases for for folks that are here, for folks that are watching online, I I think something as simple as a wiki, if you use a wiki internally, um I I think Google Sites would probably be enough. Um I I you know, I I think this is the sort of thing you could over engineer pretty quickly. Um the keys are not in my opinion are not the tool. It's The commitment to

being measurable, the transparency that everything is available internally to everyone in the company, and the grading process itself. I think beyond that I I think that the tool matters less. Um what's the cadence for planning uh for the planning portion of an OKR period? For example, is it all one week or spread over multiple? How many meetings do you have with whom in attendance? For reference, at Puppet Labs as one exact retrospective meeting, then an exact planning meeting, then team meetings, and exact review meeting all in one week. Um I think each company will likely

be a little bit different. In my own assessment, if I were if I were implementing this at my own company tomorrow, um I want this to be a relatively lightweight process. Um I don't think it necessarily needs to be in a meeting. I think this can be the sort of collaborative drafting that can happen where everyone can take a look at a doc, make some suggested changes, then have a meeting to wrestle over or negotiate over key points so that there can be some consensus. Um I think if if you you run the risk of

Meeting uh over meeting um the the the the process, which which I think could be problematic because then that's that's reinforcing the wrong thing. That the point here is to be doing the work, not meeting about doing the work. Uh so uh what would I what would I say there? I think um in terms of grading, I think it's important that people have That mid quarter check-in so they have a sense of where they're at. I think there's probably some uh I I I I think diligence, maybe an hour at the towards the end of

the quarter, and then a final uh meeting uh at the very beginning of the the following quarter to come up with a grade. Uh in terms of planning for the quarter ahead, um I think often that's a natural byproduct of the grading process Where I think about what did I get done last quarter? Are there things that are rolling from last quarter into the next? Um in which case that's, you know, sort of a copy and paste operation. Um Are there things that I wanted to be working on that I that I deliberately chose not

to do in prior quarters for prioritization purposes, and add them in. Um I think cramming it all into one week is is Probably a little too tight. I I would I would say at least bound it from a week before the end of the quarter to a week after the end of the quarter, maybe a little bit of a buffer on either side. I think that'll ultimately give you enough room in which to to give people the feeling that it's it's aiding in the planning process, not removing time from their schedule. So, Jeff asked, uh

how do you ensure OKRs roll up or or decompose down the Org structure? I I Uh trickle down might be better. Decompose suggests that they go away, um but I think I know what Jeff meant there. Um So, how how how do we do this? I talked in the in the deck about the sort of virtuous cycle, and I I really do I I is not just I believe it, I've seen it work. Um the way we ensure is you know, used to Be Eric when I was when I first joined Google, now Larry is

the CEO. Um there is a quarterly meeting, and it is given by Larry. And Larry says, "These are the most important things the company is working on. These are the grades for what we did last quarter, and now I'm going to have Vic and Urs and Susan and Jeff and each individual group head get up and explain How they did last quarter, uh and what they're working on next quarter. Now, that has a really important clarifying point. I think most people, particularly in startups, want to be working on the most important things. It was always

very helpful for me as I was going through the process of drafting my OKRs, that I would I would draft them at least in in in Draft form, and then I would attend the company OKR meeting, and I would do a reality check against the company OKRs, and would want to know, can I see in my OKRs the company OKRs? You know, are are is what I'm focused on reflected in what the company is most focused on. Um think about, you know, the the punter, The quarterback, the running back, is what they're focused on reflected

in the GM or the team owner? Um I can tell you, you know, at one extreme, there was one quarter when I was running the homepage at YouTube that I owned two of the company's five five or six OKRs. That was terrifying. Cuz I I didn't just my my OKRs weren't just reflected in the company OKRs, they were the company OKRs. Um that's Exciting, but it's also for a company of at the time, you know, about 30,000 employees, a little scary when you're you're the you're you're running the team that is responsible for a significant

proportion of what the company deems to be most important at the time. Um The reason that was a helpful process was if I couldn't see the company OKRs in my individual OKRs, Then I would, you know, that's when you take the the the red pen to your personal OKRs, and and revise. Um Jeff asked, "Cadence for checking process progress against OKRs?" I Different teams will do it differently. You might it might be more natural to do it every few weeks. Um particularly in larger teams, I think it's probably realistic that each Individual is going to

have to own the discipline of managing themselves, um and and maybe having only one or two check-ins in a quarter with their manager um specifically against OKRs. Um So, I was asked, was I was I using OKRs at Blogger before I joined Google? To to clarify, um I joined Google in '07. Google acquired Blogger in '03. Um and I was not obviously part of the Blogger team until late '08. Um by that point, Blogger was, you know, five, six years under Google's ownership, and and had really embraced OKRs. So, um sorry if I if I

didn't make that clear. Um At At last question, unless there's any uh others here in the room, I see several nodding heads that, you know, they've been jotting them down on on the on the docs, so that's great. Um At what point in a in a Company's evolution should the management team institute OKRs? Uh is it five people, 10 people, 15, more? Um Easy answer, as soon as possible. Um The sooner this is part of the DNA of the company, the sooner this is accepted, as soon as this is part of the normal rhythm that

the company gets in, The better off you'll be. Um you are there is always a point at which you can do this. I obviously I'm hosting this workshop because I believe every one of you should have some version of this approach in place at your companies. So, even if you've been around a year or two, uh it's not too late. But the longer you wait, the longer you are going to have habits form within the company about how things are done, and the more inertia you're going to have to overcome. So, even if it feels

a little artificial, if you're a company of five, the discipline that it brings is immeasurable. I committed to doing OKRs, even though at the Startup Lab right now, I'm a team of one. Now, Technically, I'm part of Google Ventures. We're a large and growing team, and I have a manager who I check in with periodically on progress. But for me, the discipline of looking ahead and saying, "These are the things I'm going to focus on this quarter. These are the things I'm not." Right? And then evaluating, "How am I doing?" Um that It's hard for

me to imagine doing my job without this discipline. Um At the end of the quarter, it makes it unbelievably easy to summarize what we've done, and and and and whether we did well or not. And I say we, I I mean me, the the for the Startup Lab. So, how many workshops did we schedule um this quarter? One of my objectives Was launching the video archives, which we launched you know, in the in the second week. I had been an objective in Q3. I'd done most of the heavy lifting in Q3, couldn't get the trigger

pulled on getting the archives up, so it slipped into Q4. My grade for Q3 wasn't as good. My grade for Q4 was less about launching cuz I knew that was going to happen in a week or two after the start of the quarter, was more important to focus on okay, now We're going to measure how much traffic are the archives getting? How many playbacks are there? How many people have asked for access to the archives? That's the sort of thing that starts to give you a sense of how that mentality can can really be embraced

and and drive action and and accomplishment. So, with that, um I'm going to wrap up. Um Let me again thank you for taking the Time. Um I teased you about not telling you how long this was going to go. I uh budgeted 90 minutes. I think I'm at about 88. So, uh you know, hey. Um I'll I'll score that a one on the OKRs. Um Please fill out the feedback form. Um for those of you uh here in person or watching online, um the the the the data is critical. As you Know, not only from

this presentation, but from others, we are very data-driven uh as a as a group. Um not just the quantitative score that gives us a sense of, you know, how we're doing on that one to seven scale, but also the qualitative. Um what takeaways did you take did you get from this event? Knowing what resonates helps me when I think about planning future Topics really know what to focus on and how to steer uh presenters to. So, that's that's really critical. Um Topics, again, uh as I said at the at the beginning of this workshop, if

you've got a topic that you would like to see us cover, uh I want to know what it is. I want to I want to hear from you. Um My job is to find the expertise and Experience at Google and make it available to you. Uh it's a great job. I love doing it. Um but I'd rather not be prescriptive and tell you what I think you should know. I'd rather know what you think you need to know uh so that I can make that available. Um Stay tuned on the announcement list. We've got a

lot of stuff coming up for the balance of Q4 and um starting to do a lot of planning for Q1 and really Excited about uh some some changes in in approach um and and format. So, uh looking forward to seeing you again online, in person. Uh until then, um thank you again. We'll see you soon. Bye-bye.