How Google sets goals: OKRs / Startup Lab Workshop

foreigner my name is Rick Clow I'm a partner at Google Ventures for those of you I haven't met before thank you for coming this morning uh thank you to those of you watching online we're here to talk about objectives and key results This is one of the critical elements of how Google handled its early execution and really grew the company and remains a pretty critical element of its DNA today. I came to Google a little over five years ago. I was part of a company like some of you. I know it's called Feedburner. Feedburner was acquired in June 2007.

The first day I joined Google was the first day of a quarter, which meant Google was already about a week and a half to two weeks into the okr process for that quarter. I know what the

okrs

meant. I was asked to write my

okrs

within my new team. I really had no idea what was expected of me. Certainly in pre-Google

startup

s I've never had a formal process for setting my

goals

or measuring

goals

, but over the last five years, I've established okrs as a Biz Dev lead as a product manager on a couple of different products and then , more recently, in my time as a partner at Google Ventures, the process had a really clarifying effect for me personally and for the teams I was a part of it helped me understand what mattered, just as importantly it helped me understand where I wasn't going to work today we're going to talk about what are good, how we implement them at Google, how to adopt this process in your own company I think one of the things that

startup

s often struggle with and one of the things that I went through a lot of time working with all of you is fighting the temptation to just say well, that's Google, right? different, we are not Google, as you will see in this presentation, Google was not Google when it was first adopted in Google.

Google was less than a year old when John Doer came to Google and proposed to Larry Sergey Marissa and others how to adopt this process in a very young and ambitious company um and I'm going to apologize in advance because I've committed at least two cardinal sins of uh de de in this case keynote not PowerPoint um so much so it wouldn't be surprising I, if someone from Cupertino came up to me and fought me for control of this remote, not only have I used slides that are almost entirely text, which is already Bad enough on its own, but I've also embedded an all-text presentation inside an ALT text presentation which is a new low for me as a presenter, but I think, as you'll see, there's a reason for doing that quick raising hands.

How many have read the book on Plex by Stephen Levy? It is a book about the origins and growth of Google until about two years ago. Well, maybe about 20. This is really a great book. Stephen had incredibly direct access not only to our founders and many early employees, but also to various people around the world. Google gave Stephen a very broad idea of ​​who we are, how we work, and talked about how in 1999, John Doer, who is a partner, as some of you may know, at Kleiner Perkins, remains a member of Google's board of directors to this day. today and wine. to Google to talk about a process that he had witnessed at Intel uh called objectives okrs and key results and Stephen in the book he talks about and I know those of you on the live stream can't see this.

I won't read the whole pages here, but it talks about how Dora got into Google and the okrs were an immediate success which, unlike what you might expect, where this could be seen as a kind of dibertization of a technology company, instead , it was immediately apparent to Google users that this data was correct, what this gave the company access to was a way to quantify rather than simply rate what they were working on and how well they were doing towards those goals. and periodically review whether they met the expectations of what it was that they were trying to do and highlighted an element here, a couple of things that are fundamental to how we think about okrs and I think you'll see the themes resonate throughout this discussion.

It was essential that the okrs be measurable. The employee didn't say that I will make Gmail a success, but I will launch Gmail in September and have a million users by November. It's really critical. There are two different points in the key result released in September and one million users by November. both measurable, the key here is that not only is the objective clear, but also the means by which you would rate whether you have met them or not, that objective is really the key here, and Marissa goes on to say that it is not a key result unless having a number the embodied ambition of okr sanctions the ability to take risks says door um and then goes on to talk a little bit more about um what else makes okrs key and it's important to note that everyone's okrs from Larry and Sergey from now on are public within the company, everyone is part of its board of directors and what is called Moma Stephen refers to him by name here in the book.

If you walk into Moma, you look for an employee right next to their cell phone, their email address, their title. it's a link to their okrs which means that not only can you see what they're working on this quarter, but you can also see what they were working on last quarter before you can see their grades and we'll talk more about the process. about how to qualify them and why different approaches can be beneficial in your companies, so I got the real presentation from John Doer. This is the embedded text presentation I talked about a few minutes ago.

But I actually got the original presentation from 1999 and I think it helps. I'm not including the entire deck. um, I tried to take what I felt were the most critical elements that illustrated both how we think about them at Google and how I think you should place them. in context to understand how they could be applied in your company, so the goal of this deck imagine I'm John doer for a moment, I present this to Larry Sergey Marissa, he started his deck with his okr for that meeting and the goal was me . I will develop a workable model for planning measured by three established key outcomes.

Note how each of these three is inherently measurable. Finish the presentation on time. I have not told any of you how long this

workshop

will take. so I think I have a good chance of finishing on time, since only I know what on time is to complete a sample set of three-month objectives and key results, so at the end of the meeting they were going to have a set of their okrs. Three, the fact that management has agreed to institute a trial system over a three-month period is pretty straightforward, so John goes on to give examples of an OKR in the context of a football team.

Now I'm a huge 49er fan, so I'm going to use the 49ers as my example. I think the key here is one of the things that when I've talked to many of you individually you've struggled with: how to connect individual goals to team goals to company goals and I think John's example from the Football actually helps a lot to clarify what that looks like, so general manager, I'll say the team owner, Jed York, his job is, he's going to win the Super Bowl, and fill the stands at 88 now the 49ers would tell you that We're going to be sold out all season long, but these are key measurable results with a goal that in this case is defined as the GM making money for the owners. what does that look like now filter that, the head coach noted that he doesn't care about filling the stands, all he cares about is winning the game well, winning the Super Bowl and the way he's going to win the Super Bowl is he's going to to have a passing attack now, in the case of the 49ers, it's a little complicated if Alex Smith is going to throw for 200 yards in a game, but maybe it's more important that we get the ball to Frank Gore, so you could say That you already know.

We're going to cumulatively rush for over 200 yards or number three in defensive stats given the 49ers defense, you could say things like we're going to have a positive net carry differential where you know we're going to have more interceptions or fumbles. comebacks than the other team, right, these are measurable things that at a high level, in this case, the head coach focuses on, now build up to the previous comment, which was: you know the GM is going to win. the Super Bowl, fill the stands, but obviously there are some things that the general manager doesn't care about and that the head coach is intensely focused on.

Go from head coach to public relations. His responsibility is to fill the stands. We are going to hire two colorful players. I'm not sure if I was in charge of PR for the 49ers that would be my approach, but if I think back to '99, I think that's when they hired Dion Sanders, so maybe that was exactly the strategy at the time. , um get media coverage. Highlighting key players, obviously this is a way to generate excitement among your fans, which may be a byproduct of what the manager focuses on, but it is absolutely one hundred percent of what the relations function focuses on. public.

Here now filter that further down, um defensive coordinator. All the defensive coordinator focuses on is obviously the defensive stats that keep the other team under 100 yards, passing, intercepting at least twice in the game, recording at least three sacks, you name it, To extend that analogy, whatever might be important given the players that that. The defensive coordinator has the offense at his disposal, on the other hand, of course, he doesn't care about interceptions except not throwing them, but now they're going to set a 75 percent completion rate goal. I should point it out here for those of you following along.

A 75 percent completion rate in football would be pretty amazing, especially in a Super Bowl, um part of a goal. We'll talk about this again as we get into the mechanics of how okrs are defined. It's just going to be a little uncomfortable. You don't always want to be on a zero to one rating scale, you don't always want to get a one, in fact the goal should be 0,6,7, so you should always have goals that are a little bit out of reach and that feel a little uncomfortable, so you always strive to do more to do better than maybe you did in the previous quarter or you think it's reasonable, in the context of the current quarter, think about now that we've talked about the offense defensive there.

It's a special teams unit, right, they're not focused on interceptions, they're not focused on how well the quarterback does, they're specifically focused on running back kicks, trying to block the punt, trying to block the goal. field, um, they have very specific objectives. who are completely different from the other guys, same with the new staff, same with the scouts, right, they might not even be on the field, they might be out watching spring practice, for a group of colleges, the idea here and what, what, uh, John Dora. I was trying to reinforce to the Google team at the time that if you put all the things together from this Foundation level into your goals and key results are reflected in the bottom line priorities of the company, but not every single one of them will be considered. a company priority, right? um uh the new staff here has a key takeaway from three Sunday articles or it could be an interview on ESPN or it could be something like that is that that's not going to be a company.

Okay, that's not going to be the focus of what the 49ers are thinking, the general manager or team owner isn't going to wake up every day saying, "You know how we're doing in those Sunday functions," but you better believe that. The head of the news team, the public relations team within the 49ers organization, is thinking absolutely only about those things now, what is that, beyond the company's goal, which again relates to filling the stands? correctly so you can start to see how you can do it quite clearly? connect these dots even if not all the key objectives and key results eventually appear in the company uh okrs now um I want to talk about why you use objectives and key results now this was John trying to convince Google Google has now lived these uh This is this approach for 13 years.

For those of you who have not yet committed to adopting this method of measuring both your goals looking forward and your rating looking back, let me talk about some of the things that Les to John's bullets here are. he was trying to make the case, let me tell you about a

google

r who has lived this for five of Google's 13 years using okrs um, what is he, what does he do? It is absolutely amazing at imposing discipline on the organization, it helps me understand individually. what I'm working on and why also helps me make conscious decisions about what I'm not working on, which is often just as important.

Make sure that by making these okrs public, all my team members and anyone interested in the company can see it.exactly what I'm working on I can see what my priorities are, why is this useful? Well, I'll give you an example when a team at Google criticizes me and wants to, say, when I was the PM of the YouTube homepage, someone pings me and says hello. We would love to promote this new product on the YouTube home page. Well, everyone would love to promote things on the YouTube homepage. The person who is going to make that presentation could help you know what my priorities were for the quarter we were working on.

They can know immediately, before even talking to me, what my likely response will be, if what they are promoting is in any way connected to the things I care about for that quarter, they have a very easy case to make. They are helping me meet my goals, if on the other hand it is a distraction, at least they know that going into the conversation they are unlikely to get much traction on that quarter now it might be helpful to plant the seed with me, say at the end of October , so when you start thinking about Q1 OKR, have them on my radar by writing down objectives and key results, specifically key results that are inherently measurable.

We're establishing how you're going to measure what you're working on and in the process of writing those okrs and we'll talk about how, in particular, how individuals set their okrs, you come to a real consensus from the individual to the manager, to the manager, to the team. , the company, depending on how big it is, can be anywhere from an individual to the CEO, where everyone understands on a pretty basic level what it is we're trying to do and, more importantly, how to measure whether we're trying to do it. doing. Are we there or not and how close are we and then finally focus the effort.

I think we'll talk more about that in a moment. What I really like about how this process ends up being adopted is that there is a really virtuous cycle that happens. Company goals are usually set by the CEO, the board of directors, the leadership team, who should have a pretty clear sense in any given quarter of what are the things that matter most to the company, and take my example of feedburner, back in the days when De Costello was our CEO had a somewhat comical way of setting it up, but the mission was to get all the feeds right if we could get all the feeds to syndicate all these RSS feeds through the Feedburners platform where we could measure all the traffic that was happening and give publishers a way to monetize that traffic was a very clear statement of what we were trying to do, which meant that when I went to New York and came back and said, " Well, this or that company wants us to do X, that's always the question." Always followed was that thing that got us closer to getting all the feeds and if it was a distraction it could be an interesting income opportunity but it didn't meet the main focus of what we were trying to achieve, then the answer was no.

These corporate objectives are established and generally communicated throughout the organization. Individual goals, on the other hand, are usually what the individual wants to work on and what the manager wants that person to work on. Instead of being a top-down discussion, it often ends up being an argument. It's a cycle: we generally know what the company wants to achieve as an individual. I want to work on important things, so I will find the things that are within my sphere and that I can do that will be most beneficial to the company. I'll make a list and then I'll communicate it to my manager or the CEO or whoever I report to and we can have a little give and take now.

The interesting thing in this process is that occasionally the things that individuals want to be working on could end up opening up new discussions about what the company should be doing, a good example of that on the call when the Google engineer named Paul Buchheit was really upset with the status of male clients on Linux, so he started thinking that maybe he could do Over time, Gmail effectively became its own practice group at Google and became one of Google's core strategic goals, which it then had a way to create your own objectives and key results for people who were working on Gmail, so let's talk about the one-on-one communication meetings where, as an individual contributor, I talked to my manager, my boss, the CEO, whoever That is, what I want to work on and what I think is the best use of my time, the one-on-one meetings at the end of the quarter and at the beginning of the quarter.

It's best for John to talk here about negotiating what those key outcomes might be. I think it's actually a pretty good term. because ultimately you're offering what I want to work on and what I'm hearing is what we want you to work on and somewhere in the middle we find the best combination of the two in a company-wide meeting. Company okrs communicate and team owners identify what they are going to work on. There is also a qualification process that happens here that we will talk about in such a typical cycle. I'm not going. I'm going to stop at this timeline just because I wrote one up specifically for the period that we're in at the end of October and I think it will help clarify what the general pace of um, you know, when you should be thinking about what steps in the process.

We'll be back to get back to that, some basic hygiene. You don't want to have 12 goals. It's too much for one person to work on. If you think that a quarter is effectively equivalent to 12 weeks if you had a total. With only one week to dedicate to any given goal, it's unlikely that you'll be able to move the needle much on any of them, so you really want to limit the number of goals you have and make sure you're very clear about what the goal is. The key results of that objective are not intended to limit your objectives.

Don't fill every objective with 12 key results. You want to be very clear. Consider that ambitious, but I will tell you personally that there was a quarter where I had seven goals. I have never been more exhausted and mentally drained by the amount of intellectual juggling that was necessary to keep all those different goals clear in my head and make a keeping track of where he was with everyone and who he trusted. and what I needed to coordinate it was just that it was too much and in the end, um uh, it finally took its toll. I mean, it was clear at the end of that quarter that I had been stretched too thin. um, John here has a number.

I don't know if the number here matters as much as the overall intention, which is sixty percent of the goals, should come from the bottom up. um. I think I would phrase this a little differently. I would say that more than half of the company's goals should come from individuals throughout the organization if there is too much top-down dictation. It's going to be difficult to inspire many of those people to work on things because they're going to tell me what to do instead of telling you what they think is the best use of their time and talent.

There's a comment here about everyone having to mutually agree without dictating. I think it's really just another way of saying what I want. said a minute ago um interesting they say one page is better, now we do all this electronically so there's no page length issue um there's a really important point here. This surprised some of you when I spoke before giving this

workshop

. a performance evaluation tool, what that means is that in the annual performance review cycle that Google goes through, okrs are not a standard element of that evaluation, let me be clear when I was reviewed in each of the last five years, when I sat down and you know, I wrote.

My summary of what I did the previous year, what I think it means for the company, whether I wanted to promote or not, at no point were my quarterly okr points taken into account in that evaluation, now there, it's always there. It is possible to include your okrs as part of that package that I found individually. It was always tremendously helpful because it really gave me an overview of what I've been doing over the last year. I was able to see the last four quarters of okrs. I could look at the grades again, we'll talk about the grading process in a moment, and I really knew within a few minutes what were the most important things I had worked on, what were the most dramatic results as a byproduct of that work, and then I could summarize my contribution. to the company, my impact in a way that before, when I thought about you, you know the previous startups before I got to Google, it was much harder to remember what I did six nine 12 months ago um I mentioned this in passing at the beginning, your goal should be 0.6 to 0.7 uh on a scale of zero to one, if you're consistently getting ones, you're not crushing them, you're locking in anything under 40 percent, you're definitely in the red. it means you missed the key result completely and that should be a forced feature to re-evaluate whether that's a goal you want to continue working on or not, okay, last slide of the built-in slide, we promise we'll keep going.

After this, John talks about this having an alignment function and ensuring that everyone in the organization is working towards the same set of results. I agree with that, that was certainly my experience at Google, it was very easy to see on my team. members where we were working on complementary things where we were working on dependent things where there was something I was doing and I needed three other people to deliver something and therefore keep track of where they were in their progress it helped me understand how likely I was to fulfill my uh okrs um, pretty organically ensures that the organization is in tune with what everyone is doing um and that virtuous cycle that I talked about a few slides ago is wonderful, I use the word organic a minute ago, it's The best thing I can think of here is that it simply ensures that you are working on things that support what the company is focused on, as are the people sitting next to you and in front of you and the end of the hallway, and there is trust and a sincere attitude. comfort that comes from knowing you're not treading water, you're not walking on molasses, you know everyone shares those commitments, um, weirdly, John even says here in the last bullet that okrs are fun, which I think is open-ended. to some interpretation, but but I think there are elements of how to make this part of the team process that I will talk about later.

Well, the keys to good okrs, just in terms of pace and process, okrs are set quarterly and annually. The okrs are not set in stone if you set them in December and in May you have evaluated them, you say you know things we thought were true in November, December is just not proving to be true, you obviously don't feel beholden to CML, check them out. are are directional AIDS no um no blinders um like I've said and I'll continue to say I think this is one of the critical elements of okrs are measurable the fact that I know whether or not we ship in September like I said, well that's measurable if we ship it in September.

Great, give it a moment, give it a one if, in fact, that was the entire key result. Talk a little more, but it's important to note here that there are okrs. For each individual there are team level okrs that summarize what the group of people is working on and then there are company level okrs that tend to reflect the three most important four, no more than five things that the company is interested in and, Of course, the OKRS are publicly available to the entire company. These cannot be isolated. It is essential that everyone in the organization's operation knows what everyone else is working on.

That's true for the CEO. Grading is also essential for this exercise. Okay, before we dive into um. Oh, I pulled some of my personal okrs from my days when I was a PM at Blogger. I have two more comments here. Two more slides I want to talk about. Differentiate between the objective and the key results and then talk a little. A little about differentiating personal, team and company objectives. um you should feel uncomfortable going into a quarter, you should be sure whether or not you can deliver on what you just said you will deliver on if you're confident that you'll achieve it. you're not trying hard enough you're not thinking big enough you're not thinking big key results are the things that will ensure you meet the goal they have to be quantifiable they have to lead to you being able to objectively grade They're right, don't set an okr that says "I'll make Gmail better" because in your opinion you've made it better because you've made it prettier, but if that degrades usability, it will cause user attrition if that means people send fewer messages per minute.

Whatever the metric, then you haven't improved Gmail and it's impossible to objectively evaluate. and in fact, the best way to put it is that I will reduce the average response time sent from whatever it is now by 30 percent. Okay, staff versus team versus company. um, personally, thecar is quite simple. What are you working on? What is the most important? For you, what are the things that at the end of the quarter you will be able to show that you did the team's okrs? They are the priorities for the team, not just the bucket of all the personal okrs.

Consider our John Doors example. 49ers personal okrs that's Alex Smith Frank Gore David Akers Vernon Davis Crabtree Moss whatever, each individual player has their own okrs and obviously those are going to be quite different Teemo offensive coordinator kr defensive coordinator head of public relations sale of tickets the new The person who coordinates the stadium, you know the season tickets, those are the team okrs and then the company that cares about the big picture, the top level, the right approach, wins the Super Bowl, eh, might have a few others in there, securing home field advantage during the playoffs might be another level of company. okr things that help clarify for each individual what are the most critical things for the organization as a whole.

Well, like I said, let's look at a couple of my okrs from my blogging days, some useful context before we dive into this in the time it took me to devote myself to Blogger um blogger was the 8th largest web property in the world and, said Incidentally, it's still in the top 10 worldwide, so that's a good trivia question the next time you're at a bar and someone asks you what it is. Google's third largest web property by pageview traffic, not visitors you know, the first is obviously Google, the second is obviously YouTube, hardly anyone knows that bloggers, the third, I take a little pride in this, like You can see, but again, for context, blotting paper was a pretty ignored product internally, pretty forgotten, at least within the US tech echo chamber, even though it was still a huge traffic driver. around the world, was in the middle of the recession.

Google was looking at different products it owned and trying to figure out if any of them were generating revenue or could generate revenue. and Blogger was making a little money at the time, but not much, so they asked me to come and turn Blogger into more of a business, so in these personal okrs that I'm going to share, I removed some of the specific details because to Illustrative purposes are not relevant and I don't want to give you any material information about some of our products, but my first goal was always to accelerate revenue growth every quarter I ran blogger, my first goal. what I needed to think about every week I was running that product, make more money, key results, first item, so this was the first quarter I was on Blogger at the time.

I think I had counted 14 clicks for any blogger. owner who wanted to put AdSense on his site, which is ridiculous. Blogger owned by Google AdSense owned by Google. I wanted to launch a monetization tab that would make it a one-click operation so the first key result is to launch a tab and by the way, I had another uh. Key results as part of this Quantified the number of publishers we would bring to AdSense as a result of this other key result written very specifically Implement AdSense host channel location targeting. There are a lot of words that I will explain in a moment to increase.

RPM in XX percent. Note that this key result is written in a way that the main goal is to increase RPMs by a certain percentage if you had implemented and launched this targeting by host channel location, which was just a way to group a bunch of blogs with similar topics . on a channel targeted to advertisers, if you had launched it and it had no impact on increasing revenue on those blogs, then you would get a zero or maybe a 0.2 on this key result. The key to achieving this key result is ensuring it is measurable. increased revenue impact and ultimately this was something that was really disciplined by Joe Krauss who hired me to run the blogger, he's the one who brought me to Google Ventures and launched three specific revenue experiments to learn what drives revenue growth. income at the end of 08 and beginning of 09.

It was a lot, you know, we didn't know what we didn't know about how to make money on Blogger, so Joe's thing was to not assume that you have the answers. There are many experiments that fall into a normal rhythm of launching something to test if it is successful. an impact and if it works well, do more of it, if you don't file it well, know that it doesn't have an impact and don't do more things like that and finally this was a couple of quarters after I did it. been at Blogger finalize the PRD Product Requirements DOC for Blogger Ad Network Secure Edge Allocation to build it next quarter, so this is again a measurable okr, a key result rather because it showed the prep work What would I be doing in this case? in the fourth quarter for something to be released in the first quarter now if I wrote the PRD completely under my own control, whether I did it or not, but I couldn't ensure that the engineers would be on hand to build it in the first quarter, You know?

At best you could get a 0.5 on this okr. I would say it's a lot more important to get the engine mapping than it is to just write the PRD so again as you grade this um you know that's going to come into play when we look at how to grade them and I'll grade them in just a second. The next goal is to increase blogger traffic by a certain percentage of our organic growth. Now Blogger was this wonderful gift because it had hundreds of millions of visitors every month and I would continue to have hundreds of millions of visitors a month whether I got out of bed or not, and they would continue to read blog posts and they would continue to visit me through search and just It was this engine at that time that was running. 10 years that year, it was inevitable that I would continue to receive traffic, and as the Internet continued to grow and more people got online, it was almost certain that traffic would increase from what it had been the year before, so my goal here was do not make sandbags. and say, "hey, increase traffic," but to make sure that I was the one pulling the lever up, the key here is to increase traffic above what I would normally do, so the key results to define this launch are three features that will have a measurable impact on Blogger.

Traffic improves our 404 handling very tactically, but if you think about a site that receives hundreds of millions of visitors, not all of them land on pages that still exist. People delete posts. Blogs disappear. If we had wanted to create a worse 404 experience for users. I don't think it would have been possible at the time we knew that blogs had their own style, they had their own themes, we were posting an error page that didn't use the default style, so if you were familiar with a Blog but it came in a 404 looked like a system error page, not an error page provided by that blog.

We were not giving you access to the archives of that blog. We weren't giving you a search box. We weren't even telling you this. it belonged to a blog, it wasn't even a link to the blog where the 404 was served, so we looked at our logs and realized we are as dumb as a box of rocks as if that was a key area where we could improve dramatically user experience and probably increasing our traffic uh while we're at it uh improving the reputation of the blogger was another goal I had um again the blogger was about to turn 10 that year largely forgotten in the US despite that continued are very useful and we had not been proactive in communicating with the press, users and partners, and I saw that part of my job was to reestablish the blogger as a member of the community that we were committed to, so that meant go out and speak at events get the blogger back in front of people who pay attention to these things again key measurable result I said I would speak at three industry events um coordinate the blogger's 10th birthday PR efforts I discovered that the blogger would be turning 10 in August of that year, so it was important to say, "Okay, you know what we're going to do to draw attention to the fact that the blogger has reached this milestone and how we can use it to further our other goals: increase traffic, increase revenue.

Some of the partnerships I launched in conjunction with this 10th birthday were done specifically to generate income, others were done specifically to generate traffic. We partnered with a company that had created an iPhone app to increase our mobile usage. Each of them ultimately served For a couple of different goals, I pulled out a list of the top end users measured by traffic and personally emailed each of them. I had been on the platform for years and had never heard from anyone on our team and felt it was essential that you know that in some cases we were making a considerable amount of money as they were one of the best things about working on that product.

Whatever money they made, they made more because Google would get a percentage of the ads that ran on their site. if they were using AdSense and they would get most of it, but I gave them my email address, my cell phone number, my Twitter account so they would know how to get in touch if they needed help and they would no longer feel like they were. The use of this nameless and faceless product is already a very tactical issue for those who understand the dmca in US law as a large amount of content. We are legally required to remove content if it is alleged to infringe another's rights, as well as a lot of music. blogs on Blogger uh the riaa and the uh and it's their international counterpart um they are big fans of sending out dmca notices and the dmca process if you were a blogger user was pretty broken and I felt like this was damaging our reputation with some of our uh our core components. , not to mention it was just a degraded user experience overall, whether it was the Blogger, whether it was the reader who wanted to read the post, so my key result was to fix this process and remove it.

Be another word here, wrong, deleting entire blogs when the goal was simply to delete a post and then ultimately improve our reputation by staying active on Twitter, which was where many of our core demographics were actively conversing with their readers among themselves. with the world and if Blogger had a problem, they would often rant on Twitter about us and you know, it was kind of a vacuum, so I wanted to keep us active on Twitter, since on Blogger it's convenient when the founders of Blogger are the founders. from Twitter, they can help you get started on that path and uh, and so we did, so let's look at the ratings.

Like I said, accelerating revenue growth was always a key for me and these scores are written to be measurable when launching the monetization tab. Well, we did it, we released it, it was a pretty significant change to the overall UI, so it got one. Here, you know, as I said before, the goal is 0.6 or 0.7, but there is a very tangible binary key result. don't give yourself a 0.7 because you'll never get a 1.0, in this case I did what I said we were going to do, it was considered critical so you'll get a um Implement AdSense Host Channel Location Targeting to Increase Results RPM by X percent, well we implemented a version of this and it didn't have much impact on RPM, so this was a failure.

We launched three revenue-specific experiments. We launched three experiments. Only two of them actually told us anything about how to drive revenue growth. It was largely because the third experiment we launched that quarter that we thought would tell us something interesting about revenue actually didn't, so that was our mistake, my mistake in not elaborating on the experiment more. effective, um and finally the PRD for the ad network wrote that the PRD was still at the end of this quarter negotiating with my engine manager about which engineers would build it, but we were pretty much in agreement on the point where we were rating these. would be built, there were still some open questions so I didn't count this as one, but overall it was a 0.8.

Now a quick note on scoring, you'll notice that the blank 0.7 at the top here next to the target is just an average of the four key results below, yes I've seen some get really cute in the shape in which they rate their OKRS by trying to weigh each of the key results and say, "well, you know that's twice as important as this other one." so I'm going to give it a 0.7 or the goals, this is the most important goal, so it's 50, it's ridiculous, the goal of this is to provide feedback not only to the individual who owns the okrbut to the group and the company.

Ultimately, it is much less critical to reduce them to the second third fourth decimal point. I always felt that I still feel that grades don't matter except as directional indicators of how you are doing if you spend more than a few minutes at the end of a quarter summarizing your grades you are doing something wrong work should focus on meeting the okrs no on not rating them um on growing traffic we said we had launched three features that have a measurable impact on Blogger traffic we launched two each of which had a measurable impact so that's good but we said we were going to do three and honestly, to be completely honest, we probably should have been able to do four or five, so this might have been a little better. about improving bloggers for handling at the end of this quarter in which this was rated. um, we hadn't gotten past the mock stage.

I was pretty sure we knew how we were going to do it. I just hadn't prioritized having the engineers build it, um, so it was a big hassle to improve the reputation of the bloggers, um, especially since a lot of this fell solely on my shoulders, it was much easier to accomplish, so let's talk a little more. about grading um as I mentioned in the previous slide Simplicity is key rate each key result, then average them to get an objective score uh and then average your objective scores to get an overall score for the quarter if you want.

Ultimately, I don't think it matters much, but it's helpful to have an idea of ​​where you are. I think that in my best quarter I reached a point above 0.7. I don't think I've ever averaged a 0.8 on all my goals. pretty sure, I'm pretty sure that's true, I think it might have been a 0.78 somewhere out there, which was a phenomenal quarter and the rating reflected that, um, I had a bunch of quarters between 0.6 and 0 .7 um, not because I was too designed to get There I was, I was in the sweet spot where I was meeting more of what I was missing, but there was still a lot of room for improvement and there were probably two or three quarters where I was below 0.6 in which it did not meet a standard at all.

I felt it was good. High scores are possible. Right, 0.8 to 1.0 again if it's binary and you did it, you get one. High scores are possible. You're unlikely to know what he was going to. to say it's about as unlikely as Mike Arrington rejoining TechCrunch, but he did last night, so maybe that's a bad example I guess, and when it happens it's exceptional, be proud of the achievement, be absolutely eager to share it with your colleagues , your manager. your team, what do you have? um, but I think it's probably more important to focus on what the low level is. you get the point just four below, um, it's not a failure, one of the reasons Stephen Levy talked about how this was so accepted so early in Google was it was data 0.4, 0.3, 0.2, zero, um, it's not a failure, it's figuring out what. stop doing or what did you learn that you didn't know when you said you would do that thing that prevented you from doing it well do you have any control to remove that obstacle in the next quarter or is it set in stone if it is set in stone? rounded solution or not focusing more on that goal or figuring out different ways to achieve it, the scores ultimately benefit everyone more in the process so they help you know what not to do what to change what to continue doing more than um that's it, ultimately where they are most useful um the company-wide score reinforces the commitment here um you, your commitment individually, your commitment as a CEO to repeatedly score at the company level at the individual level helps everyone understand that this really matters and as you get into that rhythm it's just a byproduct of doing your job, it doesn't take a lot of time but as quarters and years go by they become extremely useful so the entire company process is not a Unless the company has only four or five people you are not going to rate everyone's okrs in front of the entire company, that is impractical, but it is practical that at least once every quarter, sometime after the beginning of the next quarter , the company gets together and we're going to discuss how they did and the owners of those individual okrs I'm sorry for the team okrs, so in a larger company that's the senior vice president, you know, in a small growing company, It could be the one you have the boss. of the product, the head of engineering and the head of marketing, maybe or sales, um, each one will stand up and say, "Okay, this was the right product for the last quarter," this is how we did it, we got a 0.6, here's why we got a 0.6 Well, it's important to explain to everyone in the company why you got the grade you did and what you're going to do differently in the next quarter.

This calibration ends up being extremely helpful in keeping everyone honest and ensuring the company is as transparent as possible. be but then also say okay, what did we learn for me, that was always the most important part and then in this company meeting, the last part is setting the next quarters, now there is often continuity from the previous quarter to the next quarter sometimes. is that you know we put that project to bed, or we finished it, we sent it and it's done or we learned that that's not what we want to focus on moving forward, now we're shifting our attention to this other thing, let's talk about how you do it. implement for a moment at Google, we finally built our own system, uh, it's not a complex system, it's a bunch of text fields with text boxes next to the text fields where you can enter your ratings and then the most sophisticated one that becomes.

There's a button you can press that says average my scores and it does an immediate summary of your key results and gives that goal a score and if you want to get even more sophisticated, you can average your goals' scores to get your overall score. For the quarter, uh, that's not something that we make available to the public, so it's obviously not a tool that you have at your disposal, but it occurs to me that I think some relatively low-tech options are equally valuable Google sites in which you can create a page. sites page for the quarter and each person can be added to that page where their name becomes their quarter goals and key results, just a very simple bulleted list and then at the end of the quarter they can come back, they can edit their page and they can add their ratings next to it, pretty low tech.

A Wiki could be a little more flexible. You could use Google Docs or Google Sheets. I think that's starting to get a little unwieldy, especially as the company grows, but it's potentially one way you could do it, the only benefit of using something like Google Sheets, of course, is that If you have a lot of people, the ability to do calculations across all of them to get sort of directional cues by team, by department, what do you have? could be really useful um at least as important as the tool that you use is the communication that you do about the engagement itself.

What I mean by this is that this should not be a half-hearted effort, it should be clear to everyone in the company that for at least a three month trial period, ideally longer, that everyone in the company is behind this idea. at Google, it was always clear, not just because you could see Larry and Sergey's okrs, it was always clear that from the top management to the okrs mattered When I was on the product team, I want to tell you the impact the following email had . I can tell you that when I was putting these slides together over the last week, I knew when I got to this point in the presentation.

I was going to have a screenshot of this email because more than two years later I still remembered the email and the email is one from Jonathan Rosenberg, who used to be the head of product at Google, still very close to Google and advises a group of people. in the company about several different things, but every quarter I sent a version of this email and this was the one that was particularly bad, so I remembered it, but the email I sent every quarter for the first few weeks was a public embarrassment to whoever on your team hadn't updated their okrs for the quarter and this particular quarter you didn't submit it until early November, so it was already about five weeks into the quarter and I'll just read for you, a little bit, our okrs performance this quarter. was poor as of today at 9 PMS they still haven't posted their 4th quarter okrs, out of several hundred, it's a full month into the new quarter, the offenders are listed below and then just below you can look here, you have publicly shamed To my colleagues, two friends of mine are on this list, you haven't taken the time to tell the company what you were working on and keep saying: I don't understand why it's so hard. for many of you to take just a few minutes and list your priorities for the quarter, um, last paragraph for now, I recommend that we all torture the offenders next by asking them in the hallways why the hell they are late on their okrs for those of you who find the above statement offensive, I will be happy to be punished as long as their okrs are posted and updated, so the key here is when they are on a team and receive an email like this from the person ultimately. responsible for your, in this case, the product team, but this could have easily come from the CEO, this could have easily come from you, you know, if you're an engineer, the head of engineering, I know this is important and, as individual, I never wanted to do it.

Be on that list. I never wanted the person responsible for my team to identify me as someone holding back the rest of the group. It's pretty powerful and simple, so here's a timeline. We are arriving at the beginning of November. A couple of thoughts on things that are really critical, I think in the next month I think there are two things that are most important for a company that is thinking about adopting okrs for a three month trial first: you need to think about what your goals will be for the first quarter. You probably also want to think about the annual okrs, but thinking tactically about what are the immediate things that you know we're going to have, we'll worry about if I'm the CEO. what are the three or four most important things for 2013.

The other thing that's really critical to work on now is deciding how to capture what okrs are and how they can be shared within the company. This process will fail unless everyone in the company has a way to see what everyone else is working on, so I think it's critical that over the next four or five weeks you figure out if you're going to use a Wiki, if you're going to use Google sites, if some engineer is going to build a really simple Rails app that's just going to have a bunch of text fields and you know, log in to your username and whatever, keep it simple, don't over-design it at the beginning, It's always easy. to migrate to something more powerful in the future.

I think the key here is just knowing what it's going to be, so that if there's a problem to solve, you solve it well. It worked well before the end of the year and now in December. I think company-wide goals should be communicated for the first quarter and for the year, that way everyone else in the company, by the way, should also communicate what the heck the okrs are so that there is a general understanding within the company and by the The way this video is recorded is available for anyone in your company to view, so if anyone has questions about what it is, you can easily share this video with them, communicate the annual and quarterly goals of the first quarter for the company.

Now it is up to the people in the company is late December or early January to write their personal okrs and that should be at least a draft with a meeting with the manager and at least one iteration to reach a consensus. This is the negotiation that John Doerer was talking about in a company-wide meeting, probably around the second week of January at the latest, where company-wide goals are presented and, if applicable, depending on the size of your team, team goals within that company are now presented for the remainder of the first important quarter. That there are one-on-one meetings between individual contributors and their manager at least once every quarter.

It is not necessary to exaggerate, it is not necessary that they be weekly checks, but at least once or twice so that there is a general point in the process. where everyone in the company has gone through that exercise of saying, you know, this is what I said at the beginning of the quarter. It would be done by February 15. Am I in the ballpark if I'm fantastic? I keep doing what I'm doing yeah Don't I need to recalibrate now? Do I need to stop doing certain things in order to meet the goals I set? Where am I going to fail?

What can I stop focusing on completely so I can at least accomplisha part? few that are really important and finally you repeat this whole process so that towards the end of March it is now up to each individual to start thinking about rating their team owners okrs, start generating the team scores, the ratings and then the CEO The leadership team will score the companies' okrs and in early April there will be a company-wide meeting where they will score the first quarter's okrs and explain why they missed or hit and what they have learned about what will happen in the second quarter and this.

Just this process happens on an ongoing quarterly basis, okay, believe it or not, we're at the end of my formal presentation here, so I'm going to pull out the questions that have come in from all of you online. uh and in the room, so thanks for that, let's see what kind of tools you use. Luke asked to manage and communicate well, we talked a little about that. I've seen it go through at least three revolutions on Google. today we have a very simple tool that is really nothing more than logging in with my username in our cloud and clicking on the quarter and I add, I write a goal and then under each goal I have a button to add a key result. and add each individual key result.

I would show it to you, but again it's internal and proprietary, so it's not the kind of thing we would share in a workshop here, but you get the idea of ​​what it is. very very simple at the end of the quarter I go in and click on the score button and that's when now I'm just assigning a number in the text box next to each key result and as I mentioned above, then I have the ability to average the math really sophisticated things that are going on there, um, I think in most cases, for the people who are here, for the people who look online, I think something as simple as a Wiki, if you use a Wiki internally, I think Google sites would probably be enough, you know, I think this is the kind of thing you could design too quickly.

In my opinion, the keys are not, they are not the tool, it is the commitment to be measurable, the transparency that everything is available internally to everyone in the company and the qualification process itself I think beyond that, I think That the tool matters less, what is the planning cadence for the planning portion of an OKR period? For example, is it all one week or is it spread over several? How many meetings do you have with whom do you attend? The baseline at Puppet Labs has an exact retrospective meeting, then an executive planning meeting, then team meetings, and an executive review meeting, all in one week.

I think every company will probably be a little bit different in my own assessment if it were if I were implementing this in my own company tomorrow um I want this to be a relatively light process um I don't think it necessarily has to be in a meeting I think this can be the kind of collaborative writing that can happen where everyone can take a look at a document make some suggested changes and then have a meeting to fight or negotiate key points so that there can be some consensus. I think if you run the risk of meeting the process, which I think could be problematic. because then that's reinforcing the wrong thing, the point here is to do the work, not get together to do the work, so what would you say?

I think in terms of qualifications, I think it's important for people to have that average level. quarterly check-in so they have an idea of ​​where they are at, I think. There will probably be some diligence, maybe an hour towards the end of the quarter and then a final meeting at the beginning of the next quarter to come up with a grade in terms of planning for the next quarter . I think it often comes naturally. byproduct of the grading process in which I think about what I did last quarter. Are there things that are happening from the last quarter to the next?

In which case, you know, it's kind of a copy and paste operation. Are there things you wanted to be? working on that which I deliberately chose not to do in previous quarters for prioritization purposes and adding them. I think cramming it all into one week is probably too tight. I would say I would at least limit it to a week before the end of the quarter to a week after the end of the quarter, maybe a little buffer on both sides. I think ultimately that will give you enough room to give people the feeling that you are helping in the planning process, not taking time out of their schedule, so Jeff asked, how do you make sure that the okrs are accumulate or decompose the structure of the organization? we do this um, I talked in the deck about the kind of virtuous circle and I really do, I not only believe it, I've seen it work, the way we make sure is you know it used to be Eric when I was when.

I joined Google for the first time. Now Larry is the CEO. There's a quarterly meeting and Larry teaches it and Larry says these are the most important things the company is working on, these are the grades on what we did last quarter and now I'm on. I'm going to have Vic and Oars and Susan and Jeff and each individual group leader stand up and explain how last quarter went and what they're working on next quarter now, that makes a really important clarifying point. I think most people, especially in startups, want to be working on the most important things was always very helpful for me while I was going through the process of writing my okrs.

I would write them at least in draft form and then attend the company okrs meeting and do a reality check with the company okrs and I would like to know if I can see in my okrs the company okrs that you know , it's what I'm focused on, reflected in what the company is most focused on, think about you, you know, the punter, the quarterback. the running back is what they focus on, which reflects on the general manager or team owner. I can tell you that you know, at one end there was a quarter, when I ran the YouTube homepage, where I owned two of the companies, five and five. or six okrs that was scary because I not only my myocarria were not only reflected in the company I don't care they were the company okrs um that's exciting but it's also for a company that at that time had about 30,000 employees a little Da scary when you are leading the team that is responsible for a significant proportion of what the company considers most important at that moment.

The reason it was a useful process was if you could. If I don't see the company okrs in my individual okrs, then I would know that's when you take the red pen to your personal okrs and review them. Jeff asked Cadence to check the progress of the process with the okrs. Different teams will do it differently. It would be more natural to do it every few weeks, especially on larger teams. I think it's probably realistic that each individual has to own the discipline of managing themselves and maybe only have one or two check-ins in a quarter with their manager. um specifically against okrs um so I was asked if I was using okr as a blogger before I joined Google to clarify um I joined Google in 07 Google acquired blogger in O3 um and obviously I wasn't part of the Blogger team until late 08 um At that moment, the blogger had been under Google ownership for five six years and had actually accepted okrs, so I'm sorry if I didn't make that clear, last question, unless there are others here in the room.

I see several heads nodding. You know, they've been scoring them on the dock, so that's great, at what point in the evolution of a company should the Institute management team accept? It's five people, ten people, 15 more, an easy answer as soon as possible. The sooner this is part of the DNA of the company, the sooner it is accepted, the sooner it is part of the normal rhythm that the company enters into, the better off you will be, there is always a point at which you can do this. Obviously, I'm hosting this workshop because I think each of you should have some version of this approach in your companies, so even if you've been around for a year or two, it's not too late, but the longer you wait, the longer Habits will form within the company about how things are done and the more inertia you will have to overcome, so although it may seem a little artificial, if you are a company of five, the discipline that comes with it is immeasurable.

I committed to doing okrs even though I'm in the startup lab right now. I am a one-person team. Technically I'm part of Google Ventures. We are a large and growing team and I have a manager who I communicate with regularly about progress. but for me the discipline of looking ahead and saying these are the things I'm going to focus on this quarter these are the things I'm not doing well and then evaluating how I'm doing it's hard for me to imagine doing my job. this discipline at the end of the quarter, it's incredibly easy to summarize what we've done and whether we did it well or not, and I mean, I mean, for the startup lab, so how many workshops do we schedule?

This quarter, one of my goals was to release the video files that we released, you know, in the second week. It had been a goal in the third quarter. I had done most of the heavy lifting in the third quarter. I couldn't pull the trigger to get the files increased, so it moved to the fourth quarter. My grade for the third quarter was not so good. My grading for Q4 was less about the launch because I knew that would happen in a week or two after the start of the quarter. It was more important to focus on it.

Well, now we're going to measure how much traffic the files get, how many views there are, how many people have requested access to the files, those are the kinds of things that start to give you an idea of ​​what that mentality can really be like. I accepted and pushed for action and an achievement, so with that, I'm going to end, let me again, thank you for taking the time. I made fun of you for not telling you how long it was going to last. I budgeted 90 minutes. I think I'm about 88, so you know, hey, I'll write it down in one of the okrs.

Please fill out the feedback form for those of you who are here in person or watching online, data is critical as you know not only from this presentation but others, we are very data driven, as a group , not only the quantitative score that gives us an idea of ​​how we are doing on that scale of one to seven, but also the qualitative one. What conclusions did you draw from this event? Knowing what resonates helps me when thinking about planning future topics. Really knowing what to focus on and how to direct presenters. Those are really critical issues again, as I said in the at. at the beginning of this workshop if you have a topic you would like us to cover I want to know what it is I want to hear from you my job is to find the knowledge and experience on Google and make it Available to you, it's a great job.

I love doing it, but I prefer not to be prescriptive and tell you what I think you should know. I'd rather know what you think you need to know so I can make it available. Stay tuned. On the announcement list, we have a lot of things coming up for the rest of Q4 and we're starting to do a lot of planning for Q1 and we're really excited about some changes in focus and format so we look forward to seeing you again online in person until then thanks again see you soon bye