Sears Holding Company Study Shows OKRs Impact Bottom Line

More and more people are embracing Objectives and Key Results (OKRs) even though there's been no research that using OKRs impacts financial results. Now there is.

For the group who used OKRs we saw an increase in their average sales per hour from $14.44 per hour to $15.67, or an average increase of 8.5%. This increase is not only statistically significant, but practically significant. - Chris Mason

Chris Mason, a Ph.D. in Industrial-Organizational Psychology from DePaul University, is the Sr. Director, Strategic Talent Solutions at Sears Holding Company (SHC). In the fall of 2013, Chris and his team led one of the largest deployments of OKRs to date, rolling them out to their entire salaried population of roughly 20,000 associates. Research based on the use of OKRs over the past 18 months at SHC concludes OKRs can impact the bottom line.

Joe Kutter was the lead research scientist who analyzed the data to conclude that even a minimal use of OKRs led to higher levels of performance, including an 8.5% lift in hourly sales at a call center. Consistent use of OKRs resulted in an 11.5% increase in the chances of moving to a higher performance bracket across their corporate associate population. These may be the first experiments concluding that using OKRs makes an impact both on personal performance and on the organization’s bottom line. I recently spoke with Chris and his team. He shared with me how they stumbled into the first evidence that OKRs do in fact impact the bottom line:

Lamorte: I’ve not seen a single formal study addressing how and if OKRs improve an organization’s bottom line. Have you seen any such research?

Mason: To my knowledge, there is no research that demonstrates if, and to what extent, the use of OKRs actually improves an organization’s bottom line. Certainly, decades of research in behavioral sciences such as the work of Locke and Latham who developed Goal Setting Theory, suggests that setting shorter-term goals, giving people more autonomy so they commit to the goals, and including a stretch component, drives higher achievement. While we could assume OKRs are effective based on this general research, I’ve not seen any research supporting that OKRs, as a specific methodology, is effective.

Lamorte: Given you knew of no formal analysis documenting the effectiveness of OKRs, why did you deploy OKRs and develop an in-house software tool to track and monitor OKRs?

Mason: We actually were inspired to deploy OKRs based on the Google Ventures video in 2013, which our Chairman and CEO, Eddie Lampert, shared with senior leadership at SHC. That video and the continued leadership of our Chairman and CEO have been the driving force behind OKRs at SHC. More recently, members of our HR leadership have had the opportunity to speak with Google about their approach, and while I don’t have all the details, my understanding is that they haven’t conducted any formal research into the effectiveness of OKRs. In the book How Google Works, the argument for OKRs is simply that they make sense and support their culture.

Lamorte: Describe your process for researching the effectiveness of OKRs.

Mason: After using OKRs for over a year, we are beginning to analyze the data. Preliminary results indicate that using OKRs does indeed create a positive impact. We have two studies so far. The first one happened by accident, which actually makes it a great experimental design complete with pre and post-measures and a control group. Our outbound call centers are divided into teams of outbound call agents at various locations. They focus on add-on sales, measured by metrics like the number of calls per hour and total sales per hour. OKRs were formally launched to salaried associates at the centers, but not to these outbound sales agents (though everyone was given access to the system that houses OKRs). For reasons that are not entirely clear, a portion of the call agents opted in. It’s particularly interesting since agents opting in were spread across multiple locations and they started with OKRs in various months, which helps control for any effects of location or seasonality in our data. We started in 2013 and over the year and a half, agents opted in organically.

Lamorte: How did you compare the performance of each group of sales agents over time?

Mason: We went back to moment when they first started using OKRs. We then compared their sales performance from the month prior to using OKRs with the month when they first entered OKRs and their first month after entering OKRs.

Lamorte: Can you summarize the results?

Mason: Yes. For the group who used OKRs we saw an increase in their average sales per hour from $14.44 per hour to $15.67, or an average increase of 8.5%. This increase is not only statistically significant, but practically significant. We also saw a similar increase in the number of calls they made per hour, which helps explain the increase in revenue. Most importantly, the control group who did not use OKRs averaged the same sales each month and didn’t increase.

Lamorte: Does this 8.5% result control for the possibility that agents who decided to use OKRs might happen to be the high performers?

Mason: Yes, in fact, we controlled for this. It’s not possible that agents opting in simply improve because they are high performers. The agents improved relative to their own performance. It doesn’t matter who you were or when you started, any agent who used OKRs averaged an 8.5% lift in their own sales performance compared to their rate before starting to use OKRs.

Lamorte: This may be the first study that measures the impact of OKRs on bottom line. If so, we’re talking about an 8.5% lift in revenue. This might have major implications for getting more sales out of a call center. However, this is just one of the many groups of employees using OKRs. Do you have any data that indicates OKRs can help workers be more effective who are not in sales roles?

Mason: Yes, we next looked at the broader corporate population. Now the sample size here is roughly 12,000 workers across dozens of functional areas. This study is particularly interesting because we were also able to look across a longer time period and see how often people used OKRs between the late summer of 2013 and the summer of 2014.

Lamorte: Given you’re looking beyond a sales team, you can’t simply look at revenue per sales agent as an outcome variable. What outcome variable did you use to measure the general impact of OKRs on effectiveness?

Mason: We used the outcome of general performance level. We use a typical 9-box tool to classify this population on the dimensions of performance and potential on an annual basis. For this study, we looked only at performance, where we had classified everyone into either high performance (roughly top 20%), middle performance (middle 70-75%), or low performance (bottom 5-10%) in the summer of 2013 and again one year later in the summer of 2014. It is hard to move from medium performance to high performance, but if you can, good things such as rewards and promotions are more likely to happen. We then categorized each worker into three groups based on their use of OKRs: 1) Consistent: entered at least one OKR consistently all four quarters of that year, 2) Inconsistent: used OKRs at least one time in the year, but failed to enter them every quarter, and 3) No use of OKR at all during the year.

Lamorte: So, did the use of OKRs impact a given worker’s chances of getting classified at a higher performance level?

Mason: Yes, and the results are very interesting:

Group 3 - No use of OKRs showed no change in performance level.

Group 2 - Inconsistent use of OKRs showed some improvement. Even just using OKRs once made it 3% more likely to move up a whole performance level. In this group, we did not even consider the quality of OKRs and we still saw a statistically significant, but small impact.

Group 1 - Consistent use of OKRs were 11.5% more likely to move into a higher performance bracket.

Lamorte: OK, so what’s the bottom line conclusion here? Can we now say research supports that OKRs are effective?

Mason: We’re not saying we have definitive proof that OKRs works for everyone. However this second study may have some big implications. This result translates roughly to: “If you knew there was a tool out there that takes just a few hours each year to use which could increase your chances of high performance by 11.5%, wouldn’t you at least try it?”

Conclusion: Sears Holding Company has conducted research on the use of OKRs with 20,000 associates for the past 18 months and found evidence that even minimal use of OKRs led to higher levels of performance, including an 8.5% increase in hourly sales at a call center and an 11.5% increase in the chance of high performance across their corporate associate population. These may be the first experiments concluding that OKRs makes an impact both on personal performance and on the organization’s bottom line.

If you know of any research on the effectiveness of OKRs or any other specific approach to Enterprise Goal Management, please share it here!