Experiment
Often business leaders make decisions based on a mix of data, intuition, and experience.
Despite all of that, the best, most carefully crafted plans may fail miserably.
There must be a better way. Harvard Business School professor Stefan Thomke outlines the overlooked alternative: experimentation. His new book, Experimentation Works: The Surprising Power of Business Experiments, takes you through his research and best practices to show how to experiment and test before acting.
If you want to learn how to develop an experimentation organization, read on. I enjoyed my discussion about all things experimentation.
What was most surprising to you in studying the research into experimentation?
When I published my first book Experimentation Matters: Unlocking the Potential of New Technologies of Innovation in 2003, I made a prediction: digital experimentation tools not only had the potential to revolutionize a company’s R&D, but they could also transform entire industries by shifting experimentation—and thus innovation—to users and customers. Five years later, Apple opened its App Store, which empowered anyone, anywhere to design and distribute novel applications. By early 2017, about 2.2 million apps had become available to iOS users. And, as anyone who closely follows simulation and prototyping tools knows, their use has become pervasive in manufacturing businesses, even though companies still grapple with the integration and management issues I wrote about in 2003. As I happily watched these predictions come true, I thought that it was time to move on and study another topic.
And I was wrong because something surprising happened: In 2003, Google had just completed year five, Amazon was nine years old, and Booking.com was still an independent startup in Amsterdam. Even though I had studied the statistical and management principles that are core to experimentation, I had not closely examined their role in customer experience and business model design. I had no idea how their use would fuel the rise of today’s online businesses. When it finally came to my attention, I realized right away that large-scale, controlled experimentation would revolutionize the way all companies operate their businesses and how managers make decisions. What I saw was the scientific method fully deployed in organizations—and turbocharged! The similarities to what had happened in R&D were striking. Both revolutions were about the potential of new experimentation tools, processes, and cultures and what companies should do to unlock their power. By 2003, not all industrial companies were fully committed to accommodating their organizations to new tools and following the principles described in Experimentation Matters. A few years later, these companies had learned that they had no choice but to go full throttle if they wanted to remain competitive.
That’s when I decided that it’s time to write a new book. A few years later, in 2020, Experimentation Works: The Surprising Power of Business Experiments was published! The timing of Experimentation Works couldn’t be more fortuitous: four hundred years ago, in 1620, Francis Bacon published Novum Organum, the classical formulation of a new instrument for building and organizing knowledge: the new scientific method. Thinking and acting scientifically has had an enormous impact on the world. For centuries, we’ve built and organized scientific and technological knowledge through testable explanations and predictions. These, in turn, have given us modern medicine, food, energy, transportation, communication, and so much more.
Why do many business leaders resist it?
For an organization to fully absorb experiments, unbridled curiosity needs to drive out strong opinions and biases. That’s especially true for higher status individuals—the bosses—whose promotions most certainly involved some lucky business decisions. When it comes to novelty, even the bosses can be wrong. And that can become a big problem as illustrated by Jim Barksdale, the former CEO of Netscape, who once reportedly said: “If we have data, let’s look at the data. If all we have are opinions, let’s go with mine.” This is what happened at Amazon when an employee created a software prototype that would make personalized recommendations to customers based on items in their online shopping carts. A senior vice president was dead set against the feature because he thought that it would distract customers during checkout. The employee was forbidden to work on the project. Fortunately, he ignored the boss’s instruction and ran a controlled experiment, which showed that the feature won by a wide margin (measured by shopping revenue). It was immediately launched.
To denote a strong manager who favors a top-down approach to decision making, the term HiPPO (highest-paid person’s opinion) is frequently used tongue-in-cheek. The risk, of course, is that HiPPOs push bad ideas, either through status or persuasion, and resist experiments that prove them wrong. Handing out plastic versions of a hippopotamus, which is among the most dangerous animals in the world, to an organization can serve as a symbolic reminder of the cultural challenges it faces.
Being intellectually humble and saying the words “I don’t know” or “My idea probably has no impact” isn’t easy, since it goes against the way humans think and behave. The behavioral economist Daniel Kahneman once noted that “if you follow your intuition, you will more often than not err by misclassifying a random event as systematic. We are too willing to reject the belief that much of what we see in life is random.” In other words, humans have a tendency to see connections and meaning between unrelated things. Different theories of why this occurs include cognitive errors in pattern recognition and human evolution favoring brains that see causal relationship when there are none. A result is the human tendency to commit such errors when we observe things or hear anecdotes. Managers are no exception, especially when incentives favor finding causal relationships between variables that are difficult to measure, such as changes in leadership style and team performance. As the American author Upton Sinclair once quipped: “It is difficult to get a man to understand something, when his salary depends upon his not understanding it.”
Are there parts or elements of a good business experiment that more often go wrong or are overlooked in most organizations?
In an ideal experiment, testers separate an independent variable (the presumed cause) from a dependent variable (the observed effect) while holding all other potential causes constant. They then manipulate the former to study changes in the latter. The manipulation, followed by careful observation and analysis, yields insight into the relationships between cause and effect, which ideally can be applied and tested in other settings. To obtain that kind of learning—and ensure that each experiment yields better decisions—companies should ask themselves seven important questions: (1) Does the experiment have a testable hypothesis? (2) Have stakeholders made a commitment to abide by the results? (3) Is the experiment doable? (4) How can we ensure reliable results? (5) Do we understand cause and effect? (6) Have we gotten the most value out of the experiment? And finally, (7) Are experiments really driving our decisions?
Any of these questions can go wrong. As a start, companies may not have hypotheses that are ready to be tested. Consider Kohl’s, the large retailer, which in 2013 was looking for ways to decrease its operating costs. One suggestion was to open stores an hour later on Monday through Saturday. Company executives were split on the matter. Some argued that reducing the stores’ hours would result in a significant drop in sales; others claimed that the impact on sales would be minimal. The only way to settle the debate with any certainty was to conduct a controlled experiment. A test involving one hundred of the company’s stores showed that the delayed opening would not result in any meaningful sales decline.
In determining whether an experiment is needed, managers must first figure out exactly what they want to learn and measure. Only then can they decide if testing is the best approach to achieve their answer and, if so, what the scope of the experiment should be. In the case of Kohl’s, the hypothesis to be tested was straightforward: Opening stores an hour later to reduce operating costs will not lead to a significant drop in sales. All too often, though, companies lack the discipline to hone their hypotheses, leading to tests that are inefficient, unnecessarily costly, or, worse, ineffective in answering the questions at hand. A weak hypothesis—for example, “We can extend our brand upmarket”—doesn’t present a specific independent variable to test on a specific dependent variable and cannot yield measurable outcomes. Thus, it is difficult either to support or to reject it. A good hypothesis helps delineate those variables and suggests metrics.
How do leaders encourage a culture of experimentation?
They must embrace a new leadership model. If most decisions are made through experiments, what’s left for leaders to do, beyond developing the company’s strategic direction and tackling big decisions such as which acquisitions to make? There are at least three things:
Set a grand challenge that can be broken into testable hypotheses and key performance metrics. Employees need to see how their experiments support an overall strategic goal.
Put in place systems, resources, and organizational designs that allow for large-scale experimentation. Scientifically testing nearly every idea requires infrastructure: instrumentation, data pipelines, and data scientists. Several third-party tools and services make it easy to try experiments, but to scale things up, senior leaders must tightly integrate the testing capability into company processes.
Be a role model. Leaders have to live by the same rules as everyone else and subject their own ideas to tests. Bosses ought to display intellectual humility and be unafraid to admit, “I don’t know…” They should heed the advice of Francis Bacon, the forefather of the scientific method: “If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties.”
What are some of the myths that leaders should watch out for when embracing the experimentation philosophy?
Becoming an experimentation organization will undoubtedly cause frictions, as for every action there will be an opposing reaction. The causes that I’ve come across cover a broad spectrum: inertia, anxiety, incentives, hubris, perceived costs and risks, and so on. But I have also found that managers aren’t always aware of the power of business experiments described in my book. This failure to understand and appreciate their true benefits has given rise to seven myths that undermine innovation and are described in Experimentation Works.
Here is one example: A few years ago, I gave a presentation on business experimentation to a large audience of executives and entrepreneurs. The audience was intrigued until one participant, the founder and CEO of a national restaurant chain, energetically voiced his opposition to subjecting his employees’ ideas to rigorous tests. He strongly believed that innovation is about creativity, confidence, and vision and, in a loud voice, proclaimed: “Steve Jobs didn’t test any of his ideas.” His message was unambiguous: a greater focus on experiments will backfire, put great ideas at risk of being prematurely dismissed, and will ultimately kill intuition and judgment.
But, I countered, it’s not about intuition versus experiments; in fact, they need each other. Intuition, customer insights, and qualitative research are valuable sources for new hypotheses, which may or may not be refuted—but hypotheses can often be improved through rigorous testing. The empirical evidence shows that even experts are poor at predicting customer behavior; in fact, they get it wrong most of the time. Wouldn’t it be preferable to know what does and does not work early and focus resources on the most promising ideas? After some participants sided with this reasoning, he gradually relented. (Curiously, I later found out that his company had been a user of a popular tool for running rigorous in-restaurant experiments, yet he was unaware of it.) With respect to his comment about Steve Jobs, it’s remarkable how many people believe that their intuition and creativity can match Jobs’ track record—until they don’t. Incidentally, let me dispel another myth: Apple does run experiments.
For more information, see Experimentation Works: The Surprising Power of Business Experiments.
Boost Your Positive Intake
Photo Credit: Louis Reed.
