The Need for Business Experimentation in a rapidly changing Retail world

JULY 27, 2018
Neeraj Hirani

The retail industry has seen a tremendous transformation in the last few years. Digitization and globalization have fundamentally changed the way businesses are run. With data becoming the new decision-driver, traditional brick & mortar retailers are investing in technologies to keep track of customer behavior, purchases, and preferences. However, one also needs to analyze the data to identify actionable insights that drive profitability.

For example, consider a retail major looking to improve its sales numbers through investments using a combination of a new store layout, a different product mix, and hiring additional manpower. Before committing to huge investments, the retailer would want to know which changes increase value and by how much, and whether the program will have an impact on some stores or all of them? Historical data, surveys, and focus groups are not always reliable. And sophisticated mathematical models can always prove that a program will succeed or fail just by making tiny adjustments to analytical assumptions. Interacting, complex, multivariable, nonlinear systems such as these do not lend themselves well to prediction.

Consider promotional campaigns, discounts, and other product incentives. These are critical tools that retailers use to move product off the shelf and increase revenue. Yet, the effectiveness of these programs is often not known until after implementation. After executing a program, debates about the true efficacy of the program linger on in boardrooms. The key question then becomes how can retailers cut through this complexity?

The answer lies in controlled experimentation. Rather than depending on gut feel or guesswork, running controlled experiments provides retailers a reliable foundation for decision-making with confidence.

“It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with the experiment, it is wrong” – Dr. Richard Feynman

Experimentation is the act of trying out new ideas, processes or activities in a scientific manner to test potential improvements on a test group in comparison to a similar control group. We expose one group to a change (also called an intervention or treatment) during a given period and a similar control group is subject to no changes. Then, their performances (conversions, sales) are measured using statistical methods and analyzed to determine if the change(s) are worth implementing.

The goal of experimentation is not to be right, but to test the entire data and learn from the interventions. Exposing customers to an intervention for a small duration quickly reveals patterns that spur learning. For example, a grocery retailer testing differential regional pricing might observe that for some geographies, the pricing works well, but for some, it is questionable, and for some, it is clearly unacceptable. The retailer can then focus on the locations where the results are working and improve there, thereby gaining maximum value. At the same time, the retailer can reduce the focus on the locations where this is not working or change the intervention itself.

Experimentation, by definition, will lead to multiple failures every day. Most experiments will fail. A mindset that de-personalizes failures delves into understanding them and then discusses corrective steps to succeed is critical. What makes this tenet especially challenging is that such failures are often public and recorded as part of the evolution of thinking. So, one needs to genuinely embrace failures as a part of the learning process.

Amazon, the global e-Commerce major, is known to have a culture of agile experimentation deeply embedded in the way it approaches data-driven decision-making. Behind Amazon’s huge successes is also a huge graveyard of failed experiments. AWS, which started off as an experiment, has contributed to Amazon being the fastest growing B2B company in history.

 “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day..” – Jeff Bezos, Founder & CEO of

At any given time, Amazon runs thousands of experiments on their website to test calls to action, landing page text copy (fonts, colors, short versus long formats, bullets versus paragraphs), the sequence of steps in their checkout process, etc., to learn how they can convert customer visits into more sales. Brick & mortar retailers too can use a test & learn approach to give themselves the ability to maximize revenue opportunities via interventions (new layouts, product mix, promotions, a new marketing channel, etc.).

Capital One Financial Corporation is credited with making experimentation in the offline world famous by quick modification and termination of credit card products leading to rapid innovation. Large retail banks like Wells Fargo and PNC too followed suit, testing ideas empirically at a small scale before large rollouts. Large-scale retailers like HomeDepot, Target, Walmart, and Starbucks have already institutionalized experiments org-wide. But the bulk of the Fortune 500 retailers have barely scratched the surface. Retailers often have several potential initiatives that can easily consume 200%-300% of the organization’s resources available for investment. By adopting an experimentative mindset, executives can more effectively allocate precious resources to areas with the greatest impact and scale up “small bets” over time versus betting it all up front on a gamble.

Experimenters can start by capturing for every experiment, the 2 to 3 dimensions that are key to the experiment and the 2 to 3 outputs that matter. All experiments should be recorded centrally in a repository for the benefit of stakeholders across the organization. This allows each stakeholder to accurately assess which variables or techniques are driving more change than others. Such transparency opens up opportunities to learn from other’s successes and mistakes. Good experimenters also have the “nose” to know when to stop and change course. The idea is to fail cheap and fail fast. This makes experimentation a mixture of science and art. Capturing learnings at each step, about the why behind each decision, allows organizations to learn quickly.

In conclusion, while it is critical to understand the patterns in data, to gather insights and to build analytical approaches based on that insight, retailers have to demonstrate a knack for doing this exploration swiftly and exhaustively via experimentation. The increasing rate of change in business is questioning older paradigms and is necessitating the need to run experiments to learn from them. A high tolerance for failure coupled with a tight feedback loop will allow retail organizations to learn from their mistakes and improve. It is those organizations that will consistently learn by taking small bets and innovate incrementally who will stay ahead of the competition.

 Going forward, we will be writing a series of posts on some of the nuances of experimentation, as it pertains to the offline world. Watch this space for more!