Chapter 4: Experimenting the Efforts of a Startup

The Lean Startup method reviews the effort of a startup as experiments that test its strategy to see which parts are good and which are bad. A real experiment complies with the scientific method by beginning with a clear hypothesis that predicts what is supposed to happen. It then tests those predictions practically. Like scientific experiments are based on theory, startup experiments are based on vision. Every startup experiment aims to find out how to create a sustainable business around that vision.

We will explore real-life examples of companies to understand the startup experiments in more detail.

4.0 Zappos – Think big, start small

Being the world’s largest online shoe store, Zappos has an annual gross sale of 1 billion USD. When Nick Swinmurn founded Zappos, there was no central online site with a great choice of shoes to select from. He envisioned a new and powerful retail experience like no other. Like many e-commerce pioneers, Swinmurn could have waited long by testing his vision with warehouses, distribution partners and sales. However, he started by experimenting. He assumed that customers were willing to buy shoes online. To test this assumption, he started asking local shoe shops if he could take pictures of their shoe stocks. In exchange for permission to take pictures, he would post the pictures online and buy the shoes from the shop at full price if a customer wanted to buy them online.

While testing this assumption, other assumptions were tested too. For example, Zappos had to communicate with customers by handling payment, returns and customer support to sell the shoes. This is extremely different from market research. If Zappos relied on market research or surveys, he would have considered what customers thought they wanted. By testing the assumptions, Zappos learned that:

• There was more accurate customer demand data as Zappos observed actual customer behaviour and did not ask theoretical questions.

• They were communicating with actual customers and learning about their requirements.

• They were shocked when customers behaved unexpectedly by revealing information that Zappos might not have asked about previously. For example, what if the customers returned the shoes to them?

Zappos first experiment issued a clear and quantifiable result that either a good amount of customers would purchase the shoes or they wouldn’t. Moreover, it placed the company in a position to observe, communicate with and learn from actual customers and partners. This kind of learning is essential to quantitative testing. In 2009, Zappos was bought by Amazon for 1.2 billion USD.

4.1 Hewlett-Packard (HP) – experiment immediately for long-term changes

Caroline Barlerin, Director of the global social innovation team at HP, is a social entrepreneur trying to get more HP employees to use the company’s volunteering policy. The corporate guidelines at HP encourage all employees to spend four hours a month of their company time volunteering in their society. Caroline’s main priority was encouraging employees to use their work-based skills outside their company. However, most employees didn’t know that such a volunteering policy existed, and only a few employees made use of this. In addition, most volunteering involved manual labour, although the employees were highly trained professionals. Thus, Barlerin’s vision was to adopt thousands of employees in the company and convert them into a team for a social cause.

To relate this social project using the Lean Startup model, we see that Caroline’s project is facing extreme uncertainty as there had never been a volunteer programme at HP. Her goal was to motivate her employees to make the world a better place, but her plan had a lot of untested assumptions and an expansive vision.

By complying with traditional management practices, she spent her time planning, getting buy-in from other departments and managers and preparing a road map of initiatives for the first 18 months of her social project. Moreover, she had a strong accountability framework with metrics for her project’s impact on the company for the next four years. Yet, despite all her effort, she was far from creating one-off wins and barely knew if her vision would scale.

One assumption could be that the company’s permanent values included a commitment to improving society but that recent economic trouble had led to an increased companywide strategic focus on temporary profitability. A second assumption can be that they would find it better and more sustainable to use their actual work skills in a volunteering context which would significantly impact on behalf of the organizations to which they gave their time. Finally, there were other practical assumptions on the employees’ willingness to commit their time to volunteer, level of commitment and desire and the correct way to reach them with her message.

The Lean Startup model provides a way to test these hypotheses immediately and adequately. Strategic planning takes months to complete, and these experiments could immediately start. Let’s see how Caroline would have treated her project as an experiment:

Break down

The first step is to break down the vision into parts. According to Eric Ries, the two essential assumptions entrepreneurs make are the value and growth hypotheses.

The value hypothesis tests if a product or service provides value to the customers once they are using it. Experiments offer more accurate results to indicate that employees find donating their time to community service valuable. Opportunities can be found for a small number of employees to volunteer, and then the retention rate of those employees should be observed. How many of these employees will sign up to volunteer again? If employees willingly invest their time and effort in a volunteer program, it strongly indicates that they find it valuable.

The growth hypothesis tests how new customers discover a product or service. Similarly, this can be adopted for the volunteer program. Once the program is active, how will it spread between employees, from early adopters to mass adoption throughout the company? This program will likely expand through viral growth. If this is the case, the most important thing to measure is behaviour, that is, will the early participants spread the word to other employees.

In this scenario, an experiment would have a dozen long-term employees and provide an excellent volunteer opportunity for them. As Caroline’s hypothesis was that employees would be committed by their desire to achieve HP’s commitment to community service, the experiment would focus on employees who felt the greatest disconnection between their daily life and the company’s expressed values. The point is not to find a customer but an early adopter, as early adopters are more forgiving of mistakes and eager to provide feedback.

By using the technique that Eric Ries calls the ‘concierge minimum viable product’, Caroline could ensure that the first few volunteers had a good experience that was entirely consistent with her vision. Her goal would be to measure what the volunteers did. For example, how many of the first few volunteers complete their tasks? How many of them volunteered for the second time? How many are ready to get another colleague on board to participate in the following volunteer activity?

There are additional experiments that can expand on early feedback and learning. That is, if the growth model needs a certain percentage of volunteers to share their experiences with other employees and encourage their participation, the degree to which that happens can be tested with a tiny number of people. For example, if ten employees complete the first experiment, how many should you expect to volunteer again? If they are asked to get a colleague on board to join the next volunteer program, how many do you expect will do so? You should note that these are supposed to be the type of early adopters with the most value to gain from the program.

This whole experiment can be conducted in weeks. It can happen in line with strategic planning while the plan is being formulated. Even if the experiment provides a negative result, those results prove instructive and can impact the strategy. For example, what if no volunteers are experiencing the conflict of values in the organization that was such an essential assumption in the business plan? If this is the case, it’s time to pivot.

4.2 Experimenting is a product

In the Lean Startup model, an experiment is a product. If an experiment is successful, it enables managers to start with their campaign by enlisting early adopters, adding employees to each more experiment or iteration and eventually building a product. When the product is ready to be widely distributed, it will already have established customers. It would have solved problems and provided detailed instructions for what needs to be built. Unlike a strategic planning or market research process, this instruction will be established in feedback on what’s working today instead of predicting what might work tomorrow.

To understand this better, let’s consider Kodak as an example. When Mark Cook was Kodak’s vice president of products, he made his team answer four questions:

1. Do customers realize that they have the problem you’re trying to solve?

2. If there was a solution, would they buy it?

3. Would they buy the solution from us?

4. Can we build a solution for the problem?

Usually, product development skips right to the fourth question and builds a solution before confirming that customers have a problem. For example, Kodak provided wedding cards with luxurious text and graphics on its website. Such designs were popular among customers who were going to get married, and the team redesigned the cards for other special occasions like holidays. The market research and design procedure showed that customers liked the new cards, which justified the effort to make them.

Before the launch, Kodak discovered that the cards were complicated to understand due to how they were portrayed on the website, as people couldn’t see how beautiful the cards were. Moreover, the cards were hard to produce. This made Cook realize that they had done their work incorrectly. He, therefore, stopped the engineering team from doing anything further until they could discover how to sell and make the product. Through this experience, he took a different approach by leading his team to develop a feature set for a product that makes it easier to share photos taken in a function. They felt that an online event album would allow people who attended a wedding, conference or gathering to share pictures with other attendees online with solid privacy.

Cook led the team to discover risks and assumptions before building anything and then test those assumptions through an experiment. There were two main hypotheses:

1. An assumption that customers wanted to create the album.

2. An assumption that attendees of the event would upload photos to the online event album created by friends and colleagues.

The team at first built a simple model of the event album. It lacked many features. However, allowing customers to use the initial model early helped them reject their hypothesis. They discovered that their early customers could not create an album and lacked many features. Although the product failed, Cook explained that it didn’t mean the project was a failure. Through their initial failure, they found out that customers wanted to create event albums and wanted essential features, which was a valuable insight.

Through a Beta Launch, the team consistently learned and iterated. For example, using the online surveying tool KISSinsights, the team discovered that many customers wanted the ability to arrange the picture order before they could invite others to upload the photos. Knowing they were not ready to launch the product, Cook controlled the division’s general manager by explaining how iterating and experimenting before the marketing campaign started would provide better results. This procedure provided a better change to Kodak as employees were comfortable being measured on their progress by completing tasks.

Conclusion

Based on these examples from Zappos, HP and Kodak, we can understand that most entrepreneurs and managers face the same challenges regardless of the industry they belong to in terms of consistently struggling to build and launch new products. They find it hard to overcome management thinking that places its faith in well-researched plans. Changing this type of mentality is hard but essential for a startup to be successful, which is what Eric Ries strived to achieve in his book.