Chapter 3: Validated Learning

‘Learning’ has always been an excuse for an entrepreneur’s failure of execution. However, learning is a consolation to employees following entrepreneurs into the unknown. It is a consolation for investors to allocate money, time and energy to entrepreneurial teams. It is a consolation for big and small organizations to depend on innovations to survive. Thus, if the primary goal of entrepreneurship is to engage in organization building under highly uncertain conditions, its most important function is learning. You should know which elements of your strategy are effective to understand your vision and which aren’t working. You should learn what customers need, not what they say they need or what you think they should need. You must discover if you’re on a path which will lead to a growing sustainable business.

In the Lean Startup model, learning is rehabilitated with a concept called ‘validated learning’, which is built to demonstrate progress when one is buried in extreme uncertainty in which startups grow. It’s also the process of proving that a team has discovered valuable truths about a startup’s current and future business prospects. It’s more correct and faster than market prediction or business planning.

3.0 Understand validated learning with IMVU

Let’s understand more about validated learning with Eric Ries’s example based on his career at IMVU, a social networking site he founded in 2004. Before he and his team built the app, their concerns were what should we build and for whom? What market could we enter and dominate? And how could we build durable value that won’t be subject to erosion by competition?

They then came up with an excellent strategy to enter the instant messaging market where there were millions of consumers globally. IM is an example of a market that includes strong network effects. Thus, the more people in the network, the network is more valuable. This makes sense as the value to each person in the network is based on how many other people they can communicate with. In 2004, the top 3 networks controlled more than 80% of the whole network usage and were in the process of merging their gains in market share by sacrificing many small players. Therefore, it was common sense that it was impossible to bring a new IM network to the market without spending a lot on marketing. This is because of the power of network effects that IM products have high switching costs. To switch from one network to another, customers need to convince their friends to switch with them. This extra burden for customers creates a barrier to entering the IM market.

At IMVU, they decided on a strategy of building a product that would merge the large mass appeal of IM with the high revenue per customer of 3D video games and virtual reality. Due to the impossibility of bringing a new IM network to the market, they decided to build an IM add-on product which would be consistent with the current networks. Thus, customers can adopt the IMVU virtual goods and avatar communication technology without switching IM providers, study a new interface and bring their friends with them. For the add-on product to be productive, customers would need to use it with their current friends. Each communication will come with an invite to join IMVU. To promote IMVU, it was important for their add-on product to support as many of the current IM networks as possible and to operate on all types of computers.

3.1 Six months to launch IMVU

The add-on product was large, complicated and had many moving parts that they had to take shortcuts to launch the product on time. Although Eric and his team were scared when the launch date came, they didn’t postpone it as other entrepreneurial teams did as delaying the launch prevents many startups from receiving the feedback they need. Their past failures made them more scared of the subsequent failure and what they were most afraid of was launching a lousy product that nobody wanted.

3.2 Launch

On the day of the launch, nobody tried their product. This was disappointing, considering the number of hours they spent discussing the features to include and which bugs to fix. Thus their value proposition was so remote that customers were hardly experiencing finding out how bad their design choices were, and customers didn’t download their product.

In the following months, they tried to make the product better. They brought a fine number of customers through their online registration and download process and received daily feedback from customers. They eventually learned how to change their product’s positioning to ensure that customers would at least download it. In addition, they continuously improved the underlying product, fixing bugs daily and making new changes. Despite these efforts, they could only convince a small number of customers to purchase the product.

However, they made a good decision to set precise revenue targets early. They planned to produce 300 USD in the first month, but they barely achieved this. Each month, their small revenue targets increased, and as their revenue targets increased, their struggles increased. This brought more customers into their office for physical interviews and usability tests. Unfortunately, their entire strategic market analysis was utterly wrong, and they discovered this by experimenting instead of focus groups or market research. Customers couldn’t tell them what they needed and had never heard of 3D avatars. Instead, they revealed what they needed by their action or inaction as Eric and his team struggled to improve the product.

3.3 Talking to customers

Since they had no choice, they decided to talk to prospects by bringing them into the office and making them try the product. If the customer were a teenager who is a heavy user of IM or a tech early adopter, they would coordinate. However, if it was a mainstream customer, they thought IMVU was a weird product.

Sometimes a 17-year-old didn’t understand what an instant messaging add-on meant, so it required a lot of explanation. After so much explanation and when they asked the teenager to invite one of their friends to chat, they were pretty doubtful about it, and this kept repeating with other customers. So finally, out of desperation, they introduced a feature called ChatNow, which allows the users to push a button and get randomly matched with someone from anywhere in the world. The only common thing is that both users pushed the button simultaneously. Suddenly, in the customer service tests, people found this feature fun. However, when they wanted to add a person to the buddy list, Eric would tell them to use their regular AOL buddy list. This was how they planned to control the consistency leading to network effects and crazy growth. However, customers were unhappy with having a stranger on their AOL buddy list, to which Eric had to respond by telling them to download a whole new IM client with a new buddy list, and customers were not happy about this, considering the many IM messaging clients they have.

Therefore, they had an incorrect assumption that learning new software is challenging and that moving friends to a new buddy list is tricky, as their customers revealed this was nonsense. They wanted to draw diagrams that showed why their strategy was excellent, but the customers didn’t understand theories such as network effects and switching costs. If they tried to explain why customers should behave the way they predicted, it confused the customers.

They had a mental picture of how people used outdated software, and eventually, after many customer meetings, Eric and his team realized that the IM add-on concept was wrong. Their customers didn’t want an IM add-on as they wanted an independent IM network. Early adopters used many types of IM programs at the same time. The customers were not scared of bringing their friends to a new IM network as they enjoyed that challenge. Even their assumption that customers use avatar-based IM mainly with their friends was also wrong. They wanted to make new friends.

Although these failures were a waste of valuable time and money in developing the product, one good thing about these failures was that Eric wouldn’t have gained valuable insights about the product, which he could improve on later.

3.4 Value-creating efforts vs Wasteful efforts

Identifying which are value-creating efforts and which are wasteful efforts is the first question that any lean manufacturing advocate is supposed to ask. Learning to witness wasteful efforts and then consistently eliminating them has allowed lean companies like Toyota to dominate the whole industry.

Eric Ries stated that lean thinking defines value as providing benefit to the customer; anything apart from this is a waste. Identifying who the customer is and what the customer might find valuable in a startup is unestablished. This part of the uncertainty is an essential part of the startup’s definition. Eric realized that as a startup, they needed a new definition of value. The actual improvement they had made at IMVU was what they had learned over the first few months about creating value for customers. Anything they had done during those months that didn’t contribute to their learning was a waste. It would have been possible to learn the same insights from the customers with less effort and waste. For example, what if they had offered customers the chance to download the product only based on its proposed features before building the product. At first, no customers wanted to use their original product, so they wouldn’t have had to apologize if they failed to deliver. They could have experimented by offering customers a chance to try something and then measure their behaviour. Based on these experiences, Eric has learnt that learning is essential to startup progress. The effort that isn’t important for learning what customers want can be avoided. This is what Eric calls validated learning, as it’s always shown by positive improvements in the startup’s main metrics. It’s easy to fool yourself about what you think customers want, and it’s also easy to learn things that are not relevant. Therefore, validated learning is supported by practical data gained by actual customers.

3.5 Where can we discover validation?

Eric adopted the view that their job was to discover a union between their vision and what customers would accept, and it wasn’t to surrender to what customers thought they wanted or to tell customers what they should want. As they started understanding their customers better, they could improve their products. As they did that, the essential metrics of their business changed. Despite their efforts to improve the product in the first few days, their metrics were flat. They treated each customer every new day as new feedback. They had paid attention to the percentage of new customers who exhibited product behaviours like downloading and buying their product. Nearly the same number of customers would buy the product each day, which was almost close to 0 despite the number of changes made. However, since they pivoted from the original strategy, things changed. Consistent with a better strategy, their product development efforts became more effective as they worked smarter in line with their customers’ actual needs. Positive changes to the metrics were the quantitative validation that their learning was genuine. This was important as they could show their stakeholders, such as employees, investors and themselves, that they were making real progress and not impeding themselves. It was also the right way to think about the startup’s productivity regarding how much validated learning they received for their efforts.

For example, in one experiment, they changed their website’s homepage and product registration flow to replace the avatar chat with 3D IM. New customers were divided between these two versions. They could measure the difference in behaviour between the two customers. Not only were these customers most likely to buy the product, but they were most likely to become long-term customers. They had many failed experiments too. One time when they believed that customers weren’t using the product as they didn’t understand its benefits, they paid customer service agents to act as virtual tour guides for new customers. Nevertheless, customers were not interested.

Even after putting away the IM add-on strategy, it took months to understand why it didn’t work. After their pivot and many failed experiments, they finally understood that customers wanted to use IMVU to make new friends online. Their customer wanted something which took Eric and his team a long time to discover. All the current social products online were focused on customers’ actual identities. However, IMVU’s avatar technology was designed to help people to get to know each other without risking their safety or being subject to identity theft. Once they formed this assumption, their experiments produced positive results. Whenever they changed the product to make it easier for customers to find new friends, customers were most likely to engage. This was actual startup productivity that was effectively discovering the right things to build.

3.6 The audacity of zero

Although IMVU had early improvements, its gross numbers were small. This was because of the way businesses are studied, which is dangerous. As Eric stated, it’s always easier to raise money or get other resources when there is zero revenue, zero customers and zero traction than when you have a small amount of revenue. Zero attracts imagination, but small numbers raise questions about whether large numbers will occur.

Many people know or think they know success stories of products that have been successful overnight. If nothing has been released and no data has been collected, it’s still possible to imagine success overnight in the future. This situation creates a dangerous motive: delay getting data until you’re sure about success. Such delays have the consequences of increasing wasted work, decreasing important feedback and increasing the risk that a startup will make something that no one wants.

However, releasing a product to the market and hoping for the best isn’t a good idea either. When they launched IMVU, they were not aware of this problem. Their initial investors and advisers felt that a 300-dollar revenue plan per month was assessable, but after months with their revenue around 500 per month, some started losing faith. Thankfully, their numbers improved significantly as they pivoted and experimented, incorporating what they learned into their product development and marketing efforts. They could see an improvement that started to look like the famous hockey stick graph. However, the graph went up only to a few thousand dollars per month. Although the graph was convincing, they alone were not enough to fight the loss of faith caused by the early failure, and they lacked validated learning to provide an alternative concept to support. They were lucky to have some early investors that understood the graph’s importance and were willing to look beyond their small gross numbers to see the actual improvement.

Therefore, with validated learning, you can reduce the waste that occurs due to the audacity of zero.

3.7 Wisdom after IMVU

Although the tactics from IMVU may or might not make sense in your type of industry, it’s better to learn to see each startup in any industry as a vast experiment. The question is not whether a particular kind of product can be built. In the modern world, any product that can be imagined can be built. The most appropriate questions are “should this product be built?” and “can we build a sustainable business around this set of products and services?” according to Eric Ries. To answer these questions, you need a method to systematically break down a business plan into parts and test each part experimentally.

Conclusion

In the Lean Startup model, each product, feature, and marketing campaign a startup does is understood to be an experiment built to achieve validated learning. This approach will be across all industries and sectors, which we will look at in chapter 4.