Running Assumption Tests

The Identifying Hidden Assumptions chapter (Chapter 9) opened with a quote from Marty Cagan, in which he argued the best teams conduct 15–20 discovery iterations a week. This can sound like an overwhelming number of assumption tests. But with the right mindset, tools, and methods, it can quickly become a reality. In Chapter 9, we learned that the secret to unlocking this cadence is testing assumptions, not whole ideas. However, we still need to learn how to quickly execute our assumption tests.

There are two tools that should be in every product team’s toolbox—unmoderated user testing and one-question surveys. Unmoderated user-testing services allow you to post a stimulus (e.g., a prototype) and define tasks to complete and questions to answer. Participants then complete the tasks and answer the questions on their own time. You get a video of their work. These types of tools are game changers. Instead of having to recruit 10 participants and run the sessions yourself, you can post your task, go home for the night, and come back the next day to a set of videos ready for you to watch.

If we look at the two simulations we designed above, both could be conducted with unmoderated testing tools. Once results come in, we would simply have to watch the videos and record how many chose sports in the first assumption test and how many chose our subscription service in the second assumption test. What used to take weeks to recruit, schedule, and conduct a prototype test can now be done in a day or two.

To make unmoderated testing work well, you need to be thoughtful about who you recruit. With both of our tests, we need to recruit our own subscribers. So, we’ll need to screen for this. We also want to pay particular attention to variation (as discussed above). This is also something we can screen for. Some unmoderated testing tools also allow you to upload your own list of participants. This is particularly helpful when testing with niche audiences.

Many assumptions can be tested with quick answers to a single question. This is where one-question survey tools can be tremendously helpful. If we wanted to test the “Our subscribers want to watch sports” assumption in more than one way, we could launch a one-question survey asking them, “When was the last time you watched a sporting event?” We could use their answers to triangulate with our prototype simulation.

Sometimes we simply need to learn about our customers’ preferences. For example, if we were testing the assumption “Our platform has the sports our subscribers want to watch,” we could test this with a one-question survey. We could ask, “Please select all the sports you’ve watched in the past month.”

When using one-question surveys, we want to make sure we are following the same research rules we’ve outlined before. When asking about past behavior, we want to ask about specific instances (as you learned in Chapter 5). So, we are asking about the last week and the last month, not in general. We also want to avoid asking about what they might do in the future. We know this leads to unreliable data.

Sometimes you can use one-question surveys to simulate an experience. For example, if one of our ideas depends on the assumption “Our subscribers will tell us who their favorite sports teams are,” you might be tempted to ask customers, “Are you willing to tell us who your favorite sports teams are?” But this is a question about future behavior. The answers are unreliable. Instead, ask, “What are your favorite sports teams?” Evaluate the results based on the percentage of people who answer as compared to the percentage of people who skipped it.

However, unmoderated testing and one-question surveys aren’t the only ways to test assumptions. Sometimes we already have the data we need in our own database. For example, we might look at how many of our current subscribers have searched for sports on our platform and use this as an indicator of interest in sports. Before you dive into the data, be sure to define your evaluation criteria upfront. How many search queries will you sample? How many need to be related to sports? How will you determine “related to sports”? Remember, aligning around success criteria upfront guards against confirmation bias and ensures that your team agrees on what the results mean.

Product teams can typically test most of their assumptions with a combination of prototype tests (either unmoderated or in person), one-question surveys, or through data-mining. However, there are dozens of experiment types. If you want to do a deep dive on qualitative tests, pick up a copy of Laura Klein’s UX for Lean Startups. She does a good job of surveying a wide breadth of methods. Another great reference is David Bland’s Testing Business Ideas. The last third of David’s book is an encyclopedia of experiment types. However, don’t get overwhelmed with having to master all of these experiment types. If you keep the simple assumption-simulate-evaluate framework in mind, you’ll be well on your way to becoming a strong assumption tester.