Avoid These Common Anti-Patterns

As you design and run your assumption tests, keep these common anti-patterns in mind:

Overly complex simulations. Some teams spend countless hours, days, or even weeks trying to design and develop the perfect simulation. It’s easy to lose sight of the goal. In your first round of testing, you are looking to design fast tests that will help you gather quick signals. Design your tests to be completed in a day or two, or a week, at most. This will ensure that you can keep your discovery iterations high.

Using percentages instead of specific numbers when defining evaluation criteria. Many teams equate 70% and 7 out of 10. So instead of defining their evaluation criteria as 7 out of 10, they tend to favor the percentage. These sound equivalent, but they aren’t. First, when testing with small numbers, we can’t conclude that 7 out of 10 will continue to mean 70% as our participant size grows. We want to make sure that we don’t draw too strong a conclusion from our small signals. Second, and more importantly, “70%” is ambiguous. If we test with 10 people and only 6 exhibit our desired behavior, some of us might conclude that the test failed. Others might argue that we need to test with more people. Be explicit from the get-go about how many people you will test with when defining your success criteria.

Not defining enough evaluation criteria. It’s easy to forget important evaluation criteria. At a minimum, you need to define how many people to test with and how many will exhibit the desired behavior. But for some tests, defining the desired behavior may involve more than one number. For example, if your test involves sending an email, you might need to define how many people will receive the email, how long you’ll give them to open the email, and whether your success criteria is “opens” or “clicks.” Pay particular attention to the success threshold. Complex actions may require multiple measurements (e.g., opens the email, clicks on the link, takes an action).

Testing with the wrong audience. Make sure that you are testing with the right people. If you are testing solutions for a specific target opportunity, make sure that your participants experience the need, pain point, or desire represented by that target opportunity. Remember to recruit for variation. Don’t just test with the easiest audience to reach or the most vocal audience.

Designing for less than the best-case scenario. When testing with small numbers, design your assumption tests such that they are likely to pass. If your assumption test passes with the most likely audience, then you can expand your reach to tougher audiences. This might feel like cheating, but you’ll be surprised how often your assumption tests still fail. If you fail in the best-case scenario, your results will be less ambiguous. If your test fails with a less-than-ideal audience, someone on the team is going to argue you tested with the wrong audience, and you’ll have to run the test again. Remember, we want to design our tests to learn as much as we can from failures.