Revisiting Different Types of Outcomes

In Chapter 3, we distinguished between business outcomes, product outcomes, and traction metrics as a way to help us set the scope for our product work. In the AfterCollege story, our product outcome was to improve search starts, and we succeeded at doing that. But our business outcome was to increase the number of students who found jobs on our platform. We had to continue to instrument our product to evaluate if driving our product outcome had the intended impact on our business outcome. This work took longer than expected. We didn’t want to wait to have the final answer before pushing value to our customers, so we continued to experiment in our production environment. Our discovery continued through to delivery.

This isn’t uncommon. However, in the AfterCollege story, it was easy for us to experiment in production. We were able to get a working prototype live in only a few days. Let’s return to our streaming-entertainment example to work through a more complex case. We can learn a lot about our subscribers’ interest in sports by running the assumption tests we defined in Chapter 10. However, to test if adding sports will drive our product outcome (to increase average minutes watched) and our business outcome (to increase subscriber retention), we’ll need to find small ways to experiment with real data—in other words, in our production environment.

We can’t test if watching sports on our platform will increase viewer minutes until we have sports on our platform. This might look like a Catch-22, but it’s not. We don’t need to test with the full solution to evaluate the impact on our outcomes. For example, we could partner with a local channel to stream one sporting event on one day and evaluate the impact on viewing minutes for the subscribers who watched that sporting event. We can look at whether their overall viewing minutes went up, or if they cut out content in other areas to make time for the sporting event. Integrating all local-channel content might require new business partnerships, contracts to be signed, and APIs to be developed. But starting with one event might allow you to circumvent a good chunk of that work, allowing both parties to test their assumptions before they commit to a longer-term agreement.

Just like in the AfterCollege story, it might take even more time to evaluate if sports content will drive the business outcome (increased subscriber retention). Streaming one sporting event likely won’t have a noticeable impact on subscriber numbers. However, if it impacts viewer minutes, we can keep investing, working on the belief that increasing viewing minutes will increase subscriber retention. The key in both examples is to remember to track the long-term connection between your product outcome and your business outcome. If, over time, an increase in viewing minutes doesn’t lead to an increase in subscriber retention, then the team will need to find a new product outcome that does drive the business outcome.