A Quick Word on Science
Science-minded readers might cringe at these quick-and-dirty research methods. However, product teams are not scientists. Scientists work to understand the fundamental laws of the universe. They are seeking truths, creating new knowledge. In science (and the rest of academia), truth is determined over decades. Research studies are designed and replicated by a community of scientists. Truth starts to emerge from a meta-analysis of years of research. Even then, our truths don’t always stand the test of time. Newton’s laws of physics are true in some contexts, but with quantum physics, we are learning there are contexts in which they are not true. Newton’s truths are incomplete. In social science, we think a landmark research study means one thing, until years of more research show there was a confounding variable that wasn’t accounted for. Social-science truths often evolve. Research is messy, and creating new knowledge is hard.
Product teams, fortunately, are not creating new knowledge. Instead, we are trying to create products that improve our customers’ lives. When we launch a new product or service, we get to see how our customers interact with it. This is a fantastic feedback loop. We quickly get to measure if our product had the intended impact. We work on much faster cycles than science. It took almost 100 years before we could collect physical evidence to support Einstein’s theory of general relativity.53
So, while we want to adopt a scientific mindset and we want to think about the reliability and the validity of the data that we collect, we are not running scientific experiments. While we need to be thoughtful about our research methods, we also need to be aware that we are not validating or invalidating anything. It’s important that we recognize that our research findings are not truths—they are merely confirming or disconfirming evidence that either supports or refutes our point of view. Our goal as a product team is not to seek truth but to mitigate risk. We need to do just enough research to mitigate the risk that our companies cannot bear and no more.
This understanding of the intent behind our research will help us strike the right balance between sound research methods and the pace at which we need to work to get products into our customers’ hands.