Good question / bad question

Good question / bad question

CHAPTER ONE

    

Let’s play a game. Are the following questions good or bad? Do they pass or fail The Mom Test? If they fail it, why? And how could we improve them? Work your way through the list and then read on for some discussion.

“Do you think it’s a good idea?”

“Would you buy a product which did X?”

“How much would you pay for X?”

“What would your dream product do?”

“Why do you bother?”

“What are the implications of that?”

“Talk me through the last time that happened.”

“Talk me through your workflow.”

“What else have you tried?”

“Would you pay X for a product which did Y?”

“How are you dealing with it now?”

“Where does the money come from?”

“Who else should I talk to?”

“Is there anything else I should have asked?”

"Do you think it's a good idea?"

Awful question! Here’s the thing: only the market can tell if your idea is good. Everything else is just opinion. Unless you’re talking to a deep industry expert, this is self-indulgent noise with a high risk of false positives.

Let’s fix it: Say you’re building an app to help construction companies manage their suppliers. You might ask them to show you how they currently do it. Talk about which parts they love and hate. Ask which other tools and processes they tried before settling on this one. Are they actively searching for a replacement? If so, what’s the sticking point? If not, why not? Where are they losing money with their current tools? Is there a budget for better ones? Now, take all that information and decide for yourself whether it’s a good idea. 

Rule of thumb: Opinions are worthless.

"Would you buy a product which did X?"

Bad question. You’re asking for opinions and hypotheticals from overly optimistic people who want to make you happy. The answer to a question like this is almost always “yes”, which makes it worthless.

Let’s fix it: Ask how they currently solve X and how much it costs them to do so. And how much time it takes. Ask them to talk you through what happened the last time X came up. If they haven’t solved the problem, ask why not. Have they tried searching for solutions and found them wanting? Or do they not even care enough to have Googled for it?

Rule of thumb: Anything involving the future is an over-optimistic lie.

"How much would you pay for X?"

Bad question. This is exactly as bad as the last one, except it’s more likely to trick you because the number makes it feel rigorous and truthy.

How to fix it: Just like the others, fix it by asking about their life as it already is. How much does the problem cost them? How much do they currently pay to solve it? How big is the budget they’ve allocated? I hope you’re noticing a trend here.

Rule of thumb: People will lie to you if they think it’s what you want to hear.

"What would your dream product do?"

Sort-of-okay question, but only if you ask good follow-ups. Otherwise it’s a bad question. A question like this is like the “set” before the spike in a volleyball game: not too helpful on its own, but it puts you in a good position as long as you’re ready to exploit it.

Let’s improve it: The value comes from understanding why they want these features. You don’t want to just collect feature requests. You aren’t building the product by committee. But the motivations and constraints behind those requests are critical.

Rule of thumb: People know what their problems are, but they don’t know how to solve those problems.

"Why do you bother?"

Good question. I love this sort of question. It’s great for getting from the perceived problem to the real one. 

For example, some founders I knew were talking to finance guys spending hours each day sending emails about their spreadsheets. The finance guys were asking for better messaging tools so they could save time. The “why do you bother” question led to “so we can be certain that we’re all working off the latest version.” Aha. The solution ended up being less like the requested messaging tool and more like Dropbox. A question like “why do you bother” points toward their motivations. It gives you the why. 

Rule of thumb: You're shooting blind until you understand their goals.

"What are the implications of that?"

Good question. This distinguishes between I-will-pay-to-solve-that problems and thats-kind-of-annoying-but-I-can-deal-with-it “problems”. Some problems have big, costly implications. Others exist but don’t actually matter. It behooves you to find out which is which. It also gives you a good pricing signal.

I once had someone keep describing the workflow we were fixing with emotionally loaded terms like “DISASTER”, accompanied by much yelling and arm waving. But when I asked him what the implications were, he sort of shrugged and said “Oh, we just ended up throwing a bunch of interns at the problem—it’s actually working pretty well.”

Rule of thumb: Some problems don’t actually matter.

"Talk me through the last time that happened."

Good question. Your high school writing teacher may have told you that good stories are meant to “show, not tell”. Whenever possible, you want to be shown, not told by your customers. Learn through their actions instead of their opinions. If you ran a burger joint, it would be stupid to survey your customers about whether they prefer cheeseburgers or hamburgers. Just watch what they buy (but if you’re trying to understand why they prefer one over the other, you’ll have to talk to them).

Folks can’t be wishy-washy when you’re watching them do the task in question. Get as close to the real action as you can. Seeing it first hand can provide unique insight into murky situations. But if you can’t get in there, asking them to talk you through the last time it happened still offers many of the benefits. 

Being walked through their full workflow answers many questions in one fell swoop: how do they spend their days, what tools do they use, and who do they talk to? What are the constraints of their day and life? How does your product fit into that day? Which other tools, products, software, and tasks does your product need to integrate with?

Rule of thumb: Watching someone do a task will show you where the problems and inefficiencies really are, not where the customer thinks they are.

"What else have you tried?"

Good question. What are they using now? How much does it cost and what do they love and hate about it? How much would those fixes be worth and how big of a pain would it be for them to switch to a new solution?

I was checking out an idea with a potential customer and they excitedly said, “Oh man, that happens all the time. I would definitely pay for something which solved that problem.” 

That’s a future-promise statement without any commitment to back it up, so I needed to learn whether it was true or not. I asked, “When’s the last time this came up?” Turns out, it was pretty recent. That’s a great sign. To dig further, I asked, “Can you talk me through how you tried to fix it?” He looked at me blankly, so I nudged him further. 

“Did you google around for any other ways to solve it?” He seemed a little bit like he’d been caught stealing from the cookie jar and said, “No… I didn’t really think to. It’s something I’m used to dealing with, you know?” 

In the abstract, it’s something he would “definitely” pay to solve. Once we got specific, he didn't even care enough to search for a solution (which do exist, incidentally).

It’s easy to get someone emotional about a problem if you lead them there. “Don’t you hate when your shoelaces come untied while you’re carrying groceries?” “Yeah, that’s the worst!” And then I go off and design my special never-come-untied laces without realising that if you actually cared, you would already be using a double-knot. 

Rule of thumb: If they haven't looked for ways of solving it already, they're not going to look for (or buy) yours.

"Would you pay X for a product which did Y?"

Bad question. The fact that you’ve added a number doesn’t help. This is bad for the same reasons as the others: people are overly optimistic about what they would do and want to make you happy. Plus, it’s about your idea instead of their life.

Let’s fix it: As always, ask about what they currently do now, not what they believe they might do in the future. Common wisdom is that you price your product in terms of value to the customer rather than cost to you. That's true. And you can't quantify the value received without prodding their financial worldview.

Another way to fix it, if you’re far enough along, is to literally ask for money. If you have the deposit or pre-order in hand, you know they were telling the truth.

"How are you dealing with it now?"

Good question. Beyond workflow information, this gives you a price anchor. If they’re paying £100/month for a duct-tape workaround, you know which ballpark you're playing in. On the other hand, they may have spent £120,000 this year on agency fees to maintain a site you're replacing. If that's the case, you don't want to be having the £100 conversation.

Sometimes, both of the above will be happening simultaneously and you get to choose how you present yourself. Do you want to be a replacement for the web app at a yearly value of £1.2k or for the agency at 100x that?

Rule of thumb: While it’s rare for someone to tell you precisely what they’ll pay you, they’ll often show you what it’s worth to them.

"Where does the money come from?"

Good question. This isn't something you would necessarily ask a consumer (though you might), but in a B2B context it’s a must-ask. It leads to a conversation about whose budget the purchase will come from and who else within their company holds the power to torpedo the deal. 

Often, you'll find yourself talking to someone other than the budget owner. Your future pitches will hit unseen snags unless you learn who else matters and what they care about. This knowledge of their purchasing process will eventually turn into a repeatable sales roadmap.

"Who else should I talk to?"

Good question. Yes! End every conversation like this. Lining up the first few conversations can be challenging, but if you’re onto something interesting and treating people well, your leads will quickly multiply via intros. 

If someone doesn’t want to make intros, that’s cool too. Just leave them be. You’ve learned that you’re either screwing up the meeting (probably by being too formal, pitchy, or clingy) or they don’t actually care about the problem you’re solving. Take anything nice they say with an extra grain of salt.

"Is there anything else I should have asked?"

Good question. Usually, by the end of the meeting, people understand what you’re trying to do. Since you don’t know the industry, they’ll often be sitting there quietly while you completely miss the most important point.

Asking this question gives them a chance to politely “fix” your line of questioning. And they will!

This question is a bit of a crutch: you’ll discard it as you get better at asking good questions and as you get to know the industry.

Rule of thumb: People want to help you, but will rarely do so unless you give them an excuse to do so.

Using the mom test

You’ll notice that none of the good questions were about asking what you should build. One of the recurring “criticisms” about talking to customers is that you’re abdicating your creative vision and building your product by committee. Given that people don’t know what they want, that wouldn’t be a terribly effective approach. Deciding what to build is your job.

The questions to ask are about your customers’ lives: their problems, cares, constraints, and goals. You humbly and honestly gather as much information about them as you can and then take your own visionary leap to a solution. Once you’ve taken the leap, you confirm that it’s correct (and refine it) through Commitment & Advancement, which we’ll look at in Chapter 5.

It boils down to this: you aren’t allowed to tell them what their problem is, and in return, they aren’t allowed to tell you what to build. They own the problem, you own the solution. 

Before we move look at ways to confirm that you’re building the right product to fit your customers’ lives, we’re going to look at fixing some of the ways conversations go wrong.