CHAPTER THREE

Asking important questions

Once we know about The Mom Test and start trying to ask non-biasing questions, sometimes we over-compensate and ask completely trivial ones. Asking someone how old they are isn’t biasing, but it also doesn’t move your business forward. You have to apply The Mom Test to the questions which matter. Otherwise you’re just spinning your wheels.

In addition to ensuring that you aren’t asking trivialities, you also need to look for the world-rocking scary questions you’re shrinking from. The best way to find them is to run thought experiments. Imagine that the company has failed and ask why that happened. Then imagine it as a huge success and ask what had to be true to get there. Find ways to learn about those critical pieces.

You can tell it’s an important question when the answer to it could completely change (or disprove) your business. If you get an unexpected answer to a question and it doesn’t affect what you’re doing, it wasn’t a terribly important question.

Every time you talk to someone, you should be asking a question which has the potential to completely destroy your currently imagined business. 

One of my companies had some legal ambiguities around content ownership. It was okay in theory, but lacked strong precedents. I was always a bit nervous we would get “found out.” Their execs were excited, their creatives were thrilled. We had even brought the stubborn tech team around to our side. But in all my early customer conversations, I never asked to talk to their lawyers. For whatever reason, that was an important question which I shrunk from, and not asking it cost us at least half a million bucks.

There’s no easy solution to making yourself face and ask these questions. I once heard the general life advice that, for unpleasant tasks, you should imagine what you would have someone else do if you were delegating it. Then do that. 

And remember, you’re allowed to ask about money. You're a startup. It's okay.

Love bad news

One of the reasons we avoid important question is because asking them is scary. It can bring us the upsetting realisation that our favourite idea is fundamentally flawed. Or that the major client is never going to buy. Although it seems unfortunate, this we need to learn to love bad news. It’s solid learning and is getting us closer to the truth.

If you’ve only got one shot, then bad news is bad news. If your bungee jump doesn’t work, that’s bad. If you’ve sunk your retirement savings into opening a cafe and it doesn’t work, that’s bad. If you hustle together $50k to start your business and spend all $50k on your first idea only to see it fail, that’s bad.

On the other hand, if you have $50k and spend $5k to learn you’re running down a dead end, that’s awesome. You can use the rest to find a viable path to your goal.

Similarly, if you have an exciting idea for a new product and go talk to a couple customers who don’t actually care about it, then that’s a great result. You just saved yourself however much time and money it would have cost to try building and selling it. 

We go through the futile process of asking for opinions and fish for compliments because we crave approval. We want to believe that the support and sign-off of someone we respect means our venture will succeed. But really, that person’s opinion doesn’t matter. They have no idea if the business is going to work. Only the market knows. 

You’re searching for the truth, not trying to be right. And you want to do it as quickly and cheaply as possible. Learning that your beliefs are wrong is frustrating, but it’s progress. It’s bringing you ever closer to the truth of a real problem and a good market.

The worst thing you can do is ignore the bad news while searching for some tiny grain of validation to celebrate. You want the truth, not a gold star. 

Some of the most informative (and thus best) responses you can get are along the lines of, “Umm, I’m not so sure about that” and "That's pretty neat." Both are lukewarm responses which tell you they don’t care.

In this context, “best” means learning, not selling. If you’re a sales guy going door to door selling gadgets and someone doesn’t care, then that is a bad result: you’re not getting paid. But if you’re trying to decide whether to invest your time and money in developing, building and promoting that gadget, then lukewarm is a terrific response. It gives you a crystal clear signal that this person does not care. It’s perfectly reliable information you can take to the bank. 

The classic error in response to a lukewarm signal is to “up your game” and pitch them until they say something nice. Unless they’re holding a check, the only thing to gain from “convincing” them are false positives. You’re not here to collect compliments; you’re trying to learn the truth. Their lukewarm response already gave you that.

If they’re still engaged in the conversation, it’s worth asking a couple follow-up questions to understand the nature of their apathy. Is the “problem” not actually that big of a deal? Are they fundamentally different from your ideal customers? Do they not care about the specific implementation? Are they worn out and skeptical from hearing too many pitches, like cafe owners in the aftermath of Groupon? Are they just plain tired today?

After that, say a big thanks and leave them to their day. They’ve probably helped you more than the guy who said he loved it.

Look before you zoom

Another way we miss the important questions is by instead spending our time on ultimately unimportant details. This can happen when we get stuck in the details before understanding the big picture.

Most people have lots of problems which they don’t actually care enough about to fix, but which they’ll happily tell you the details of if you ask them. Before you have solid evidence that you’re fixing a meaningful problem for your customer segment, you can really mess yourself up by zooming in too quickly.

A (really, really) bad conversation:

The reason this conversation is so very bad is because, if you aren’t paying attention, it seems like it went well. If you focus the conversation too quickly on one problem area, you can think you’ve validated a “top” problem when you haven’t. You just led them there.

If you ask me what my biggest problem with staying fit is, I’ll probably tell you it’s the time cost of going to the gym. But then, if you build me a workout-at-home app, I’m going to ignore it. Even though commute time is the biggest problem with getting in shape, the whole area of fitness just isn’t something I care about enough to act on. My #1 fitness problem is still an unimportant one.

Let’s start the conversation again and fix it when it starts to go off track:

A good conversation:

The premature zoom is a real problem because it leads to data which seems like validation, but is actually worthless. In other words, it’s a big source of false positives.

Let’s re-run the same conversation, but instead of immediately zooming in on exercise, we’ll start more generic, since we aren’t sure that fitness is a must-solve problem:

A good conversation:

However, we don’t always need to start the conversation from the square one of do-they-care-at-all. Sometimes we know the problem exists as a top priority and we can safely zoom in immediately. 

For example, let’s say we help drive qualified leads to small business websites. We know that marketing is always a top 3 problem for small businesses, so we can focus on it and start the conversation by zooming in with a question like:

Versus:

Now imagine you’ve built the aforementioned marketing tool for small businesses and realise it could also be used to help bloggers get more attention. You’re wondering if you have another strong customer segment. However, since blogs have a harder time monetising traffic than small businesses, we can’t necessarily assume that they’re also going to happily pay for traffic.

For example, I have a blog which I quite like (but which I don’t take seriously as a business). If you’re talking to me and you ask, “Hey Rob, what are your top problems with marketing your blog?” Then you’ve just prematurely zoomed on a non-problem. 

I’m going to tell you some sort of answer which sounds legit but is actually misleading. I might say that my keywords are all kind of generic, so it’s hard to attract the right people. Or that I walk a fine line between writing beginner and advanced articles, so it’s tough to attract the right people. Whatever. Problems exist with my blog marketing, but the whole space just isn’t something I’m too fussed about. I write because it’s fun, not because it’s how I pay my rent.

To have a useful conversation, you need to zoom back out to ask about my blog in general, rather than marketing my blog.

A good conversation:

At this point, you might be suspicious about how much I actually care about my blog. When it’s not clear whether a problem is a must-solve-right-now (e.g. you’re selling a painkiller) or a nice-to-have (you’re selling a vitamin), you can get some clarity by asking cost/value questions like the following.

“Does-this-problem-matter” questions:

• “How seriously do you take your blog?” 
• “Do you make money from it?”
• “Have you tried making more money from it?”
• “How much time do you spend on it each week?”
• “Do you have any major aspirations for your blog?”
• “Which tools and services do you use for it?”
• “What are you already doing to improve this?”
• “What are the 3 big things you’re trying to fix or improve right now?”

Some of these questions are generic, but give us signals that we can anchor on and dig around. The bulk of them are about finding out whether the person we’re talking to is taking this space seriously. Are they spending money or making money? Is it in their top 3? Are they actively looking for solutions?

When you fall into a premature zoom, you can waste a ton of time figuring out the minutia of a trivial problem. Even if you learn everything there is to know about that particular problem, you still haven’t got a business.

Look at the elephant

Lastly, sometimes we comfort ourselves by asking questions which don’t actually de-risk the business or resolve those critical, big, scary, lurking questions. We ignore the elephant in the room.

Let’s say we suspect that teachers from the poorest schools are completely overloaded, and that our tools would save them time so they could better educate their students. We go talk to them and confirm that yes, they are completely overloaded. We then spend weeks with them, figuring out exactly what their dream tool would do. Unfortunately, we've missed the elephant, which is that the poorest schools may not have the budgets available to pay us what we need. We're liable to spend a huge amount of time exploring a real and urgent problem, only to hop into the deadpool due to our customer's budgeting issues. 

Successful startups tend to depend on multiple failure points. In this case it is both the needs of the teachers and the ability of the schools to pay us. If any of these conditions doesn't exist, we have to significantly overhaul our idea. It's tempting to obsess over the most interesting of several failure points and ignore the others. It's a great way to miss important questions.

Beyond the risks of our customers and market, we also have challenges with our own product. Overlooking the product risks is just as deadly as overlooking the goals and constraints of our customers. Take the following conversation with a professional public speaker. Is it full of good data or bad data?

An ambiguous conversation:

At this point, let’s say I’m confident that getting gigs is important to him. I also know what it’s worth and how he’s currently accomplishing it. So I zoom in to introduce the problem I’m solving and the way I want to approach it.

So what’s the result? Beyond being excited, we got some concrete data about how valuable this could be to him. Plus, it involved a commitment to be one of our early users. It sure seems good.

But what did he actually tell us? He said that if we can get him more gigs, then he’ll pay a cut. Well, obviously. Who doesn’t want free money? His needs are clear: he wants to make more money by speaking. If we can send him work, he’ll share some of it with us. This was never really in doubt. 

The key phrase of “if you could get me more gigs” is basically shifting the burden from the customer to your product. Even though you’ve found a pain, your success is dependent on a bunch of other factors, such as your ability to grow a healthy supply of paying gigs which are a good fit for him. Will you be able to do that? It remains to be seen. 

This situation is easier to spot in the online advertising industry. Imagine running customer conversations with an advertiser to try to understand their pains so you can convince them to advertise on your site. They’d sort of look at you blankly and say, “Listen, if you can get enough page views, we’ll pay you for them.” In fact, it’s such a well established would-pay-to-solve-problem that you don’t even need to talk to them to set it up. You just plug in an ad network and you’re done.

Similarly with affiliate commissions. If you sell a company’s products, you get a cut. That’s just how it works. You don’t need to explore or validate or understand their problems. The risk resides in your ability to get lots of traffic and sell lots of products. If you can, they’ll pay you. 

In all of these examples, the risk is in your product, not in the market or the customer. The customer will pay if your product gets big enough.

• Product risk — Can I build it? Can I grow it? Will they keep using it?
• Market risk — Do they want it? Will they pay? Are there enough of them?

You don’t want to overlook one or the other. I met a founder who had wasted 3 months on worthless customer conversations. He wanted to start a company building gadgets to track the fertility of farm animals, ultimately boosting birthrates and thus revenue. When he talked to farmers, he asked questions like, “Would you switch trackers if something cheaper and more effective was available?” That’s the same as asking someone whether they would like more money. The answer is always “Yes.” The farmers responded along the lines of, “If you can build what you say you can build, I’ll equip my whole herd.” The problem is, he couldn’t build it. The risk was in the product.

I’ve also seen this strike several of the recent companies who want to use mobile/realtime deals to drive foot traffic to bars and clubs. They run customer conversations with bar owners who confirm that: yes, they would like more customers on the slow nights; and yes, they would pay you if you could send customers on demand. The founders take this as strong validation (“They have the problem and committed to pay!”) without recognising that the vast majority of the risk is in the product, not the market. Bars will pay, but only if you can amass a huge audience of consumers. Then the founders talk to consumers and ask if they would use an app which always pointed them to booming parties with cheap booze. Again, obviously yes. But that doesn’t tell us whether we can actually get to that critical mass of users.

Video games are pure product risk. What sort of question could you ask to validate your game idea? “Do you like having fun?” “Would you like to have even more fun?” Practically 100% of the risk is in the product and almost none is in the customer. You know people buy games. If yours is good and you can find a way to make them notice it, they’ll buy it. You don’t need to rediscover people’s desire to play video games.

This isn’t to say that you shouldn’t talk to anyone if you’re building something with product risk. In the case of the farm fertility monitor, it’s good to know that the farmer isn’t opposed to switching tech once he’s already got something installed, for example. For the nightclubs, it’s good to know that they’re at least theoretically willing to pay for promotion. It would be tragic to succeed at the hard work of creating the technology or community only to learn nobody will pay for it. 

What all this does mean is that if you’ve got heavy product risk (as opposed to pure market risk), then you’re not going to be able to prove as much of your business through conversations alone. The conversations give you a starting point, but you’ll have to start building product earlier and with less certainty than if you had pure market risk.

Prepare your list of 3

Always pre-plan the 3 most important things you want to learn from any given type of person. 

Pre-planning your big questions makes it a lot easier to ask questions which pass The Mom Test and aren’t biasing. It also makes it easier to face the questions that hurt. If we go through an unplanned conversation, we tend to focus on trivial stuff which keeps the conversation comfortable. Instead, decide what to ask with your team in a calm environment.

Your 3 questions will be different for each type of person you’re talking to. If you have multiple types of customers or partners, have a list for each.

Don’t stress too much about choosing the “right” important questions. They will change. Just choose the 3 questions which seem murkiest or most important right now. Doing so will give you firmer footing and a better sense of direction for your next 3.

You might get answers 1-3 from customer A, answer 4 from customer B, answers 5-7 from customer C. There’s overlap and repetition, but you don’t need to repeat the full set of questions with every participant. Don’t feel obliged to repeat questions you already have reliable data on. Pick up where you left off and keep filling in the picture.

Knowing your list allows you to take better advantage of serendipitous encounters. Instead of running into that dream customer and asking to exchange business cards so you can “grab a coffee” (exactly like everyone else), you can just pop off your most important question. And that goes a long way toward Keeping it Casual.