CHAPTER TWO

Avoiding bad data

They say that to bankrupt a fool, give him information. Practically everyone I’ve seen talk to customers (including myself) has been giving themselves bad information. You probably are too. Bad data gives us false negatives (thinking the idea is dead when it’s not) and—more dangerously—false positives (convincing yourself you’re right when you’re not). 

There are three types of bad data:

1. Compliments
2. Fluff (generics, hypotheticals, and the future)
3. Ideas

Sometimes we invite the bad data ourselves by asking the wrong questions, but even when you try to follow The Mom Test, conversations still go off track. It could happen because you got excited and started pitching, because you had to talk about your idea to explain the reason for the meeting, or because the conversation is just stuck in hypothetical la-la-land.

These things happen. Once you start to notice, it’s easy to get back on track by deflecting compliments, anchoring fluff, and digging beneath ideas.

Deflect compliments

Most of your meetings will end with a compliment. It feels good. They said they liked it! 

Unfortunately, they’re almost certainly lying. Not necessarily intentionally. They might want to be supportive or to protect your feelings. Or your excitement might be rubbing off on them. 

Even if they really do like it, that data is still worthless. For example, venture capitalists (professional judges of the future) are wrong far more than right. If even a VC's opinion is probably wrong, what weight could that of some random guy's possibly have?

With the exception of industry experts who have built very similar businesses, opinions are worthless. You want facts and commitments, not compliments.

The best way to escape the misinformation of compliments is to avoid them completely by not mentioning your idea. If they happen anyway, you need to deflect the compliment and get on with the business of gathering facts and commitments. 

Before we look at how to properly deflect compliments, here's what happens when you take them at face value:

A bad conversation:

Let’s try that again while properly deflecting the confounding compliments:

A good conversation:

If we’re early in the learning process, the meeting could end here quite happily. We have the learning we came for. If we were slightly later-stage and already had a product, we might continue by zooming in and pushing for commitments or sales.

Remember though: you don’t need to end up with what you wanted to hear in order to have a good conversation. You just need to get to the truth. Here’s a good conversation with a solid negative result.

A good (negative) conversation:

You can always be happy with a conversation like the above. You saw through the false compliment and found the facts behind the mirage. If the conversation is going well, I’d try to have them talk me through their process anyway so I can try to figure out whether it’s an industry wide non-problem or something specific to their particular situation.

Did you notice that in the conversations above, practically every response contains a sneaky compliment? They are pervasive, constantly trying to trick us into thinking the conversation “went well”.

Ignoring compliments should be easy, but it’s not. We so desperately want to hear them that we are often tricked into registering them as positive data points instead of vacuous fibs. Sometimes it’s easier to spot the symptoms than to notice the original compliment.

Symptoms (in the meeting): 

• “Thanks!”
• “I’m glad you like it.”

Symptoms (back at the office):

• “That meeting went really well.”
• “We’re getting a lot of positive feedback.”
• “Everybody I’ve talked to loves the idea.”

All of these are warning signs. If you catch yourself or your teammates saying something like this, try to get specific. Why did that person like the idea? How much money would it save him? How would it fit into his life? What else has he tried which failed to solve his problem? If you don’t know, then you’ve got a compliment instead of real data.

Anchor fluff

Fluff comes in 3 cuddly shapes:

• Generic claims (“I usually”, "I always", "I never") 
• Future-tense promises (“I would”, "I will")
• Hypothetical maybes ("I might", "I could")

When someone starts talking about what they “always” or “usually” or “never” or “would” do, they are giving you generic and hypothetical fluff. Follow The Mom Test and bring them back to specifics in the past. Ask when it last happened, for them to talk you through it, how they solved it, and what else they tried. 

The world’s most deadly fluff is: “I would definitely buy that.” 

It just sounds so concrete. As a founder, you desperately want to believe it’s money in the bank. But folks are wildly optimistic about what they would do in the future. They’re always more positive, excited, and willing to pay in the imagined future than they are once it arrives.

The first startup I worked at fell for the “I would definitely buy that” trap and subsequently lost about 10 million bucks. They mistook fluffy future promises and excited compliments for commitment, wrongly believed they had proven themselves right, and wildly over-invested.

The worst type of fluff-inducing question is, “Would you ever?” Of course they might. Someday. That doesn't mean they will. Fluff-inducing questions include:

• “Do you ever…”
• “Would you ever…”
• “What do you usually…”
• “Do you think you…”
• “Might you…”
• “Could you see yourself…”

You don’t need to avoid these questions 100% of the time. They aren’t exactly toxic. It’s just that the responses are useless. The mistake is in valuing the answers, not asking the questions. In fact, sometimes these questions can even help you transition into more concrete questioning. 

Transitioning from a fluffy question to a concrete one:

To use a more tangible example, let’s say you’re designing some sort of inbox management tool:

A good conversation, anchoring generic fluff:

In this case, we took the generic claim, “My inbox is always under control” and added the important caveat: “Except when it’s not, in which case it’s a total nightmare to recover from.” While using generics, people describe themselves as who they want to be, not who they actually are. You need to get specific to bring out the edge cases.

Let’s say you’re building a mobile loyalty app to help stores give deals and discounts to their most loyal customers and you hear the guy in line in front of you complaining:

A bad conversation (pitching and accepting fluff):

By switching into pitch mode, we just wasted a perfectly good opportunity for learning and instead got a fistful of fluff. Let’s try again.

A good conversation:

You can’t help but laugh when you hear this one. “Someone should definitely make an X!” “Have you looked for an X?” “No, why?” “There are like 10 different kinds of X.” “Well I didn't really need it anyway.”

Long story short, that person is a complainer, not a customer. They’re stuck in the la-la-land of imagining they’re the sort of person who finds clever ways to solve the petty annoyances of their day.

Beyond rousting some poor soul’s consumeristic hypocrisy, anchoring the fluff can also yield useful signals:

So he’s an actively searching potential user, but we’d need to get critical mass with the stores he goes to before he'll be happy. Maybe we could take over a small university town first. Or he might say:

So he was on the fence, but needs better perks. Maybe we could find a way to force merchants into deeper discounts like Groupon was able to do. He also has privacy concerns. Or he could respond with:

So all we need to do (for this particular user) is to out-execute and simplify. We could try to be the Instagram to their flickr.

The list goes on. There are tons of useful responses you can get. Even learning that the person is a non-customer is useful. To get toward this truth, you just need to reject their generic claims, incidental complaints, and fluffy promises. Instead, anchor them toward the life they already lead and the actions they’re already taking.

Dig beneath ideas

Entrepreneurs are always drowning in ideas. We have too many ideas, not too few. Still, folks adore giving us more. At some point during a good conversation, the person you’re talking to may “flip” to your side of the table. This is good news. They are excited and see the potential, so they’ll start listing tons of ideas, possibilities and feature requests.

Write them down, but don’t rush to add them to your todo list. Startups are about focusing and executing on a single, scalable idea rather than jumping on every good one which crosses your desk. 

Let’s say you’re mid-conversation when this idea drops:

What do you do here? The wrong response is to write “sync to Excel” on your todo list and then move on. That’s the fast-lane to feature-creep. Instead, take a moment to dig into the motivations behind the request.

Or they might say:

Or:

At my first company Habit, we were adapting our product to sell to enterprise companies. MTV told me they needed analytics and reports for their campaigns. 

I made a big mistake by accepting the feature request and face value and beginning the next meeting with a demo of our shiny new analytics dashboard (custom-built to solve their request, of course). They “ooh’ed” and “ahh’ed” appropriately and I left thinking we’d nailed it. It offered a zillion options and could carve up your data every which way. It was technically and aesthetically lovely. 

Unfortunately, 90% of what we had built was irrelevant. We just didn’t know that yet.

They started calling every Friday asking me to email over a CSV (data file) of the week’s stats, so we added CSV export to the dashboard. Later, they asked for the report as a PDF instead of an CSV, so we obediently built PDF export. That took longer. 

Salt was rubbed in the wound when, weeks later, they were still calling me every Friday and asking me to export and send over the same stupid analytics report. And every week, I would do so while politely explaining that, you know, we built this awesome self-serve dashboard so they could have their data whenever they wanted. And then, the next Friday, they’d call me.

It turned out we had entirely missed the real reason they’d been excited about our analytics demo. In fact, we'd missed their whole motivation for wanting analytics.

The memory of being burned by feature requests was still fresh in my mind when they asked if we could add their logo and colours to the reports. I asked a couple incredulous questions about why in the world they wanted this feature when they didn’t even use the ones I had already built, like an exasperated Dad at Christmas: “But you don’t even play with the toys I bought for your birthday!”

So, I finally—and inadvertently—did the smart thing when I asked, “Why do you want this feature? What do branded reports get you that unbranded ones don’t? It’s the same data, right?” She replied, “Oh yeah, of course. I mean, nobody even reads these. Our clients just like to get something emailed to them at the end of every week and we think they’d be happier if it was a bit fancier, you know?” I knew exactly. 

They had asked for analytics. We had jumped to the conclusion that they wanted to better understand their data. But they had really wanted a way to keep their own clients happy. If we had properly understood that, we would have built a totally different (and much simpler) set of features. 

Consider how much easier our lives would have been if we’d understood the motivation behind the request. Instead of enabling the exploration and export of all campaign data, we could have just always exported the few high-level numbers a big brand manager would be interested in. And instead of a self-serve dashboard, we could set up a little scheduler to send it to them every Friday. In fact, we didn’t even need to build a dashboard at all. And instead of coding up a layout and branding system for the reports, we could have had an intern hand-build them each week. All wasted because I didn’t ask the right question. I sure wish I had those 3 months back!

When you hear a request, it’s your job to understand the motivations which led to it. You do that by digging around the question to find the root cause. Why do they bother doing it this way? Why do they want the feature? How are they currently coping without the feature? Dig.

You should dig in the same way around emotional signals to understand where they’re coming from. Just like feature requests, any strong emotion is worth exploring. Is someone angry? Dig. Embarrassed? Dig. Overjoyed? Dig!

I once overheard a founder interviewing someone at a cafe table next to me. The founder mentioned a problem and the guy responded, “Yeah, that’s pretty much the worst part of my day.” The founder jotted something down in his notebook, and then moved on to the next question. What!? It’s the worst part of his day and you’re not going to figure out why? That’s insane. You’ve got to dig.

Questions to dig into feature requests:

• “Why do you want that?”
• “What would that let you do?”
• “How are you coping without it?”
• “Do you think we should push back the launch add that feature, or is it something we could add later?”
• How would that fit into your day?

Questions to dig into emotional signals:

• “Tell me more about that.”
• “That seems to really bug you — I bet there’s a story here.”
• “What makes it so awful?”
• “Why haven’t you been able to fix this already?”
• “You seem pretty excited about that — it’s a big deal?”
• “Why so happy?”
• “Go on.”

These nudges don’t need to be complicated. People love talking about their opinions and emotions. Digging into a signal is basically just giving them permission to do a brain dump. 

Stop seeking approval

As we’ve seen, compliments are dangerous and sneaky. So if we can nip them in the bud before they bloom, so much the better. The main source of compliment-creation is seeking approval, either intentionally or inadvertently. 

Doing it intentionally is fishing for compliments. You aren’t looking for contradictory information. You’ve already made up your mind, but need someone’s blessing take the leap.

Symptoms of Fishing For Compliments:

• “I’m thinking of starting a business... so, do you think it will work?”
• “I had an awesome idea for an app — do you like it?”

Accidental approval-seeking is what I call “The Pathos Problem.” It happens when you expose your ego, leading people to feel they ought to protect you by saying nice things.

This comes up when you tell someone about an idea you obviously care about (which is pretty much always, since otherwise you wouldn’t be asking). Even if you give folks permission to be honest and ask for criticism, they’re still going to pull their punches.

Symptoms of The Pathos Problem:

• “So here’s that top-secret project I quit my job for... what do you think?”
• “I can take it — be honest and tell me what you really think!”

To deal with The Pathos Problem, keep the conversation focused on the other person and ask about specific, concrete cases and examples. Once someone detects that your ego is on the line, they’ll give you fluffy mis-truths and extra compliments. Disregard that data and use The Mom Test to re-focus on the person, their life, and their goals. People rarely lie about specific stuff that’s already happened, regardless of your ego. 

Some famous entrepreneurs don’t suffer the effects of The Pathos Problem, but you should ignore their advice since it’s not reproducible if you aren’t them. Guys like Elon Musk, Reid Hoffman, and Gordon Ramsey are all notorious for actively seeking negative feedback. It evidently works for them. But nobody is worried about hurting Elon, Reid, or Gordon’s feelings. You and I must be more circumspect.

In short, remember that compliments are worthless and people’s approval doesn’t make your business better. Keep your idea and your ego out of the conversation until you’re ready to ask for commitments. 

Cut off pitches

Being pitchy is the dark side of the “seeking approval” coin. Instead of inviting compliments by being vulnerable, you’re demanding them by being annoying. It’s when you hold someone hostage and won’t let them leave until they’ve said they like your idea. Normally, compliments are well-intentioned. In this case, they’re just trying to get you out of their office.

“Won’t-take-no-for-an-answer” is generally a good quality for a founder to have. But when it creeps into a conversation that’s meant to be about learning, it works against you.

Symptoms:

• “No no, I don’t think you get it...”
• “Yes, but it also does this!”

If you slip into pitch mode, just apologise. You’re excited about your idea. That’s good! Otherwise you wouldn’t have taken this crazy leap in the first place. But suddenly, you find yourself five minutes into an enthusiastic monologue while the other person nods politely. That’s bad. Once you start talking about your idea, they stop talking about their problems. Cut yourself off and say something like:

“Whoops—I just slipped into pitch mode. I’m really sorry about that—I get excited about these things. Can we jump back to what you were just saying? You were telling me that…"

If they say they really want to hear about what you’re working on, promise that you’ll tell them at the end of the meeting or loop them in for an early demo, and that you just want to talk a bit more about their stuff before biasing them with your idea.

Talk less

You can’t learn anything useful unless you’re willing to spend a few minutes shutting up (even if you have something really smart to say).

After you introduce your idea (either intentionally or accidentally), they’re going to begin a sentence with something like “So it’s similar to…” or “I like it but…” It’s tempting to interrupt and “fix” their understanding about how it’s totally different and actually does do that thing they want.

Alternately, they’ll raise a topic you have a really good answer to. For example, they’ll mention how important security is, and you’ll want to cut in and tell them you’ve thought about all that already. This is also a mistake.

In both cases, the listener was about to give you a privileged glimpse into their mental model of the world. Losing that learning is a shame. You’ll have the chance to fill them in later. Plus, it’s annoying to people if they start trying to help you and you cut them off to correct them.