Chapter Eleven: Measuring Impact

I was excited to join AfterCollege as their Vice President of Product and Design. AfterCollege helps new college graduates find their first job out of school. I had experience in the recruiting industry, but little experience with this particular customer segment, so I kicked off my continuous-interviewing habit during the first week on the job.

Right away, I learned something surprising. I asked college seniors to tell me about their experience looking for a job. Most expressed the same sentiment over and over again. The vast majority of job boards (including ours) asked students two questions: 1) What type of job do you want? 2) In what location? Unfortunately, most of the students that I talked to didn’t know how to answer either of these questions. They had no idea what types of jobs they were qualified for, and they were open to living in many places. The average 22-year-old doesn’t have enough work experience to even be aware of what types of jobs exist, and they have the location flexibility to go where the best opportunities are. Some had a preference to return home or to stay near their college town, but about half were willing to go anywhere. It didn’t take long for me to realize this was a huge problem. We had to stop asking college students questions they couldn’t answer.

As a product team, we realized that we had proprietary data that could help us solve this problem. We had years of behavioral data about which types of jobs students applied to, and, more importantly, we knew from working with employers what types of students they wanted for different types of jobs. In other words, we were in a great position to tell students what types of jobs they were qualified for if they told us what they studied.

Instead of asking, “What type of job do you want?” and “Where do you want to work?” we realized we could ask students, “Where do you go to school?” “What are you studying?” and “When do you plan to graduate?” We suspected these questions would be much easier for students to answer, and we thought we could use their answers to recommend jobs to them.

Our long-term vision was to develop a machine-learning algorithm that matched students to the best jobs based on their own preferences and what we knew employers wanted. But before we invested in a machine-learning algorithm (we didn’t know anything about machine learning yet), we needed to learn if this idea was worth investing in. It was so different from what everyone else was doing, we needed to make sure it would work.

To build a quick prototype, we realized that, as working professionals, we had more experience with job types than most college seniors and that we could create a crude approximation of our matching algorithm by creating saved searches for each of the areas of study in our system. Instead of having a sophisticated machine-learning algorithm behind the scenes, we simply crafted search queries ourselves. For example, if a student indicated they were an English major, we might search for marketing, content management, journalism, speech writing, and public-relations jobs. It wouldn’t be perfect, but we thought it would be better than what students were entering themselves.

We also knew it would help us quickly evaluate how successful our new idea might be. We had dozens of assumptions we wanted to test. Would college students trust our recommendations? Would they be open to exploring jobs they themselves didn’t select? Would they be confused by our unique interface? After all, every other job board asked them to enter what type of job they wanted. Could we collect enough feedback to continue to refine our algorithm? Would our metrics show that this solution was better than what we currently had?

We knew we wanted to start small, as this idea was full of risk. So, we started by diverting a small percentage of our traffic to a new search page. Students entered their area of study, and we ran the relevant “saved search” behind the scenes. We were able to get this working prototype live in just a few days. We then watched what happened.

In our traditional “What type of job do you want?” interface, only 36% of students started a search. Two thirds of our site visitors never even started their job search. With our new “Tell us what you studied” interface, 83% of our visitors started their search. This was a huge improvement. Our new questions were much easier to answer, so more students were able to start their search.

But what happened after that? We found that students who entered job types and locations were more likely to view and apply for jobs, but not by much (only about 10%). We suspected it was because these students already had an inkling of what they wanted to do and where they wanted to work. But for everyone else, that interface simply didn’t work. In our new interface, we saw a small drop-off in the number of searchers who viewed and applied for jobs, but we saw many more students start their search, so our overall performance was much better in the new interface. We knew right away it was safe to keep investing in this idea.

But where should we go from here? We didn’t have a production-quality product. Remember, we tested with a crude prototype that we cobbled together in just a few days. We still had many more assumptions to test. We didn’t feel like we were done with discovery. But we also were seeing better results with our prototype than we were with our production-quality product.

We debated about whether we should switch all of our traffic to the new prototype. We were seeing great results from our early assumptions tests, but we still had one key question to answer: Did our new idea drive our desired outcome? Our desired outcome wasn’t to increase search starts, nor was it to increase job views or job applications. It was to increase the number of students getting jobs through our platform. We thought if we could get more students starting their search, we would increase the number of students who found jobs on our platform. But job views and applications aren’t always leading indicators of hires. A student can view and apply to many jobs and never hear back from an employer. We needed to make sure that students were as likely (and hopefully, more likely) to find a job with our new interface as they were with our old interface. We decided we needed to continue to split our traffic until we could confirm that our new interface supported our desired outcome.

This story illustrates a few key lessons. First, it’s easy to get caught up in successful assumption tests. The world is full of good ideas that will succeed on some level. However, an outcome-focused product trio needs to stay focused on the end goal—driving the desired outcome. We need to remember to measure not just what we need to evaluate our assumption tests, but also what we need to measure impact on our outcome.

Second, this story also highlights the iterative nature of discovery and delivery. Many teams ask, “When are we done with discovery? When do we get to send our ideas to delivery?” The answer to the first question is simple. You are never done with discovery. Remember, this book is about continuous discovery. There is always more to learn and to discover. The second question is harder to answer. In the AfterCollege story, we had already started the delivery work. Our prototype had a working interface that real customers could use. We were collecting real data. Our discovery required that we start delivery. Measuring the impact of that delivery resulted in us needing to do more discovery.

This is why we say discovery feeds delivery and delivery feeds discovery. They aren’t two distinct phases. You can’t have one without the other. In Chapter 10, you learned to iteratively invest in experiments, to start small, and to grow your investment over time. Inevitably, as your experiments grow, you are going to need to test with a real audience, in a real context, with real data. Testing in your production environment is a natural progression for your discovery work. It’s also where your delivery work begins. If you instrument your delivery work, discovery will not only feed delivery, but delivery will feed discovery.

In this chapter, you’ll learn how to instrument your product so that you can evaluate assumption tests using live prototypes. You’ll learn how to measure the impact of your delivery work, using your desired outcome as your North Star. And you’ll learn how to keep your discovery and delivery tightly coupled so that you never have to wonder if you are ready for delivery.