Measure Impact on Your Desired Outcome

In addition to instrumenting what you need to evaluate your assumption tests, you also want to measure what you need to evaluate your progress toward your desired outcome. Our outcome at AfterCollege was to increase the number of students who found jobs on our platform. For our assumption tests, we were measuring search starts, job views, and job applications, but these metrics were only leading indicators of our desired outcome.

Over time, we also wanted to move closer to measuring our outcome itself, so that we could track progress week over week and quarter over quarter. When I started at AfterCollege, we didn’t have a way of measuring how often a student got a job. We lost track of students after they applied for a job. The post-apply steps like interviewing, receiving an offer, and accepting an offer all happened off of our platform. We needed to find a way to incentivize students to tell us when they got a job or employers to tell us when they made a hire.

Some people in the company argued that we should measure our success by job applications. After all, we had no control over who a company hired or how a student interviewed. But the number of job applications was an easy metric to game. It would be easy to encourage students to apply to many jobs, but this wouldn’t necessarily increase their success of finding a job. If we wanted to measure the value we created for our customers, we knew we needed to measure when a student got a job. We couldn’t be afraid to measure hard things.

Since most college students have little to no interviewing experience, nor do they know how to negotiate offers, we decided that we could use this lack of knowledge to help us measure what happens after they completed an application. Twenty-one days after a student applied for a job, we sent the student an email and asked them what happened. The email gave them four options:

1. “I never heard back.” If they selected this option, we encouraged them to find new jobs to apply to.
2. “I got an interview.” If they selected this option, we gave them tips for how to prepare for their interview.
3. “I got an offer.” If they selected this option, we gave them tips on how to evaluate and negotiate their offer.
4. “I got the job.” If they selected this option, we congratulated them.

Not everyone replied to our email. In fact, when we first launched it, only 5% of job applications (not applicants) netted a reply to the email. But over time, we grew that to 14%, and, by the time I left, we were at a 37% response rate. That’s not perfect, but it gave us some visibility into what was happening after an application. I know that if we had kept iterating on that email, the response rate would have continued to improve. We probably would have experimented with other ways of collecting the same data. We knew that, if we were relentless, we would find a way to track our desired outcome.

Here’s the key lesson. Just because the hire wasn’t happening on our platform didn’t mean it wasn’t valuable for us to measure it. We knew it was what would create value for our students, our employees, and ultimately our own business. So, we chipped away at it. We weren’t afraid to measure hard things—and you shouldn’t be, either.