What I learned from reading 8,000 recruiting messages

If you’ve ever been involved in hiring software engineers, you know how frustrating the process can be. One of the toughest parts is getting candidates’ attention. In an effort to understand what makes engineers respond to some recruiting messages and not others, I teamed up with the fine folks at Hired.

Hired is a 2-year-old marketplace where engineers create profiles, and companies bid on those engineers, auction-style. A personal message accompanies every bid a company makes on an engineer, providing a rich data set comprised of thousands of candidates, hundreds of companies, and thousands of messages between them.

We analyzed a sample of about 8,000 recruiting messages, examining a number of factors including company prestige and size, engineers’ desired salaries, degree of message personalization, and whether messages came from recruiters or from engineers/founders themselves. Our goal was to determine which factors would be predictive of an engineer deciding to engage with a company (referred to in this post from now on as the introduction rate).

In the process, we discovered that:

  • despite what people claim, money is hugely important in determining which opportunities to pursue
  • message personalization is very important, but not all personalization is created equal
  • left to their own devices, engineers and founders sound just as spammy as recruiters in their attempts to woo new talent

Here’s a graph of all the factors we looked at and their relative importance in predicting introduction rates. These values come from running a logistic regression on the factors that were most statistically significant. The y-axis represents the standardized value of the coefficients in our regression. It’s on a scale of -1 to 1 and represents how heavily each of these factors weighed into predicting whether a candidate was interested in an opportunity. Bars that are below the x axis mean that that factor has a negative effect. Note that all the graphs in this post are interactive, so you can hover and do other fun things.

These findings require a bit of explanation, so below, I’ve drilled down into a few of the more interesting takeaways.

Money

Although Hired’s data provides a pretty good approximation for email recruiting, recruiting messages in the wild don’t generally come with a numerical offer value. I was curious about how money might change the game, so in addition to examining what aspects of the messaging itself are sticky, we also looked at the effects of setting expectations around compensation up front.

Of all the different factors that went into whether a candidate was interested in a job, money was by far the most significant. This particular metric refers to offer (bid) amount divided by a given engineer’s self-reported preferred salary. In other words, the more an offer exceeds some target the engineer has set, the more likely it is that he will respond affirmatively.

How much more likely? Assuming a preferred salary of $120,000, there’s a 45% introduction rate for interview requests right at that amount. Dropping the offer down to $110,000 results in a 34% intro rate. Upping it to $130,000 shoots the intro rate up to 54%. In other words, for an engineer with a $120,000 preferred salary, paying $10k more leads to a 20% higher chance of introduction, whereas paying $10k less leads to a 25% lower chance. To put it another way, even dropping an offer by $10,000 can be significantly detrimental. It’s also interesting to note the inflection point that occurs in the introduction rate is right where the offer amount is equal to an engineer’s preferred salary.

Despite the strong evidence that money talks when an engineer decides whether or not to engage with a company, we did see an interesting disparity in what engineers reported as important to them versus how they actually behaved. As you can see below, lack of interest in the company’s value proposition was twice as common as any other reason for not engaging with a company, including insufficient compensation. Of course, stronger compensation may make a candidate reconsider a given company’s value, and a company with greater value may be less likely to offer under market, so the effects aren’t independent.

Money is a convenient attribute to cite as important because it’s so easy to measure, but I’d be very surprised if it’s telling as much of the story as it appears to. In other words, before racing to the conclusion that engineers are driven primarily by greed, I would like to posit another idea. Figuring out what people want in a job is hard, given just someone’s work history and self-summary. Even in a world where a recruiter has perfect information about a candidate’s professional desires, framing a prospective employer’s offering in a succinct way that resonates with a candidate is still really difficult. The fact that so few messages on the platform were truly personal is indicative of that (more on that below). Money is an objective metric that encapsulates not only the financial strength of the business behind the offer, but also how much they need your skills. I believe that if there were a way to quantify “how interesting the projects you’ll be working on are to you”, “how great the people you’re working with will be”, the impact someone can make, and so on, those numbers would easily be as significant as cash.

Personalization

I’m a recruiter, and the nature of the beast is that I somewhat regularly send out cold emails. I used to be an engineer, however, so I viscerally despise recruiter spam. To prevent cognitive dissonance from melting my brain, I wanted to understand how much personalization is enough. To test how much personalization mattered when it came to introduction rates, I broke up the messages into 3 categories.

1. Not personal at all.
This message could have gone to anyone on the platform who met the criteria for a given position and still would have made sense. Example:

Impersonal message

2. Somewhat personal.

This message mentions something about you that’s easily identifiable and maybe ties it back to the company. Example:

somewhat_personal_1

and

somewhat_personal_2

3. Totally personal.

This message was clearly meant for you, you unique and beautiful snowflake. It might talk about your past work in depth or mention some projects that you would be interested in for very specific reasons or appeal to your specific sensibilities when talking about the company vision. We didn’t redact this particular sender’s info because this message is a shining beacon of all that is good and right in cold emails. Example:

personal

Despite the title of this post, reading almost 8,000 messages and scoring them for how personal they were was clearly intractable. Instead I wrote a little script that hashed messages with an edit distance of less than 60 characters (effectively covering recipient and sender first & last names) to the same bucket and surfaced messages that varied by more than this threshold amount. Those messages I scored manually.

First, here’s the distribution of personalization across all messages (n=7818):

As you can see, the vast majority of messages were effectively form letters, and it was especially surprising how few genuinely personal messages there were (a whopping 60 out of 7818 or ~0.8%).

So how much did personalization matter? For context, the average introduction rate on the entire platform was about 49.6%. Here’s the breakdown of introduction rate by personalization level:

What really struck me here was that adding a little bit of personalization didn’t appear to matter at all; in fact, the introduction rates for both impersonal and somewhat personal messages were virtually identical. In other words, the time it takes to drop in the moral equivalent of “Ohai I see you went to Local Good College, well so did our founders, go Local College Sports Team!” or just casually mentioning a candidate’s past employer or projects should probably be spent on something else. On the other hand, truly personal messages had a 73% introduction rate on the platform.

Unfortunately, all told, there were only 60 truly personal messages, probably because taking the time to write something unique is really hard. If you’re not an engineer yourself, knowing what to say can also be really difficult. To that end, I broke up each personalization tier by whether it came from a recruiter or from an engineer/founder. Perhaps not entirely surprising is that more impersonal messages came from recruiters than from engineers and founders (per capita).

What was surprising is the sheer volume of form-letter like messages that DID come from engineers and founders. In fact, over 85% of the messages that engineers and founders sent out were form letters; this is especially surprising given that those groups are the ones who are often on the receiving end of this kind of barrage.

Based on these findings, it seems like if you know enough about the subject matter, putting in a bit of effort and making messages truly personal is probably worth it. This is quite hard to do in the real world, where info about a candidate can be scattered all over the internets, if it’s even there at all. On a platform like Hired, however, where the candidate goes out of their way to write up a self-summary, talk about their interests, provide a resume, and more often than not, a GitHub profile, and where most of the people doing the hiring are also the ones working on the product and are in a position to talk about projects in a meaningful way, going personal is very likely worth the mental effort.

Who the message comes from

Personalization notwithstanding, I was also curious about whether who the message came from (recruiter, engineer, or founder) mattered when it came to introduction rate. To control for personalization, I focused on just the impersonal messages (n=6827). The breakdown of introduction rates was surprising:

Recruiter: 53%
Founder: 49%
Engineer: 47%

In other words, recruiter introduction rates were statistically significantly different from the others, and recruiter messages did better than messages from both engineers and founders. I was quite surprised by these results, so I did some sleuthing. First, let’s talk a bit about in what situations recruiters are likely to be the ones sending messages.

As a company grows, founders are less likely to be the ones doing recruiting. Below, you can see what portion of messages came from recruiters as a function of funding stage:

Saying that recruiters get more or less introductions overall isn’t truly meaningful because perhaps major corporations might have really strong brands, happen to have more recruiters doing messaging, and can potentially pay higher salaries. When I controlled for these things, the effect largely went away, and introduction rates were pretty much the same across the board, except for some outlier recruiter messaging. Here’s a good example.

recruiter_message

What made messages like this particularly compelling was beyond me, but then I looked at non-recruiter messages in the same batch.

nonrecruiter_message

So what’s the difference? I think the secret sauce is in the tone. The recruiter message sounds genuinely excited and what it lacks in technical depth, it makes up for in enthusiasm. In other words, this:

recruiter doge.jpg

The non-recruiter message, on the other hand, is a bit more dry, canned, and tentative. It also doesn’t do a good job of getting across the scope of the opportunity. Perhaps the takeaway here is that, if you can’t go personal, you should at least go enthusiastic. The last thing I want to encourage here is a barrage of doge recruiting, but if push comes to shove, it may be in your best interest to hire good recruiters who know how to hit the right notes. While this kind of recruiting isn’t a substitute for genuinely engaging with your audience, having experience sending out tons of messages is going to make you better at it than someone who is uncomfortable with reaching out to strangers and hasn’t done enough of it to overcome their gag reflex.

Conclusion

So, at the end of the day, what is it that makes recruiting messages more sticky? What drives engineers to be interested in certain opportunities and not others? What can you do now to get better at attracting engineers to your company?

  • If you can, go deeply personal with your outbound recruiting messages. Talk about what you’re working on, how that ties into what the candidate has already done, and why what you’re doing matters to them. Name dropping and shallow, faux personalization attempts simply don’t cut it.
  • Reachouts from founders aren’t intrinsically more valuable, unless they’re personal and targeted.
  • Writing really good recruiting messages is hard, and when engineers and founders put on their recruiting hats, they don’t necessarily fare better than recruiters themselves. Good recruiters, on the other hand, are worth a lot. People who do it professionally are going to be able to craft more engaging messaging than those who don’t.
  • Trying to underpay good people isn’t going to make you any friends. And, being transparent about salaries out of the gate, assuming those salaries are competitive, may be a good strategy for standing out.

Lastly, I’d like to throw out some acknowledgements. As always, Statwing made the statistical analysis required for this post a delight. If you liked the pretty, interactive graphs, check out Plotly.  A huge thank you to Elliot Kulakow of Hired for all his help with the SCIENCE and the pretties. And finally, thanks to everyone who proofread this behemoth.

Looking for a job yourself? Work with a recruiter who’s a former engineer and can actually understand what you’re looking for. Drop me a line at aline@alinelerner.com.

15 Responses to “What I learned from reading 8,000 recruiting messages”

  1. Alex

    I never considered about doing something like this, what algorithm did you use to sort if messages are personal or not ? Simple pattern recognition or something more complicated ?
    I’m looking to do a similar experiment with some messages and I am looking for an algorithm good enough to sort spam/impersonal messages from my list.

    Reply
    • aline

      I just looked at Levenshtein distance between messages from the same company/recruiter. If the difference was over a certain threshold, I’d review that message manually.

      Reply
  2. Tikhon Jelvis

    I think the fact that founders and engineers send out a lot of form letters is not so surprising. If I was in that position, I’d likely do the same thing—after all, that’s what I see 90% of the time myself, so it feels like the default approach. If everyone else is doing it, it must make sense!

    Honestly, I’m not sure if I would even have realized that trying to personalize every email was an option at all. It just wouldn’t have come to mind. I’d still send some personal emails, but those would be things separate from the default recruiting process—people I already met in person or online, mostly.

    Anyhow, that would my thought process. (After reading this, of course, I’d approach it differently, but I don’t think it’s going to come up in the near future, unfortunately.)

    Reply
    • aline

      An open question for me is where the lines cross, i.e. how much personalization is just enough? Writing personal messages is super time-consuming, so finding right where that intersection is would be awesome (and it’s something I’m working on).

      Reply
      • ben schnell

        My conclusion from this (amazing) post was that a mix of impersonal and super personal is the best bet. Write super personal emails to people who look like a super perfect fit, and write impersonal emails to a large group of people who don’t put enough info about themselves on the internet to know if they would be a perfect fit or not, but have the necessary job history. Nobody loves sending spam, but resistance-to-sending-spam has to be balanced against the numbers-game nature of our work, in my opinion.

        Reply
  3. Michael Varley

    This is a really great post Aline thank you so much for the insights. Most suprising was the little difference in response rates from Founder to Recruiter. I think the message is clear dont be a lazy spammer know your audience, be personal but not creepy and transparent as possible on salary. This may take a little more work but seems the outcomes for the company will be greater. Always great to have the data to back these points up however.

    Reply
  4. Mike Pitta

    Great post Aline! This data is incredibly insightful. I was about to try and drag some of our engineering leaders into the recruiting process; it seems like it may not be as necessary as originally thought.

    Reply
  5. Chuck Dailey

    Aline, I am amazed at the time you must have spent working on such an arduous task. Your analysis has found some incredible use (IMO) for future recruiting techniques. I am just beginning my own recruiting firm and have been connecting with potential clients using the same tone and personalization that you have found to be most effective, mostly because it is how I have always communicated with people anyhow (INFP). Thank you for sharing!

    Reply

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>