On September 26th, the U.S. Department of Labor filed a suit against Palantir Technologies, alleging that Palantir’s engineering hiring practices discriminate against Asian applicants. I don’t have any salacious insider information about this suit, but I do have quite a bit of insight into how technical hiring works. Palantir and the DOL are really arguing over using resumes versus employee referrals to screen job candidates, when smart companies of a certain size should primarily rely on neither. In other words, rather than Palantir, standard hiring practices are really what should be on trial.
The DOL’s suit is based on the unfavorable disparity between the number of “qualified” Asian applicants for 3 engineering roles between 2010 and 2011 and the number of resulting Asian hires. Below (as taken from the full complaint), you can see the roles covered by the suit, the number of applicants and hires, and the odds, according to the DOL’s calculations, that these disparities happened by chance:
My issue with this setup is simple: what does qualified actually mean? According to the complaint, “Palantir used a four-phase hiring process in which Asian applicants were routinely eliminated during the resume screen and telephone interview phases despite being as qualified as white applicants.” A four-phase hiring process is typical in tech companies and refers to a resume screen followed by a call with a recruiter, followed by a technical phone screen where the applicant writes code while another engineer observes, and concluded by a multi-hour onsite interview.
To determine basic “qualification,” the DOL relied, at least in part (and likely heavily), on the content of applicants’ resumes which, in turn, boils down to a mix of degree and work experience. Resumes are terrible predictors of engineering ability. I’ve looked at tens of thousands of resumes, and in software engineering roles, there is often very little correspondence between how someone looks on paper and whether they can actually do the job.
How did I arrive at this conclusion? I used to run technical hiring at a startup and was having a hell of a time trying to figure out which candidates to let through the resume screen. Over and over, people who looked good on paper (had worked at companies like Google, had gone to schools like MIT, and so on) crashed and burned during technical interviews, whereas candidates without pedigree often knocked it out of the park. So, I decided to examine the resumes of everyone who applied over the course of a year as well as those of past and current employees. After looking at hundreds of resumes and looking at everything from years of experience and highest degree earned to G.P.A and prestige of previous employers, it turned out that the thing that mattered most, by a huge margin, wasn’t any piece of information about the candidate. Rather it was the number of grammatical errors and typos on their resume.
Don’t believe me that screening for education and experience doesn’t work? Then consider the following experiment. A few years ago, I showed a set of anonymized resumes from my collection to 150 engineers, recruiters, and hiring managers and asked them one question: “Would you interview this candidate?” Not only did participants, across the board, fail at predicting who the strong candidates were (the odds of guessing correctly were roughly 50%, i.e. the same as flipping a coin), but, much more importantly, no one could even agree on what a strong candidate looked like in the first place.
Organizations realize that resumes are noisy and are forced to explore other, more reliable channels. In the case of Palantir and many other companies, this boils down to relying on employee referrals, and that may be the DOL’s strongest argument. According to the complaint, “…the majority of Palantir’s hires into [the three positions listed in the suit] came from an employee referral system that disproportionately excluded Asians.” Using referrals as a hiring channel is an extremely common practice, with the rationale being that it’s a reliable source of high-quality candidates. This makes sense. If resumes were reliable, then referrals wouldn’t be such a valued channel.
Despite its ubiquity, is relying on referrals grounds for a discrimination suit? Perhaps. But referrals were Palantir’s perfectly reasonable attempt to find a better screen than resumes. Palantir just didn’t go far enough in looking at other options. What if, instead of being bound to hire from the quasi-incestuous sliver of your employees’ social graph, you could have reliable, high-signal data about your entire candidate pool?
Until recently, when hiring, you had to rely on proxies like resumes to make value judgments because there simply wasn’t a good way to get at more direct, meaningful, and unbiased data about your candidates.
We now have the technology to change that. A slew of products anonymize candidate names, entirely occluding race. A whole other set of tools enables you to send relevant take-home exercises to all your applicants and automatically score their submissions, using those scores as a more indicative resume substitute. And there’s my company, interviewing.io, which I’m totally plugging right now but which also happens to be the culmination of my attempts to fix everything that’s pissed me off about hiring for years. interviewing.io matches companies with candidates based entirely on how those candidates have been doing in technical interviews up until that point. Moreover, every interview on our platform is blind — by the time you unmask with a candidate, you’ve decided whether you’re going to bring them in for an onsite, and you’ve used nothing but their interview performance to make that decision.
Whichever solution ends up being the right one, one thing is clear. It’s time to shut down outdated, proxy-based hiring practices. That doesn’t mean paying lip service to diversity initiatives. It means fundamentally rethinking how we hire, paring away every factor except whether the candidate in question can do the job well.
Any other kind of hiring practice is potentially discriminatory. But even worse, it’s inefficient and wasteful. And it’s ultimately the thing that, unlike Palantir, truly deserves our wrath.
“Don’t believe me that screening for education and experience doesn’t work? Then consider the following experiment. A few years ago, I showed a set of anonymized resumes from my collection to 150 engineers, recruiters, and hiring managers and asked them one question: “Would you interview this candidate?” Not only did participants, across the board, fail at predicting who the strong candidates were (…).”
How did you measure true “strength”? From your other entry: ” I had visibility into how they performed on the job afterwards. The remainder of resumes came from engineers I had worked with directly.” The kind of performance that was visible to you was not visible to those who wanted to hire? How come?
This was an enlightening read for me. Thank you for sharing. I also enjoyed going back and reading about your research regarding errors on resumes and their correlation to good hires. With a degree in English, I love finding tidbits like this to support my love of the major and how it can help all sorts of people be more professional and better communicators.