how it happened
- whoten years of résumésmostly from men
- the gapAI ranking modellearned maleness as a signal
- the resulttool scrappeddowngraded "women's" and two colleges
what everyone assumed
A team set out to take the guesswork out of screening, which is the most reasonable goal in hiring. A model trained on the company’s own past hires would surely be more consistent than a tired recruiter on a Friday, and consistency reads as fairness. The belief underneath was simple and widely shared: a machine has no opinion about who you are, so its ranking must be neutral.
what happened
The work began in 2014. The system was designed to rate applicants from one to five stars, learning what a strong candidate looked like from roughly ten years of résumés the company had already received. Because the field those résumés came from skewed heavily male, the pattern the model learned was that maleness correlated with success. For a stretch the tool ran and ranked and looked like a measurement, while what it had actually absorbed was the shape of the past.
step by step
- 2014a tool to rate applicants one to five starsIt learns from about ten years of résumés the company already received.
- the training data skews heavily maleThe model learns that maleness correlates with success.the assumption breaks here
- around 2015it penalises the word women'sGraduates of two all-women colleges are downgraded too. Engineers neutralise the terms they can name.
- October 2018the project is reported scrappedThe company could not be sure the model had not found quieter proxies.
how it surfaced
By around 2015 the team found that the model was penalising résumés that contained the word “women’s,” as in “women’s chess club captain,” and downgrading graduates of two all-women colleges. Engineers edited the system to treat those particular terms as neutral. That fix held for the terms they could name, and it could not promise the model had not found other, quieter proxies for the same thing. The company lost confidence that the tool was gender-neutral, stopped using it for hiring decisions, and Reuters reported the project scrapped in October 2018.
A model trained on who got hired before learns to repeat who got hired before.
Objectivity was never a property of the tool; it was a property someone assumed, because the output arrived as a number.
picture it at a company like yours
The company here is made up. The other two paths are not: they happened, and each links to the story.
- in this storythe day it mattersA strong candidate is ranked low for reasons nobody on your team can see or explain.what it costs youYou find the bias only after it has shaped a year of hiring decisions made in your name.
- or it goes like thisit happenediTutorGroup, 2023howA screening rule quietly rejects applicants by age.what leavesa discrimination settlementwho ends up with ita regulator acting for the applicants
- or it goes like thisit happenedMobley v. Workday, 2025howRejections add up across thousands of applicants nobody compares.what leavesa collective lawsuitwho ends up with iteveryone screened out
sources
your turn
what you can do about it
this week
For one scoring or ranking tool you rely on, find out what data trained it and who has ever checked its outputs against the groups it scores.
Somewhere in your company there may be a tool that ranks people, applicants, tickets, accounts, risk, and its output arrives as a clean score that nobody argues with because a number does not look like an opinion. The question the quiet months ask is what the score learned from. The Amazon pattern was findable by reading the rankings themselves.
go deeper
if you're not sure where to start
Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.
open the checklist →