Daily Learning Experience · AI & Machine Learning
You are a number between 1 and 10, printed at 4 a.m. in a county jail in Florida. Two teams of very careful people are about to spend two years arguing about whether you are fair — and the astonishing answer is that they will both be correct.
Broward County, Florida · 2013–2016 · and a theorem published on 19 September 2016
Act One · The form
It is four in the morning and somebody is reading questions off a screen. Not accusing questions. Odd ones. How old were you the first time you were arrested? Do you feel discouraged at times? Would a hungry person have the right to steal?
The answers go into a piece of software called COMPAS, and out the other end comes a whole number between 1 and 10. That number will sit in a file that a judge may read before deciding whether you wait for your trial at home or in a cell. Between 2013 and 2014, Broward County, Florida, produced 18,610 of these numbers.
Try it. The scoring below is a small, honest, made-up model — real COMPAS weights are not public — but the questions are the kinds of questions the real intake interview asks, and it behaves the way the real thing behaves: heavily driven by age and by how many times you have been arrested before.
Answer as anyone you like — yourself, a character, a guess. The number assembles as you go.
Answer the questions to see a score.
True of the real questionnaire too. Hold on to that fact — we are going to come back and take it apart in Act Six.
Act Two · Two headlines, seven weeks apart
On 23 May 2016, four reporters at ProPublica — Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner — published a story called Machine Bias. They had obtained the Broward scores, waited two years, and checked who was arrested again.
“The algorithm made mistakes with black and white defendants at roughly the same rate but in very different ways.”ProPublica, Machine Bias, 23 May 2016
Here is the difference they meant. Among the people who were not arrested again in the following two years — the people the number got wrong — Black defendants had been labelled high-risk at 44.9%, and white defendants at 23.5%. Almost twice as often. And among the people who were arrested again, it was white defendants who had been waved through as low-risk far more often: 47.7% against 28.0%.
Seven weeks later, on 8 July 2016, the company that makes COMPAS — Northpointe — replied with a report titled COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity. It did not dispute the arithmetic. It produced different arithmetic.
Turn the table the other way, Northpointe said. Don't ask “of the people who didn't reoffend, how many were labelled high-risk?” Ask “of the people we labelled high-risk, how many actually reoffended?” Answer: 63% of Black defendants and 59% of white defendants. A 7 means about the same thing whoever is holding it. That is what a fair number is.
Both sets of numbers came out of the same 6,172 people. Both were computed correctly. Neither side was lying. For two years the two camps talked past each other in public — and then two mathematical papers, five weeks apart in the autumn of 2016, explained why the argument could never have been won.
Act Three · The bench
Below is a cohort of 5,278 people — 3,175 in one group, 2,103 in the other, the same sizes as ProPublica's. They are invented, but they are not arbitrary: their score distributions and their reoffending rates were fitted so that when you set the cut where Broward set it, the published Broward numbers come out to within half a percentage point.
One thing was built in deliberately: a 7 means the same thing in both groups. Everybody with a 7 has the same chance of being arrested again, regardless of which group they are in. The score is honest in that specific sense — and, in the real data, so was COMPAS.
Your job is simple. Move the cut until the three lamps are all green. Take as long as you like.
Everyone at or above the cut is labelled HIGH RISK. Everyone below it is labelled low risk. Drag the cut and watch what happens to the four boxes.
Every dot is a person. Filled dots were arrested again within two years; hollow dots were not. The vertical line is your cut.
Act Four · Why the lamps fight
Here are the two facts that do all the damage. The first is a ladder: for each score from 1 to 10, the fraction of people who really were arrested again. The two lines lie almost on top of each other. That is the score keeping its promise.
Same score, same chance — for both groups. Hover a point for the numbers.
The share of each group at each score. Group 1 is spread almost evenly; group 2 piles up at the bottom. Nothing about the ladder changed — only who is standing on which rung.
Because the groups sit differently on the same ladder, they have different base rates — different overall fractions who are arrested again. In the real Broward data those rates were 0.51 and 0.39. Northpointe said so themselves, and pointed straight at it as the cause.
“Differences in the base rates of blacks and whites for general recidivism (0.51 v. 0.39) … strongly affected the Sensitivity and Specificity tradeoffs observed in the PP study.”Northpointe, COMPAS Risk Scales, July 2016
In October 2016, Alexandra Chouldechova of Carnegie Mellon wrote down the reason in one line of algebra. It is not a deep line. You could derive it on a napkin. It is simply true, always, for any test of anything:
Live, from your own cut. FPR = false-positive rate, FNR = false-negative rate, PPV = share of the flagged who really did reoffend, p = base rate.
Read it slowly. If you force PPV to be the same in both groups (Northpointe's fairness) and you force FNR to be the same (half of ProPublica's), then FPR is no longer yours to choose — it is pinned by p/(1−p). Two different base rates, two different false-positive rates. There is no algorithm anywhere, however clever, that escapes an identity.
Act Five · The theorem
On 19 September 2016, Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan posted a paper called Inherent Trade-Offs in the Fair Determination of Risk Scores. They wrote down three things everyone agrees a fair score should do, and then proved that nothing can do all three.
“We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously.”Kleinberg, Mullainathan & Raghavan, arXiv:1609.05807
“Except in highly constrained special cases” is doing a lot of work in that sentence. There are exactly two escape hatches, and the paper names them: the score is perfect, or the two groups have equal base rates. Below, you can open each hatch by hand and watch the lamps go green.
Same cohort, same bench. The left dial makes the world equal; the right dial makes the score a prophet. Watch the three lamps.
Notice what the two hatches actually are, once you push the dials all the way. The first says: make the two groups equally likely to be arrested again. That is not a request to the programmers. That is a request to history — to policing, to housing, to schools, to who gets stopped on which street. The second says: be never wrong. Also not available.
And there is a third, silly hatch the theorem allows and nobody wants: flag everybody, or flag nobody. Slide the cut to 1 or to 11 on the bench above and you will find the lamps behave — because a number that says the same thing about everyone is perfectly fair and perfectly useless.
Act Six · The stamp comes off
Back to Act One. The form never asked. That is true, and it is the single most common defence of any system like this. So let's test what it's worth.
Below, a tiny model is trained in your browser — real gradient descent, about thirty lines of arithmetic — on the non-race answers alone. Its only job is to guess which group a person is in. If race is genuinely absent from the data, it should do no better than a coin.
Five ordinary, race-free features. Untick any you think are unfair to use, then train and see how well the model still guesses.
Press train. Watch the loss fall and the accuracy climb.
In our simulated cohort the guesser lands somewhere near 70% — not because anything sinister was hidden in the data, but because in a society where two groups have lived differently, almost every honest measurement of a life carries a little of the group in it. Where you were arrested. How old you were the first time. How often you have moved. A postcode is not a race, and a postcode is not nothing.
This is why deleting the race column does not make a model race-blind. It only makes it race-blind to you.
Act Seven · One hundred and thirty-five questions too many
In January 2018 two researchers at Dartmouth, Julia Dressel and Hany Farid, ran an experiment so plain it is almost rude. They took 1,000 of the Broward defendants, wrote each one up as a short paragraph, and showed them to strangers on the internet — 400-odd people with no training in criminology at all, paid about a dollar, twenty cases each.
Pooled by majority vote, the strangers were right 67.0% of the time. COMPAS, on the same cases, was right 65.2% of the time.
Then they went further. They fitted an ordinary logistic regression — the simplest respectable model there is — using just two numbers about each person: their age, and how many prior offences they had. Accuracy: 66.8%. Two questions did the work of a hundred and thirty-seven.
You can watch that happen. Below, a logistic regression is trained in your browser on our cohort, using only age and priors, and its ranking is scored against the ten-point score itself.
AUC is the chance that the model ranks a randomly chosen person who did reoffend above one who didn't. 0.5 is a coin. 1.0 is a prophet.
Press train.
There is a reading of this that says: then the algorithm is fine, it's just simple. There is another that says: then the 137-question interview was mostly theatre. And there is a third, which is the one that keeps people up at night: a two-variable model has exactly the same impossibility problem. The trap in Act Three is not a bug in COMPAS. It is a property of prediction.
Act Eight · Somebody still has to choose
Every time you dragged that slider you were doing something a machine cannot do for you. Move it left and you catch more of the people who really will be arrested again — and you also hold more people who never would have been. Move it right and you free more innocent people — and some of the people you free will hurt somebody. There is no setting with zero of both. The dial is a moral instrument wearing a numerical costume.
On 13 July 2016 — five days after Northpointe's rebuttal — the Wisconsin Supreme Court decided State v. Loomis. Eric Loomis had argued that being sentenced with the help of a secret, proprietary risk score violated his right to due process. The court allowed the scores to keep being used, but ordered that every one of them arrive with a written warning attached: that the algorithm is proprietary and cannot be inspected, that it was built on group data and not on him, and that it must never be the sole basis of a sentence.
Two years earlier, in August 2014, US Attorney General Eric Holder had told the Sentencing Commission he was worried:
“They may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and in our society.”Eric Holder, US Attorney General, August 2014
And the deepest problem is one no theorem touches. All of this — every number on this page, ProPublica's and Northpointe's alike — is built on re-arrest, because re-arrest is what gets written down. Re-arrest is not the same as re-offending. It is a measurement of behaviour and of policing, mixed together, and nobody has ever managed to unmix them. The base rates that make the theorem bite are themselves a record of how the world was watched.
Six honest wobbles
Ninety-eight years of scoring people
The exam
No trick questions. One of them has an answer that annoys almost everybody.