Why a 95% Accurate Model Can Still Fail Unevenly

Overall accuracy is an average across every case a model was tested on. Averages hide exactly the kind of unevenness that matters here — a model can be excellent for most patients and meaningfully worse for a specific group, and the overall number will still look reassuring, because the group doing well outnumbers the group doing badly.

Work a simplified version of that out to see how much an average can hide. Imagine a model tested on a population that is 90% one group and 10% another, and every single one of its errors happens to land in the smaller group. An overall accuracy of 95% is entirely consistent with that model being closer to 50% accurate for the group making up a tenth of the test set — the ten percent barely moves the overall number, precisely because it is ten percent. The one number on the slide cannot tell you, on its own, whether that is what happened.

The uneven training data in the chart below is one way this happens. A quieter, more common one is that a model ends up predicting who has historically received care rather than who actually needs it, because the label it was trained on was never really “is this person sick” — it was something easier to measure that the people building the model used as a stand-in.

One widely studied example did exactly this. A commercial algorithm used across US health systems to identify patients who would benefit from extra care management was trained to predict future healthcare costs, on the reasonable-sounding assumption that sicker patients cost more. But at the same level of illness, Black patients had historically generated lower healthcare costs than white patients — not because they were healthier, but because of unequal access to care that had nothing to do with the algorithm. The model faithfully learned the pattern in the data it was given: cost predicted risk score, and Black patients, despite being just as sick, got systematically lower risk scores and were flagged for extra help less often. Researchers estimated that correcting this one substitution — training on actual illness instead of cost — would have raised the share of Black patients identified for extra care from under one in five to nearly one in two.

Nobody involved chose to build a biased algorithm. Cost looked like a defensible, measurable proxy for need, right up until someone checked whether it actually measured the same thing for every group it was used on. That mechanism matters more than the specific example: a model does not need to be told anyone's race, sex, or income to reproduce exactly the inequality already baked into whichever number it was told to predict.

The chart below simulates a second, more direct version of the same problem. Same model, same day, one overall accuracy figure — and three subgroups underneath it that received wildly different shares of the training data before the model was ever deployed.

A proxy variable standing in for illness is one mechanism. Two more are both well documented, and both are simpler than anything involving a learned model.

Neither of these is a training-data problem a bigger dataset trivially fixes, because the gap isn't random — it reflects who historically had access to the dermatologists, the studies, and the equipment that generated the data in the first place. A model built on that record inherits the record's blind spot rather than correcting it.

A group that made up 7% of the training data is not a rounding error to that group — it is the entire population the model performs worse for, every single time it is used on someone from that group. The model was never lying about its overall accuracy. It was just never asked to report the number that actually mattered to the patients it was failing.