ROC & AUC
Scores across every cutoff, folded into one curve and one number
- A sorting model doesn't hand back a straight answer. Someone has to set a cutoff for how sure counts as "yes".
- The ROC (Receiver Operating Characteristic) curve is a single line tracing performance across every cutoff, from one end to the other.
- The closer the curve hugs the top left corner, the better the model. A curve lying flat on the diagonal is no help at all.
- AUC (Area Under the Curve) is that entire curve reduced to one number. Closer to 1 is better, and 0.5 is the same as guessing.
- Being independent of any cutoff makes it good for comparing models, but it can look more generous than it deserves when one category is rare.
Contents
1The analogy
The fridge has several cartons of milk, all sitting some days past their printed date. Some are still fine; some have gone bad. There's no way to know without opening them, so the call has to be made from the date alone.
Deciding "toss it the moment the date passes" means never drinking spoiled milk, at the cost of tossing plenty that was still fine. Deciding "keep it up to a week past" means almost nothing gets thrown out, at the cost of drinking spoiled milk sometimes. Neither rule is simply correct — moving the cutoff by a day just trades what gets gained for what gets lost.
So a single cutoff never gets tested alone. Sliding through every day from zero to ten days past the date, and writing down "how much spoiled milk got caught" and "how much good milk got tossed" at each one, captures the whole character of this decision on one page.
2In detail
The cutoff is set by a person, not the model
A sorting model doesn't output "this" or "that" — it outputs a score for how likely something is. Where that score has to cross before it counts as "this" is a line a person draws. That line is the cutoff.
Lower the cutoff and anything even slightly suspicious counts, so more real cases get caught, at the cost of catching some that aren't. Raise it and only the sure things count, so fewer false alarms happen, at the cost of missing more real ones. It's exactly like sliding the date cutoff back and forth.
So a score measured at one fixed cutoff is only the score at that cutoff — move the line and the same model's number changes. Judging how good a model actually is at telling things apart means sweeping across every cutoff, not freezing on one.
One point per cutoff, strung together into a curve
Set the cutoff at its lowest and measure two things: how many real "this" cases got caught, and how many "that" cases got wrongly caught along with them. Those two values become the coordinates of one point.
Raise the cutoff a notch, measure again, plot another point, and keep going to the end — the points line up into a row. Connecting them is the ROC curve. The bottom-left corner is catching nothing at all; the top-right corner is catching everything.
A strong model's curve hugs the top left. That means there's a cutoff that catches nearly everything real while barely raising a false alarm. A curve lying along the diagonal means every cutoff gains exactly as much as it loses — no help at all.
Reducing the curve to one number
A curve carries a lot of information, but it's awkward for lining several models up side by side. So the area underneath it gets measured and reduced to one number: AUC. The closer the curve sits to the top left, the larger that area, so a bigger number means a model that sorts things better.
That number comes with an easy reading attached. Pick one real "this" case and one real "that" case at random — AUC is the probability the model scores the "this" case higher. A value of 0.9 means it gets the ordering right about nine times out of ten.
0.5 is the same as a coin flip. Getting the order right half the time means there's no information there at all. Below 0.5, the reading flips — a model that scores worse than chance is usually a sign something's set up backwards, not a model that's somehow "wrong on purpose usefully."
Where it turns overly generous
What makes AUC convenient — no cutoff to pick, and a number that barely moves when the ratio between categories shifts — is exactly what turns into a trap in one situation: when the category being searched for is very rare.
Say the task is finding ten cases out of ten thousand. Wrongly flagging a hundred out of the other nine thousand ninety looks like a tiny share when it's spread over that many. The curve still hugs the top left and AUC still comes out high. But in practice, catching those ten real cases means checking a hundred false ones by hand — not a very usable tool.
For situations like this, a different curve focused only on the rare category gets checked alongside it — one plotting the share of flagged items that were real against the share of real ones actually caught. That curve behaves far more honestly when the rare category is the one that matters.
3More precisely
The two axes of the ROC curve have fixed names. The vertical axis is the share of real "this" cases caught, called recall. The horizontal axis is the share of real "that" cases wrongly caught. Both axes are computed within their own group, so the curve's shape barely shifts even when the ratio between the two groups changes.
The analogy has a limit. Milk gets judged from a single value — the date — while a model weighs many pieces of information at once to produce its score. That mismatch actually points at something important, though: the date isn't spoilage itself, just a stand-in that tracks it, and a model's score isn't the right answer itself either, just a stand-in that tracks it. A curve hugging the top left means that stand-in is a good one. A real curve can slide its cutoff in arbitrarily fine steps, so it actually looks like a staircase as fine as the number of data points.
Picking a model from AUC alone is worth being careful about too. AUC only says how well something orders things — it says nothing about whether that score can be trusted as an actual probability. Using it for real means picking a cutoff, and that cutoff should be chosen by weighing the cost of missing something against the cost of a false alarm.
4Try it yourself
5Common misconceptions
It's easy to think a high AUC means a model that gets things right, but actually it only means the ordering is good — pick the wrong cutoff and the number of correct calls can still be terrible.
It's easy to think an AUC of 0.5 means half the performance, but actually it means no information at all, the same as guessing.
It's easy to think the ROC curve can be trusted for any problem, but actually it paints a more generous picture than reality when the category being searched for is very rare.
7One-line summary
In shortThe ROC curve traces performance across every cutoff on one line, and AUC folds that curve down to a single number for comparing models against each other.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02