What is Evaluation Metric? — one analogy

1The analogy

Bring a car in for inspection, and the inspector doesn't tell you whether it's a good car. Instead they pull out a checklist. How much braking force it produces, whether the exhaust reading falls within limits, whether the headlights clear the required brightness, whether there's corrosion underneath — each item gets measured separately.

So what you walk away with is a set of numbers by item, not a verdict of "good." A car with strong brakes and a car with clean exhaust might be two different cars. Depending on which box you look at first, the same car reads as well-made or as needing work.

An evaluation metric is one box on that checklist. Saying an AI "does well" carries no meaning on its own — it only means something once you say which box you're measuring.

2In detail

Picking the yardstick comes first

An inspection station decides what it's measuring today before the car ever pulls in. The items have to be fixed first, so the equipment can be set up and a pass line drawn. Building an AI works the same way. What counts as "doing well" gets decided before training even starts.

Skip this order and trouble follows. Pick the metric after the model is already built, and the temptation is to pick whichever metric makes the result look best. Swapping metrics after the fact to find a flattering number isn't measuring — it's cherry-picking.

A metric decided in advance steers the whole build. It spells out which direction counts as improvement, so the metric effectively becomes the goal.

"Doing well" means something different per item

Compare two cars. One brakes hard but runs dirty exhaust; the other runs clean but brakes are shaky. There's no single answer for which car is better. Drive mostly at night and the headlights matter most; drive near a school zone often and the brakes matter most.

AI is the same. Catching everything that matters and never crying wolf on something harmless are two different jobs, and pushing one up can pull the other down. That's exactly why accuracy, precision, and recall get tracked as separate items — they measure different things.

So before trusting a leaderboard, check which metric did the ranking. Swap the metric, and it's not unusual for first and third place to trade spots.

Watch only one item, and the model gets trained to please it

If an inspection station only checked exhaust, a carmaker could tune the engine to run clean during that specific test and nothing more. It passes inspection, but real-world driving is a different story. Track a single yardstick, and hitting that yardstick quietly replaces the real goal.

The same happens while building AI. Tell it to raise one number, and training bends everything toward that one number. If shorter answers happen to score better on the metric, answers keep getting shorter. If guessing the most common option scores well, it keeps guessing that.

So in practice, several items get watched together. One primary metric sets the direction, with a few guardrail metrics standing alongside it to catch whatever that primary metric might quietly break.

What isn't measured doesn't get fixed

An item missing from the checklist gets no attention from anyone. A loose seat goes unnoticed at the inspection station if it's not on the list. Metrics work the same way. Whatever gets measured improves; whatever doesn't quietly gets worse without anyone noticing.

So picking a metric should start from what you actually want, not from what's easy to measure. Whether an answer was actually useful, whether the person came back to use it again — these are hard to turn into a number but close to the real goal. When an easy-to-measure number stands in for it, remember that number is only ever a stand-in.

A metric isn't chosen once and done, either. When the users and the situation change, the checklist needs revisiting too.

3More precisely

An evaluation metric is a rule that compares a model's output against the correct answer and boils the comparison down to one number. Which metric fits depends on the job. Classification tasks lean on accuracy-style metrics; predicting a number leans on metrics that measure how far off the guess was; generating text sometimes leans on scores given by a person or by another model. If the data used to measure overlaps with the data used for training, the number gets inflated — measuring only on data the model has never seen is the rule.

The analogy breaks down in one place. A car inspection has a pass line set by law; AI metrics have no such absolute standard. The same metric measured on different data can swing wildly, so numbers from different datasets can't be compared directly. A number only becomes comparable once you note which data and which conditions it was measured under.

Worth separating a metric from a loss function too. The penalty that steers training and the report card a person reads afterward are different numbers. They don't always point the same direction — a shrinking penalty during training and an unchanged report card can happen at once.

4Try it yourself

Google's Machine Learning Crash Course ailearn.space Follow the section that measures the same model with several metrics — it shows how the verdict shifts as the metric changes.

5Common misconceptions

It's easy to think a high metric score means a good model, but actually the same model's score can shift a lot depending on which item was measured on which data.
It's easy to think a metric only measures the result, but actually deciding what to measure also decides the direction of training and what gets fixed.
It's easy to think picking one good metric is enough, but actually watching only one lets the model bend entirely to it, so a guardrail item usually sits alongside it.

7One-line summary

In shortAn evaluation metric is the fine print behind the phrase "doing well," and which box gets measured decides what the model ends up good at.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02