What is Image Classification? — one analogy

1The analogy

A cut of meat sits on the counter at a butcher shop. The butcher glances at it once and calls out flatly, "shoulder." Not two names at once, not a hedge like "could be shoulder, could be neck." One name, and that's it.

The names the butcher can call out come only from the chart hanging on the wall. Bring in a cut that isn't on the chart, and you won't get "unknown cut" as the answer — you'll get whichever name on the chart comes closest. That chart is the entire way this shop divides up the world.

One more thing: what comes back is just a name. Where that cut was originally sliced from, or which side of the tray it's sitting on — none of that gets said.

2In detail

Picking only from a fixed list

Building a classifier starts with deciding the label list. Say you decide on three: dog, cat, bird. From then on, no matter what photo comes in, the answer is one of those three. Feed it a photo of a mug, and you'll still get back dog, cat, or bird.

Knowing this makes results far easier to read. When a weird answer pops out, the model often isn't broken — it's frequently just been asked a question it can't actually answer. That's why real deployments often add an "other" slot to the list and train the model to route unrelated photos there.

Even a label that is on the list can wobble if what's in front of it looks nothing like what it trained on. Train it only on daytime photos and it flounders on the same object at night. That's exactly why varying the background and angle across several shots makes a webcam classifier noticeably better.

The answer comes out as scores

A classifier never flatly states "this is a cat." It scores every label instead — cat 0.82, dog 0.15, bird 0.03, say. All the scores are set up to add to 1, so you can read each one roughly as a percentage.

What we see as "the answer" is just the highest of those. But a top score of 0.9 and a top score of 0.35 are very different situations. The first is fairly confident; the second is merely a bit ahead of the rest. Turn an object slowly in front of a webcam and you can watch the score bars wobble — that wobble is a picture of how uncertain the model is.

That's why real services often only accept the top score as the answer once it clears some threshold, and treat anything below it as "not sure."

It won't tell you where something is

The biggest limit of image classification is position. Feed it a photo of a desk and you'll get "desk," but it won't say where on the photo the desk sits, or how many cups are on top of it. If a photo has several things in it, the classifier gets pulled toward whichever one stands out most.

One step up from here is object detection, which boxes every object; one step past that is segmentation, which cuts out shapes pixel by pixel. One label for a photo, one box per object, one label per pixel — the more precision you need, the heavier the training data gets, too. Classification just needs a name written on each photo, but boxes and shapes have to be drawn in by hand, one by one.

You can try it right now with a webcam

Image classification has a lower bar to entry than the tasks above it mostly because training data is easy to gather. Sorting photos into folders by kind is basically all the prep it takes.

A common trick is to take a large model that's already looked at a huge pile of photos and retrain only its last part on your own photos. The ability to recognize fine patterns and parts carries straight over, and only the label-picking part gets taught fresh. That's why a bare ten or so photos per category is often enough for a usable classifier — the browser-based demos that finish in a few minutes all lean on exactly this trick.

3More precisely

Image classification takes one photo as input and produces a score against a predetermined list of labels. The last step usually spreads those scores out so they add up to 1, which lets you treat the result like a probability. That said, this value isn't exactly the same as human confidence — it's known to run higher than it should.

Because the folder names a person sorted the photos into become the labels directly, how the data was gathered heavily shapes the result. If one category has far more photos than the rest, or was only ever shot against one background, the model can end up mistaking that background for the label itself.

The butcher shop analogy breaks down in one place. A person judges by touch and smell as well as sight, but a classifier's only clue is whatever is captured in that one photo. And instead of picking a single name, there's also a version that attaches several labels to one photo at once — tagging a single photo "beach," "sunset," and "summer" together. In that case the scores aren't spread to add up to 1; each label gets judged on its own.

4Try it yourself

5Common misconceptions

It's easy to think a classifier says "I don't know" when it doesn't know, but actually it forces out one answer from the list even for a photo that fits nothing on it.
It's easy to assume a high score means it's definitely right, but actually it's common for the model to confidently give a high score to the wrong answer in a situation it never trained on.
It's easy to think good classification also tells you where an object is, but actually getting position out requires changing both the training data and the last part of the structure.

7One-line summary

In shortImage classification looks at one photo and picks a single label from a predetermined list, so it tells you what something is but never where it is or how many there are.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02