What is Object Detection? — one analogy

1The analogy

Books lie jumbled together on a bookstore display table. To take stock, calling the whole table "a table of books" isn't enough — a colored tag has to go in next to every single book. A blue tag for a novel, a green tag for a cookbook. Once the count is done, you can tell at a glance that there are three cookbooks on one side of the table and two novels in the corner.

Whoever places the tags doesn't only tag the obvious ones. Ambiguous books get a tag too, and if the same book later ends up with two or three tags stacked on it, only the best match gets kept and the rest get pulled. A book that's half hidden and blurry still gets a tag, marked "not sure."

Standing up a single "table of books" sign and tagging every book individually take completely different amounts of work. Object detection does the tagging.

2In detail

Several at once, in one pass

Image classification gives one photo one answer. Object detection doesn't even start with a fixed number of answers — a photo might have nothing in it, or it might have twenty things. This not-knowing-the-count-in-advance is exactly what makes the problem so much harder.

Modern methods lay a grid over the photo and ask every square, all at once, "is there something here?" Because it scans the whole photo just once and gets every square's answer at the same time, it's very fast — tens of times faster than the older approach of hunting down objects one at a time, which is exactly what makes a box able to follow an object around in real time on webcam video.

Keeping several box templates of different shapes ready for each square is another trick. A tall narrow one, a flat wide one, one close to square — laid down ahead of time and then stretched and nudged a little to fit the real object. That fits far better than drawing a box from a blank slate.

Four things come with every answer

One line of an object detection result looks like "cat, top left, 120 wide by 90 tall, confidence 0.87." Unlike classification, which only gives a label, position and size come along with it.

Confidence is worth watching closely. Show every box down to the faintest confidence and the screen drowns in boxes. So only boxes above some threshold get kept — raise the threshold and the screen gets cleaner but blurry objects get missed; lower it and you catch everything but pick up false positives too. When a box flickers in and out in a demo, that's confidence hovering right around the threshold.

Cleaning up overlapping boxes

An object almost never gets exactly one box. Since several neighboring squares each spot the same object and each puts out its own box, before cleanup the screen is stacked with near-duplicate boxes.

So the box with the highest confidence gets kept first, and whatever overlaps it heavily gets thrown out. Then the process repeats on what's left — keep the next-highest, drop what overlaps it — until nothing more overlaps. How much overlap gets tolerated is adjustable too; set it too strict and one of two objects sitting right next to each other can vanish entirely. That's how a person or two can disappear from a crowded photo at this step.

What's hard about it

Small objects are especially difficult. As the picture shrinks going through layer after layer, a far-off object is left with only a few squares, and its trace disappears. That's why detectors combine a wide view and a close-up view together when making a call.

Occluded objects are tricky too. Even a person can't agree on exactly where to draw a box around a chair that's half hidden. If the people who built the training data didn't agree on where the line goes, the model gets confused the same way. And, just as with image classification, anything not on the list simply never comes out — no matter how unusual the object you hold up to the webcam, the screen stays silent if it falls outside the model's trained list.

3More precisely

Object detection takes one photo and produces several results, each made up of four box coordinates, a label, and a confidence score. Performance is usually measured by how much a predicted box overlaps the correct box, counting it as a hit once that overlap clears a set threshold, then averaging scores across labels into one number.

Training data is also far heavier than for classification. Every single object in every photo needs a box drawn around it and a name attached, which multiplies the human effort it takes to prepare one photo several times over.

The bookstore analogy breaks down in one place. A person tags one book at a time by looking at it, but a model doesn't recognize objects one by one — it produces thousands of candidate boxes across the whole photo at once, then filters them down. The handful of clean boxes you actually see are what's left after that filtering. And a person can lift books to look underneath even when they're stacked, but a model only ever judges from what's visible on screen, which is why it struggles with anything occluded.

4Try it yourself

5Common misconceptions

It's easy to think object detection traces an object's exact shape, but actually it only gives an axis-aligned box, and background comes along for the ride inside it.
It's easy to assume no box means no object, but actually a candidate may have existed and simply been filtered out for falling under the confidence threshold.
It's easy to think the boxes on screen are everything the model produced, but actually what you see is only what's left after cleaning up the overlaps.

7One-line summary

In shortObject detection finds every object in a photo and attaches a box, a label, and a confidence score to each one, with cleaning up the overlapping boxes as part of the same package.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02