What is Computer Vision? — one analogy

1The analogy

Stand in front of a painting in a museum and there's a placard beside it. Instead of leaving the painting as a blob of paint, the placard spells it out in words: "Harbor at dusk, three ships, a lighthouse at the left edge." Looking at the same painting, once someone points out what's where, it finally clicks.

Computer vision is that placard. To a computer, a photo is just a dense grid of colored dots, meaningless on its own. Computer vision feels out the edges and patterns hidden in that pile of dots and turns them into words: what's there, how many, and roughly where. It's not giving the computer eyes — it's attaching a placard that turns dots into a description.

2In detail

To a computer, a photo is a table of numbers

Zoom way into a photo on screen and tiny squares line up in a grid. Each square carries a number for how much red, green, and blue mixed into it, and one photo is that number repeated across millions of squares. That table is all a computer ever receives.

A person has never seen that table. Eyes automatically clump things together and separate objects from background. A computer has to build that whole process from scratch, calculating how different a square is from its neighbors, and what shape that difference traces out, to find the clumps.

From dots to edges, edges to objects

Processing doesn't leap all at once — it climbs in layers. At the bottom, it only looks at brightness differences between neighboring squares. A line of sharp differences in a row is an edge. The next layer up looks at how those edges cluster together: repeat side by side and it's a stripe, close into a loop and it's something round.

Go up further and it looks at how these pieces are arranged. A round shape with two horizontal lines above it and a vertical post below starts to look like a "something," once enough of these arrangements stack up. Lower layers catch patterns generic to any photo; higher layers catch the traits specific to one particular object.

There was a time when people designed these layers by hand, one at a time. Now the standard approach shows it a huge pile of example photos and lets each layer decide for itself what to look for, and performance jumped a lot after that shift.

What you ask changes the shape of the answer

The same photo can trigger entirely different work depending on what's asked. Ask "what is this photo," and you get one label for the whole picture. Ask "what's where," and you get a box drawn around each object with a name attached.

Ask "exactly which pixels belong to it," and every single dot gets sorted into object or background, which is what powers erasing a background or recoloring just one subject. Beyond these, reading text out of an image, measuring how far away something is, and tracking the same object across video frames are each their own separate branch.

It's already everywhere

A phone camera focusing and brightening a shot, a photo app sorting pictures into albums on its own, a video call blurring only the background — all of this is the same technology. A parking gate reading a license plate, a factory line picking out a scratched product: same family.

It shows up in less visible places too. Turning a photographed document into readable text, checking the condition of leaves on a farm — the repetitive visual checks a person used to do by eye have moved over here in large numbers.

Where it gets fooled, and where it fails

Even with better performance, it fails in places very different from where a person would. Hit it with an angle it's never trained on, lighting it's never seen, or an object half-hidden behind something else, and its scores drop off fast. Well-known experiments have added noise to a photo, invisible to the human eye, and gotten a completely different answer out of it.

What's more troubling is that it doesn't hesitate when it's wrong. The placard doesn't say "not sure" — it attaches a plausible-sounding description regardless. That's exactly why it matters to decide ahead of time where a human eye needs to double-check.

3More precisely

Computer vision is the name for the whole field of pulling meaningful information out of visual signals. Inside it sit several branches with different goals: classification, detection, segmentation, tracking, depth estimation. People used to hand-design the rules for finding patterns for a long time, but learning-based methods built around convolutional networks became the mainstream, and more recently, architectures originally built for text have been carried over to images too.

The analogy breaks down in places. A museum placard knows the background of the painting and what the artist intended, and explains with that in mind; computer vision has no idea what anything means, it only reads the statistics of a pattern. There's a well-known case where telling a dog from a wolf actually came down to whether there was snow in the background — it's not rare for the clue it latches onto to be something a person would never think to look at. A plausible-sounding description doesn't mean it looked at the same thing a person would, which is exactly why a confident answer and a correct one aren't the same thing here.

4Try it yourself

5Common misconceptions

It's easy to think a computer sees the way a person does, but actually it's computing the arrangement of brightness numbers, so it fails in places completely different from where a person would.
It's easy to think a sharp photo guarantees a correct answer, but actually an angle or lighting it's never trained on can produce a wrong answer even from a very clear photo.
It's easy to think finding a face and recognizing whose face it is are the same task, but actually detecting that something is there and identifying what it is are separate jobs with different goals and methods.

7One-line summary

In shortComputer vision attaches a placard to a photo that's really just a table of numbers, turning what's there and where it is into words a computer can work with.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02