What is Image Captioning? — one analogy

1The analogy

Slide a photograph into a projector, switch on the light, and a picture appears on the wall. Standing beside it, a narrator adds a line: "An old mailbox stands in the middle of a field." Advance to the next slide and the next line follows.

Image captioning is a machine taking over that narrating job. It scans what's on the wall, works out what's there, what state it's in, how things are arranged relative to each other, and folds all of it into one sentence.

There's a skill to narrating well. List absolutely everything visible on the wall and the listener gets worn out; say only "it's a landscape" and it's no help at all. Deciding what to mention and what to skip is half the job.

2In detail

A reading half and a writing half work as a pair

The front half handles the picture. It scans the photo in small pieces, pulls out color and shape features, and turns the result into several bundles of numbers. Nothing in any language comes out yet at this stage — just information like "there's a round reddish shape here, with something flat underneath it."

The back half handles the sentence. Given the number bundles the front half produced, it picks the next word piece, attaches it, and repeats. It's the same trick a text-writing model uses to pick its next piece from the sentence so far — the only difference is that the reference material here is picture information instead of preceding text.

Splitting the two apart pays off. The picture-reading half reuses everything it already learned about photos, and the sentence-writing half reuses everything it already learned about language. The only new thing to learn is the bridge connecting the two.

What to say and what to skip

Several descriptions can fit the same photo. "There's a cup on the desk" works, and so does "steam rises from a cup on a desk by a window catching the morning light." Which one counts as good depends entirely on what it's for.

Alt text works best short and precise, since someone who can't see the screen needs to picture the scene from hearing it once. A description meant for photo search works better long and dense, since you don't know what words someone will search with later, so packing in more words worth matching pays off.

That's why the same model gets steered with a request like "in one sentence," "describe it in detail," or "read out any text you see." Left with no request at all, it defaults to whatever length and tone was common in its training material.

Learning from paired pictures and descriptions

The training method is simple. Pair up a photo with a description a person wrote for it, show the model the photo, and have it guess the description. Nudge it a little each time it misses, and repeat across hundreds of millions of pairs.

So whatever character that description data carries shows through in the results. A common scene in the training material produces common phrasing; a place or object that's underrepresented there gets described in vaguer terms when a photo of it comes up. The habits of whoever wrote the original descriptions carry through too.

Sometimes it says things that aren't in the frame

This is the most-cited weakness. Show it a kitchen and it might claim a pot that isn't actually there, or miscount the plates on a table. Instead of reporting what it actually sees, the sentence drifts toward "a scene like this usually has one of these."

It's especially weak on anything that needs precise counting or checking — small text, exact counts, left versus right, how far apart things are. A description reading smoothly is no guarantee it's correct, so anywhere accuracy actually matters, it's safer to have a person check it once.

3More precisely

Image captioning connects an encoder that reads the picture to a decoder that writes the sentence. The encoder splits the photo into pieces and turns each into a vector; the decoder refers to those vectors while picking one token at a time to complete the sentence. This whole structure now usually lives inside a larger multimodal model that handles pictures and text together, and generating a caption is just one of many requests you can hand that model.

The projector comparison breaks down in places. A narrator knows things outside the slide — when and where it was taken, what the photographer meant to capture — and speaks from that. The model only has what's inside the frame. Any backstory it states beyond what's shown isn't knowledge — it's invented. And where a narrator looks at the wall and picks their words, the model turns the whole picture into numbers first, then writes the sentence one piece at a time from the start, so a wrong pick early on drags the rest of the sentence along with it. A narrator can also pause, look again, and correct themselves mid-sentence; the model has no way to glance back at the wall once it starts writing.

4Try it yourself

Say What You See (image prompting practice) ailearn.space Try putting a picture into words yourself first — you'll feel exactly how tricky the job is

5Common misconceptions

It's easy to think this is the same as finding objects in a photo and naming them, but actually it adds a layer on top of that — weaving relationships and situation into a sentence.
It's easy to think a natural-sounding description means the content is accurate, but actually a smooth sentence and an accurate report of what's on screen are two separate things.
It's easy to think the same photo always produces the same description, but actually the length and tone you request can change the description a lot, even for the same photo.

7One-line summary

In shortImage captioning is a narrator's job handed to a machine, with a picture-reading half and a sentence-writing half working together to produce one line.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02