What is Multimodal? — one analogy

1The analogy

Say you go to a lost-and-found to claim your bag. One person hands over a photo saved on their phone. Another describes it out loud: a maroon cloth bag with one strap coming loose. A third just writes down the day, the time, and the platform number.

Three completely different forms handed over. But in the mind of whoever's receiving them, all three point to the same thing. See the photo, hear the description, read the timestamp — all three lead to the third shelf over there. The forms differ; what they're pointing at is one and the same.

That's why you can ask with a photo and get an answer back in words, or describe something and have it pull up the matching picture.

Multimodal is that lost-and-found.

2In detail

Different forms land in the same spot

Text, images, and sound look nothing alike on the surface. Text is a line of characters, an image is a grid of dots, sound is a wave rising and falling over time. As they are, there's no way to compare them.

A multimodal model turns each of the three into its own bundle of numbers. What matters is that once converted, all three land in the same space. The number bundle that comes out of a photo of a cat and the number bundle that comes out of the word "cat" are trained to end up sitting close together.

Once they're in the same spot, everything after that works the same way. Compare and connect them regardless of which form they came from. That's how it becomes possible to judge whether a photo matches a sentence, or whether a sound lines up with a caption.

What it makes possible

The most common use is looking at a photo and describing it. Snap a photo of what's in the fridge and ask what to cook, or photograph a hand-drawn table and ask for it to be organized. Anything tedious to type out can just be shown instead, cutting the effort.

It works the other way too. Describe something in words and find a matching picture, or feed in a long recording and ask for just the fixed set of things you need pulled out. Mixing several forms into one request works too — feed in a photo of a manual page along with a written question, and it looks at both together to answer.

There's one reason this way of working feels natural: what a person runs into never comes in just one form to begin with.

The answer comes in several forms too

It isn't only the input side that splits into several forms — the output can too. The same question can come back as text, be read aloud, or be drawn as a picture.

Still, one single model handling all of that by itself is rare so far. More often, several parts are stitched together and act as one team. It looks like one seamless thing to the person using it, but underneath, a part that looks at images, a part that generates words, and a part that produces sound can each be doing their own piece.

Where it falls short

Taking in every form doesn't mean it's equally good at all of them. Generally, images lag behind text, and sound lags behind images. It comes down to how much training material exists for each form.

It's especially weak on fine detail. Misreading small text in an image, mixing up left and right, miscounting how many of something there are — all common. It reads the overall mood well but stumbles on close inspection.

So for anything where an exact number or an exact position matters, it's worth having a person check. Feed in a photo that's bright and clear, with the part you care about large in the frame, and results improve noticeably. Filling one photo with a lot of things is worse than showing only what you want to know about, made large.

3More precisely

A modality just means one channel information can arrive through. Text is one channel, images are one channel, sound is one channel. Multimodal means handling two or more of those channels, and each channel gets its own front-end piece that converts it into a number bundle. Only after training places those results in the same space can they actually be mixed and matched.

The lost-and-found analogy breaks down in a spot. A human clerk asks a follow-up question when the photo and the description don't match; a model sometimes builds a plausible-sounding answer even while the two don't line up. That's where it starts claiming something is in a photo that isn't actually there. A person also scans a picture by moving their eyes around it and looking closely, while a model tends to take the whole thing in at once, which is part of why it misses small details.

Handling several forms doesn't mean handling them all the same way either. Images get sliced into small patches and given an order; sound gets unrolled over time and handled more like a picture of itself. It can look, from outside, like everything's dropped into one window, but each form gets its own separate treatment underneath.

4Try it yourself

ChatGPT (OpenAI) ailearn.space Snap a photo of a handwritten note or a receipt and ask it to organize the contents — the difference is immediate

5Common misconceptions

It's easy to think being multimodal means being equally good at every form, but actually the skill gap between forms is often quite large.
It's easy to think feeding in an image means it reads the text inside accurately, but actually small text and blurry photos get misread often.
It's easy to think taking in several forms is just one added feature, but actually it only becomes possible through separate training that places the forms in the same space.

7One-line summary

In shortMultimodal is the ability to move text, images, and sound into the same space so one model can handle all of them together, which is what lets you ask in one form and get an answer in another.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02