BERT
A model trained to fill blanks by reading both sides at once
- BERT is a language model trained by punching a hole in a sentence and learning to fill it back in.
- It can look at both sides of the blank at once — that's what sets it apart from designs that only look backward.
- It doesn't continue text. Its job stops at working out the meaning and handing that off.
- That makes it strong at jobs with a short, clean answer, like classification, search, and question answering.
- A model trained once gets a small amount of extra fine-tuning and gets reused across many different jobs.
Contents
1The analogy
Open a mudroom shoe rack and each cubby usually holds one pair of shoes. Cover one cubby with a sheet of paper and ask "what was in there?" and you can guess surprisingly well.
The trick isn't staring at the covered cubby itself. If there are hiking boots on the left and a kid's pair of sneakers on the right, the range of shoes that would fit in the middle narrows fast. What's above and below, and whether it's summer or winter, are clues too.
What matters is that you can see both sides of the covered cubby at once. Cover the right side with your hand too, and the guess gets a lot harder — you're back to working off half the shelf.
That's the whole trick BERT is built around, just done on sentences instead of shoes.
2In detail
It practices filling in hidden blanks, endlessly
The way BERT learns is simple. Take any piece of text, hide roughly one word out of every ten, and have it guess what belonged there. The answer is already sitting in the original text, so nobody has to label it by hand.
One sentence can generate several practice problems on its own — hide the first word for one problem, the fifth word for another. Practically all the text on the internet becomes a workbook this way, which let it absorb an enormous amount without anyone touching it by hand.
This training builds more than a knack for guessing. Filling a blank correctly requires understanding sentence structure and how words relate to each other, so that understanding gets absorbed along the way, too.
Seeing both sides is the whole difference
A model that only reads in one direction can only see the words before the blank. Read only as far as "I went to the bank to ___ money," and it has to guess without knowing what comes after — deposit and withdraw both fit equally well from that side alone.
BERT also sees what's after the blank. "I went to the bank to ___ money, and afterward my balance had gone up" narrows things down immediately, because the balance going up rules one option out. Being able to see the other side changes the judgment even for the identical blank.
It hands off meaning instead of continuing the text
BERT isn't a conversation partner. Feed it a sentence and it doesn't write the next sentence — it packs the sentence's meaning into a bundle of numbers and hands that over. Whatever needs to make an actual decision from those numbers gets attached separately, downstream.
This suits certain jobs well: telling whether a review is positive or a complaint, routing a support message to the right team, finding documents with similar meaning, or pointing to the exact span of a long document that answers a question. None of these require inventing new text — only understanding correctly.
It shows up especially often in search. Finding documents with related meaning even when the wording doesn't overlap requires treating a sentence's meaning as numbers, which is exactly the role this design fits.
A small finishing touch on top of what it already knows
The blank-filling stage is slow and expensive, but it only has to happen once. After that, a small amount of extra training on examples specific to the job is enough.
Around ten thousand labeled reviews is enough to turn it into a sentiment classifier; a few thousand support messages turn it into a routing classifier. Building either from scratch would need far more data and time — instead, a model that already knows the language just gets a final touch-up.
3More precisely
BERT stacks only the encoder half of a transformer. It trained on two tasks. One was masked language modeling: hide part of the input and reconstruct it. The other was judging whether two sentences actually follow one another in order — a task that later research found added little value and that some later versions dropped.
The analogy breaks down in a couple of places worth naming. A shoe cubby is a visible unit, but the pieces BERT works with are chopped even finer than a word, so one word can span several pieces. And a shoe rack only asks about one covered cubby at a time, while BERT processes an entire sentence at once, refining the meaning of every position simultaneously.
Seeing both sides comes at a cost, too. Because it can see what's ahead, it can't be reused directly for continuing text — predicting what comes next requires not knowing it yet. That trade-off is exactly why designs like this and designs built for writing tend to specialize rather than merge into one, each one giving up something the other is good at.
4Try it yourself
5Common misconceptions
It's easy to think BERT holds a conversation like a chatbot does, but actually its job stops at summarizing a sentence's meaning and handing it off — it isn't built to continue text.
It's easy to think seeing both sides makes it strictly better than a forward-only design, but actually being able to see ahead is exactly what makes it hard to use for writing new text.
It's easy to think BERT is old technology nobody uses anymore, but actually it's still widely used wherever classification or search needs to be fast and lightweight.
7One-line summary
In shortBERT learned language by filling in blanks using clues from both sides, which makes it a model built for understanding meaning precisely rather than for generating new text.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02