Pretraining
Learning the basics before anyone decides what the job will be
- Pretraining is the stage where a model soaks up the basics from a huge pile of text before anyone has decided what it will be used for.
- Nobody labels the correct answers. The text itself already contains them, so the questions and answers are generated automatically.
- Spelling, sentence structure, and general knowledge about the world all get absorbed together, in one pass, during this stage.
- It takes a long time and costs a lot, so it's usually done once and reused in many places.
- However far the training material reaches is exactly where the model's picture of the world ends.
Contents
1The analogy
Getting a driver's license doesn't teach you "the route from your house to your office." What you learn at the test center works everywhere: how the wheel and pedals respond, what lane markings and road signs mean, what to do when a light changes.
While you're learning, nobody knows yet whether you'll be commuting, hauling cargo, or driving on country roads. Since nobody knows the destination, there's no specific route to memorize either. So the teaching aims at basics useful on any road at all.
Once licensed, people scatter down different roads. Driving works even in a neighborhood you've never seen, and anything that needs special know-how — delivery driving, say — gets picked up separately, later. Pretraining is that test center.
2In detail
It learns before the purpose is decided
The old way ran backward. To build a sentiment classifier, you'd gather a huge pile of reviews with correct labels attached and train only on that. To build a translator, you'd start over from scratch again. The destination got picked first, and only that one route got memorized.
Pretraining leaves the destination blank. It simply reads huge amounts of text — internet pages, books, encyclopedias — to pick up how language gets used, since what it'll be used for can be decided later.
Flipping that order is exactly why language models spread into so many fields so fast. There's no longer a need to start from scratch every time a new job comes up.
The workbook writes itself
Training that needs a human to label answers was expensive. Marking a hundred thousand reviews as good or bad one by one takes an enormous amount of human labor.
Pretraining skips that step. Hide part of a piece of text and have the model guess it, and the answer is already sitting right there in the original — grading comes free. Text that no human ever touched can go straight into training as-is.
That's what let the volume involved grow to a scale that was never possible before — enough text to sweep through that no single person could get through it in a lifetime.
Several layers settle in at the same time
It's a simple, repetitive exercise — guessing a blank — but several things settle into place inside it at once. At the bottom, spelling and letter patterns take shape: which letter tends to follow which.
On top of that, sentence structure settles in — how a subject and a verb pair up, how the pieces of a sentence connect. This is the point where an awkward sentence starts feeling awkward, without ever having read a grammar book.
At the very top, general knowledge about the world piles on. Filling in the blank in "Pizza is an ___ food" requires already knowing what pizza is. Getting good at filling blanks just happens to drag knowledge along with it.
It's slow and it's expensive
One run of pretraining takes weeks to months and eats an enormous amount of computing hardware. Changing direction partway through is hard, too — if a problem with the training material turns up after training is well underway, there's often no easy way back.
So this stage isn't something just anyone does. Instead, it's become standard to take a model that already finished pretraining and build on top of it — the same way it's easier to hire someone who already has a license than to run a whole new driving school.
What comes after getting the license
A model fresh out of pretraining alone continues text well but is clumsy at following instructions. Ask it a question and, instead of an answer, it might string out a series of similar-sounding questions instead. That's exactly what you'd expect from something that only ever practiced continuing text.
So a polishing stage follows. Examples pairing an instruction with a good answer teach it how to hold a conversation, and human choices between candidate answers shape its tone and manner. Only after this does it become the thing people actually use.
3More precisely
Pretraining trains on problems generated from the data itself, without a labeled answer key for any specific task. This approach is called self-supervised learning. Left-to-right models mostly use the task of guessing the next piece; understanding-focused models mostly use the task of reconstructing a hidden piece. Either way, no human-made answer key is required.
The analogy breaks down in a place worth naming. A driving test has a clean pass-or-fail line, but pretraining has no such finish line — it's closer to steadily shrinking the error until improvement levels off, and then stopping there. And a licensed driver keeps learning from new road conditions encountered afterward, but a model that's done with pretraining doesn't learn on its own. It knows nothing about anything that happened after its material was collected, and teaching it something new requires retraining it or handing it the relevant material each time. A test center also only ever produces one kind of license, while a single round of pretraining ends up serving as the shared starting point for many differently specialized models built on top of it later on, each one going off in its own direction.
4Try it yourself
5Common misconceptions
It's easy to think pretraining runs on data that people have labeled with correct answers, but actually the text itself serves as the answer key, so almost no human labeling goes into it.
It's easy to think a model is ready to converse the moment pretraining finishes, but actually it needs to go through a stage that teaches it to follow instructions before it becomes the thing people recognize.
It's easy to think a model keeps learning as it holds a conversation, but actually once pretraining ends, its contents are fixed, and nothing from a conversation gets absorbed back into it.
7One-line summary
In shortPretraining is the stage where a model fills in its basics in language and general knowledge from a huge pile of text, before anyone has decided what it will be used for.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02