LoRA
Leaving the main body alone and training only a small add-on
- LoRA (Low-Rank Adaptation) locks the main body in place and trains only a small add-on sitting in front of it.
- What it learns is a tiny sliver of the whole, so training is light, and the result is a single file a few hundred times smaller than the main body.
- The add-on snaps on when needed and comes off when it isn't. One main body can swap between several add-ons, or layer more than one at once.
- It's the usual way to package a particular art style or character in image generation, and a common way to fit a language model to a company's tone.
- It can't stuff in something the main body never knew at all. It's closer to leaning what's already there in a certain direction.
Contents
1The analogy
An optician doesn't fix your eyes when your vision is off. The eyes stay exactly as they are — a thin lens gets held up in front of them instead. Sitting in the exam chair, you try lens after lens until the letters snap into focus, and glasses get made to that exact prescription. Nothing about how your eyes work has changed, yet what you see is entirely different. And that lens is small and light. Keep one pair for driving and another for reading, swap between them as the moment calls for it, and your eyes stay exactly the same the whole time. Don't need them — take them off, and that's the end of it. LoRA leaves a large main body untouched and holds a thin lens up in front of it instead.
2In detail
The main body stays locked
Retuning a whole model shifts every internal value a little. LoRA locks those values entirely instead. Not a single value in the main body changes while training runs.
Instead, a small add-on gets attached to each layer, and only that add-on gets trained. During a calculation, the value the add-on produces gets added onto the value the main body produces before passing along to the next step — the same order light follows passing through a lens on its way to the eye. The eye hasn't changed; it's just seeing through one extra layer before the light reaches it.
Why the add-on can stay so small
Here's the question this naturally raises: if the main body holds billions of values, what can an add-on with only a few million possibly accomplish?
The key is that what needs to change isn't the values themselves — it's how much they need to shift. The shift needed to fit a certain tone or art style tends to be simple in structure. It boils down to a handful of directions, with everything else sitting close to zero. Something with a simple structure like that can be captured with nothing more than two thin layers stacked together. That's why shrinking the add-on down to a few hundredth of the original size still captures nearly all of the needed shift.
How thin to make it is a setting a person chooses. Thinner means a smaller, faster file that can hold less of a shift; thicker means the reverse.
Swapping and stacking
Retuning the whole model leaves behind a brand-new model the same size as the main body, every single time. Ten uses means ten large files piling up. LoRA leaves you with one main body and ten small add-on files. Storage space and the hassle of moving files around are a different order of magnitude.
It's flexible to use, too. Swap in one art style now, another later, or layer two add-ons together and blend them. You can even dial how strongly each one gets applied. Downloading a style file for an image generation tool and dropping it in is exactly this.
What "light" actually means
Training gets easier on several fronts at once. Fewer values to correct means less intermediate data to hold onto during training too. So models that would be out of reach to retune fully become workable to adapt on personal hardware.
The result stays light as well. If the main body runs tens of gigabytes, the add-on commonly drops to tens of megabytes. That lightness is exactly what made a whole culture of sharing style files possible.
What it can't do
A lens only shapes the light your eyes already receive. It can't make visible something that never reached your eyes in the first place. LoRA is the same. It's a poor fit for stuffing in a field the main body has no knowledge of at all.
There's a limit when the shift needed is very large, too. The kind of shift it can hold has a fixed shape, so pulling the main body toward a character entirely different from what it already knows still calls for retuning the whole thing. When there's plenty of data and every last bit of performance matters, full retuning remains the choice.
3More precisely
LoRA represents the shift to be added to a layer's weights as the product of two small matrices, and trains only those two matrices, instead of updating the weights directly. The name comes from the observation that this shift can be captured well enough at low rank. The value that sets how thin to make it is called the rank, and there's a separate value that sets how strongly it gets applied.
The analogy breaks down in places. A pair of glasses sits as one lens in front of the eyes, but LoRA attaches in several places, one per layer, and which spots to attach to is itself a choice to make. There's another gap: take glasses off and you're back to normal, but a trained add-on can be merged straight into the main body. Merging it in speeds up calculation a little, at the cost of losing the ability to swap it in and out. And an add-on has to match its main body — drop it onto a different one, and it either goes strangely off, or doesn't attach at all.
4Try it yourself
5Common misconceptions
It's easy to think LoRA is something entirely separate from fine-tuning, but actually it's one branch among several ways of adapting a model to a purpose — the branch that leaves the main body untouched.
It's easy to think one add-on file works anywhere, but actually it has to match the main body it was built on to attach properly.
It's easy to think being light means performance drops to match, but actually for matching tone or art style, it often lands close to what full retuning achieves.
7One-line summary
In shortLoRA locks a large main body in place and trains just one thin add-on layer in front of it, so it can be swapped in and taken off whenever needed.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02