Depth vs Width
How many layers to stack versus how wide to make each one
- Depth is how many layers a signal passes through; width is how many computing units sit in one layer.
- Depth is processed one after another; width is processed all at once. That difference is what splits speed and hardware needs apart.
- Stacking deeper lets a model build new judgments on top of earlier results. Simple features grow into complex concepts.
- Making it wider lets a model look at more things at the same time within one stage. It has more room to hold information.
- Pushing only one of the two too far is a bad trade. Real models grow both together, keeping a reasonable ratio.
Contents
1The analogy
Boxes arriving at a delivery warehouse pass through several stages. First they're sorted by region, then that split gets sorted again by neighborhood, and finally sorted once more by delivery zone. How many stages there are is depth.
Look inside one stage and you'll find several sorting lanes running side by side. Ten lanes means ten boxes get handled at the same time. How many lanes sit in one stage is width.
Warehouses of the same overall size can still be built differently. One might run six stages with few lanes each; another might cut it to three stages with twice the lanes. The character of the path a box travels ends up completely different, even when the two warehouses cost about the same to build and staff.
2In detail
What stacking deeper gets you
One layer, on its own, does something fairly modest — it mixes up the incoming values a little differently and passes them on. But stack that modest step over and over, and things change. The first layer finds a simple signal, the second layer builds a new signal on top of it, and the third layer builds on top of that again.
Models that work with images show this staircase clearly. Early layers pick out edges where brightness suddenly changes. Middle layers gather those edges into patterns. Later layers gather patterns into recognizable objects. Make the network shallow, and the chance to climb that staircase disappears entirely.
Depth comes with a cost, though. Because layers have to be passed through in order, doubling the number of layers roughly doubles how long an answer takes to arrive. Going deeper also makes it harder for a training signal to reach the earliest layers. It took skip connections and normalization devices before truly deep models could actually be trained at all.
What making it wider gets you
Widening means laying down more lanes within one stage. It lets the same stage look at more different features at once. More room to hold things also means more capacity to memorize.
Width's advantage is speed. The calculations within one layer don't have to wait on each other, so it fits well with hardware built to run thousands of operations at once. Unlike adding layers, widening doesn't stretch out how long an answer takes to appear.
But widening is expensive. Double the width, and the amount of numbers a single layer holds roughly quadruples. Keep widening on its own, and you gain more storage without gaining any new stage of judgment.
Given the same overall size, which is better?
There's been steady experimentation into how to split a fixed total size between depth and width. The result tends to be this: swinging to either extreme is a loss, but within a reasonable range, overall size matters far more than the ratio.
So real designs rarely stray far from ratios that have already been validated. As a model gets scaled up, depth and width usually grow together, adjusted just enough to keep training stable.
There's one thing worth remembering as a reader: you can't compare models by layer count alone. A model with fewer layers can still be the bigger one, if it's wide enough.
Where speed and hardware needs diverge
If cutting down the wait for an answer is the priority, depth is what to look at first. Layers have to be passed through one at a time in order, and no amount of better hardware can shrink that number of steps.
If memory is the tight constraint instead, width is what to look at first. A wide layer holds a lot of numbers at once, which eats into graphics memory fast.
So at the same overall size, a deep-and-narrow model is at a disadvantage for wait time, and a shallow-and-wide model is at a disadvantage for memory. Neither is simply better — what the model is meant to be used for decides the answer.
3More precisely
Depth refers to the number of layers, width to the number of computing units in one layer. In language models, the number of blocks is usually called depth, and the number of values each piece carries is usually called width. The amount of numbers held in one layer grows roughly with the square of the width and in proportion to the number of layers. Double the width and you get roughly four times the numbers; double the depth and you get roughly double.
The analogy breaks down in a few places. Lanes in a warehouse all do the same job, but the units within one layer of a network each look at a different feature. A box also keeps the same shape as it moves through stages, but the bundle of numbers passing through a network gets entirely replaced with new values at every layer. And where a warehouse only ever gets slower as stages are added, a network's added accuracy from going deeper often pays back the extra time. Design ends up being a problem of tuning both axes together, always.
4Try it yourself
5Common misconceptions
It's easy to think deeper always means better performance, but actually if width doesn't keep pace, adding depth alone buys you very little.
It's easy to assume layer count is the same thing as model size, but actually a model with fewer layers can still be far bigger if it's wide enough.
It's easy to think widening is cheaper than adding a layer, but actually doubling the width roughly quadruples the number of values involved.
7One-line summary
In shortDepth is the number of layers a signal passes through in sequence, width is how wide each one is, and a well-built model grows both together instead of pushing just one.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02