Model Architecture

Three things a blueprint spells out

The first is the number of layers — a shallow handful, or dozens or hundreds stacked deep. Deeper stacks can hold more complicated rules, but training gets trickier as they grow.

The second is what goes inside each layer. There are parts with different jobs: one that scans small patches of an image, one that lets pieces of a sentence look at each other, one that refines a value before passing it along. The third is how they connect — straight up one after another, with shortcuts that skip several layers, or split off sideways into two branches.

The blueprint and the furniture are different things

The two things people mix up most when talking about models are architecture and weights. Architecture is the empty skeleton; weights are the numbers training fills it with. A skeleton alone can't produce any answer, and weights alone don't tell you where they belong.

When you download a model published online, what's usually inside the file is mostly the weights. The architecture is written separately, in the code or configuration that defines the model. The two have to match exactly for the model to come alive — like furniture arriving on a truck with no blueprint, you'd need that drawing to know which room each piece belongs in.

The shape of the data decides the architecture

In an image, neighboring dots tend to be closely related, so an architecture that slides a small square window across works well. Sentences are different — the first word and the last word can connect directly, so an architecture where far-apart pieces can look straight at each other suits them better.

Sound cares about the order things happen in, and numbers arranged in a table often get by with something quite simple. Choosing an architecture is really telling the model, ahead of time, what kind of data it's dealing with. Pick one that fits well, and the same data produces good results with far less training.

What a designer has to weigh

Deep versus wide is the first fork in the road. Going deep means passing through many steps to capture complicated rules; going wide means handling more information at once within a single layer. At the same overall size, which is better depends entirely on the problem.

Performance isn't the only concern. How fast an answer needs to come back, whether it fits in graphics memory, how expensive training will be — all of that comes bundled in. A model meant to run on a phone starts from a different blueprint than one meant to run in a data center.

That an architecture change wipes out what was learned is also a real burden. The moment a wall moves, the furniture already placed no longer fits. That's why it's common to reuse a proven blueprint and just scale it up or down, rather than draw a new one. Bringing out a brand-new architecture needs a clear enough advantage to justify it.