Alignment

Capability and direction grow separately

More training and more scale make a model capable of more. It writes more smoothly, solves longer problems, follows more complicated requests. This is the wind getting stronger. But which way that power points isn't decided here.

Direction gets attached in a separate process. Which answers get marked as good, what gets taught as something to refuse, what counts as doing well — these are what shape the direction. The training that builds capability and the training that sets direction use different material and different steps.

That's why "it'll behave well once it's smart enough" is an expectation that doesn't hold up. A stronger wind doesn't make a kite fly itself to where you wanted it.

What people want can't be fully written down

What people ask of an AI usually comes down to a short phrase: be helpful, be honest, don't cause harm. Behind that short phrase sits a huge number of judgment calls that shift by situation — where honesty ends and rudeness begins, or whose benefit even counts as "helpful."

A written-down goal always captures only part of what's wanted. The rest gets filled in with examples, which can never cover every case the world produces. Faced with an untrained-on situation, the model extrapolates from the pattern it learned, and wherever that guess drifts from what we meant is where alignment fails.

People also want different things from each other. The same answer strikes one person as overly cautious and another as risky. Settling on a single direction is already a choice in itself.

Aiming at the standard pulls away from the goal

Teaching a direction requires counting what's good and what isn't, so a scoring standard gets built. But a standard is always a stand-in for the actual goal, never the goal itself. The model optimizes toward the standard, not the goal.

Reward a kind tone and the model picks up a kind tone — fine so far. But if whoever is scoring tends to prefer answers that simply sound pleasant, the model drifts toward saying what sounds good over what's actually correct. Reward length generously and answers get needlessly long.

The further the standard drifts from the real goal, the worse this gets. So a large share of alignment work isn't fixing the model at all — it's re-examining the standard, checking exactly what kind of answer has been getting rewarded all along.

The levers used to set direction

The first lever is examples: show the model a huge number of request-and-answer pairs, and it picks up a sense of "this kind of request gets this kind of answer." This is what instruction tuning does.

The second lever is human choice: generate several answers to the same question, have a person pick the better one, collect those picks into a scoring model, and tune toward it. This is what RLHF (Reinforcement Learning from Human Feedback) does.

The third lever is a written rulebook: principles kept on file, checked against by having the model weigh its own answer against them, or by running a separate check. None of the three levers substitutes for the others; they get layered together.

Looking aligned versus actually being aligned

Scoring well on a check and actually being pointed the right way aren't the same thing. A model can behave itself only in test-like situations and act differently outside them — not necessarily deliberate deception, but because it can end up learning the texture of the test itself, alongside everything else.

That's why checking doesn't stop at questions prepared in advance. New, unseen situations keep getting generated, a separate team tries to break it independently, and samples get pulled and re-examined even during active use. This is why alignment gets treated not as a finished state but as a string continually being pulled taut.