Prompt Adherence

Well-made and as-requested are different yardsticks

Looking at a generated result, a person notices two things at once. One is craft: are the lines clean, do the colors look natural, are there no odd-looking fingers. The other is adherence: is what was written actually there.

The two move independently. A gorgeous landscape can turn up without the lighthouse that was actually requested, and a rough-looking picture can include every single element that was asked for. Picking a tool by craft alone is exactly why people end up rewriting the same request over and over while using it.

That's why evaluations that compare tools score the two yardsticks separately. Roll them into one combined number and there's no way to tell which side is falling short.

Counting by hand

The most reliable method is a person checking directly. It starts by breaking the request into items: a red umbrella, three people, a rainy night street — pieces that can each be checked. Then, looking at the result, each item gets marked present or not, and the fraction that made it in becomes the score.

Placing two results side by side and picking whichever is closer to the request is also common. Picking one of two wobbles less from person to person than assigning a number does.

The catch is that it's slow and expensive. Standards drift a bit from person to person too, so a single result is usually shown to several people, with how much they disagree recorded alongside the score.

Automated grading

Grading can also be handed to another model instead of a person. A common approach shows the result to a model that describes pictures in words, then checks how closely that description overlaps with the original request. Another approach asks a model that measures pictures and text on the same scale how close the two are.

Lately, a popular method breaks the request into yes-or-no questions and asks them one at a time: is the umbrella red, are there three people. The fraction of yes answers becomes the score — the same item-counting a person would do, just handed over to a machine.

Automated grading is cheap and can get through thousands of pictures in a day. In exchange, whatever the grading model can't see, it never catches — mangled lettering on a sign or an odd shadow can slip right past. So the usual approach is a rough automated pass first, with a person double-checking a sample afterward.

The fields that miss most often

The spots where adherence tends to break down are fairly consistent. First is count: ask for three and two or four often show up instead. Second is spatial relationships: words like left, above, or behind, which set where things sit relative to each other, tend not to hold.

Third is attributes bleeding together: ask for a red hat and a blue bag, and the hat can come out blue with the bag red. Fourth is text: lettering on a sign or a label tends to come out as a similar-looking but meaningless pattern. Fifth is exclusion: ask for something to be left out, and it often shows up anyway.

What gets lost when adherence goes up

There's a dial that pushes the result to follow the request more forcefully. Turn it up and the requested elements do get included, but colors blow out and the result turns stiff. Turn it down and things look natural and varied, but the request gets followed loosely.

Rather than cranking that dial all the way to nail one perfect result, it's often better to set it at a reasonable point, generate several pictures, and pick whichever lands closest to the request. Adherence is a value best judged as an average across several pictures, not a single result's grade.