A/B Test
A test that shows two versions to real people and compares results
- A/B testing means putting two versions in front of real people and comparing what actually happens.
- The split is everything. Assign people at random, and the only difference left between the two groups is the one thing being tested.
- What gets measured, and for how long, is decided before the test starts. Pick the winning number afterward and it stops being a test.
- It settles what gut feelings and meeting-room arguments cannot, by looking at real usage.
- A small difference can't be told apart from chance. That's why enough people and enough time matter.
Contents
1The analogy
A shoe factory wants to try a new rubber for its soles. Whether the new rubber really wears down slower isn't something you can tell by pinching a sample in the office. So a hundred pairs go out with the old sole, and a hundred more with the new one, and everyone wears their pair for a month. A month later the shoes come back, and a ruler measures how much rubber is gone.
The split matters most. Hand the new soles only to people who hike every weekend, and the new sole will look more worn no matter how good it is. So who gets which sole is decided the way you'd draw straws, at random, and everyone wears theirs for the same length of time. The only difference left between the two groups is the sole itself.
That is what an A/B test is. Put two versions side by side, let people actually use them, and compare the results on whatever was decided beforehand.
2In detail
Randomness is what keeps the test honest
How you split the two groups is close to the whole test. Hand office workers the old soles and delivery riders the new ones, and a month later there's no way to tell whether the difference came from the rubber or from how much walking each group did.
So the split leaves human judgment out entirely. Number people and split by odd and even, or draw lots, whichever way, which group someone lands in has nothing to do with anything about them. That way the hard walkers and the easy walkers end up mixed evenly into both groups.
Everything else gets matched as closely as possible: the same length of time, the same way of handing out shoes, the same ruler doing the measuring. Leaving exactly one thing different is the whole skeleton of the method.
Decide what you're measuring before you start
If sole wear is what you set out to measure, measure sole wear. If the difference looks unimpressive a month later and someone suddenly brings up how frayed the laces got, the test falls apart. Measure a pile of things and pick out whichever looks good afterward, and a number that turned out well by chance gets mistaken for a real difference.
So before the test starts, write down in a sentence what's being measured and what counts as a win. Pick one main number to watch, and add a few more that would catch any damage the main number's improvement might quietly be causing.
Checking in too often during the test is worth watching for too. Day-to-day numbers bounce around, and stopping on a day you happen to like turns the result into that day's luck, not a test.
How many pairs, and for how long
Testing ten pairs isn't enough to say the new sole is better. Walking habits vary so much from person to person that within ten people, that variation buries whatever difference the rubber made. The smaller the difference you're trying to measure, the more people and the more time it takes.
The time needs to be generous too. People tread carefully in the first few days simply because the shoes are new, so old habits don't show up yet. Wait until the novelty wears off before measuring, and the numbers reflect ordinary use.
Season and day of the week matter as well. Test only during the rainy season and all you learn is that season's result. That's why both groups run during the same stretch of time; different timing means comparing weather, not soles.
How AI services actually use it
This method decides things like whether to switch to a new model, how to word a prompt, or how long an answer should run. Incoming users get split randomly in half, one side sees the current version, the other sees the new one, and the numbers decided in advance get compared.
It earns its keep especially because lab scores and real usage often disagree. A new approach can lead on a prepared test set, and yet people stick with the old one longer once it's actually in front of them. What people actually experience isn't in the test set.
Showing two answers side by side and asking which is better is a common form of it too. Hide which answer came from which source, and the choices that pile up stop being swayed by a name.
3More precisely
An A/B test is a controlled experiment: subjects get randomly assigned to two groups, exactly one condition differs between them, and the outcomes get compared. Random assignment means the groups' other traits average out, which is what lets the difference in results be attributed to the condition that changed. Whether a difference is large enough to not be chance gets checked statistically, and the number of people needed is usually worked out before the test starts.
The analogy breaks in one place. A sole doesn't change how the person wearing it feels, but with a service, one group's experience can spill into the other. Two people show each other their screens, or post reviews, and the groups blend together. That's why the unit being split sometimes has to be a region or a time slot instead of a person.
A test doesn't always hand over the right answer either. Something that looks good over a short stretch often turns bad if left running. A flashier prompt lifts immediate engagement but drives more people away months later. That's why short-window numbers and long-window numbers get watched together.
4Try it yourself
5Common misconceptions
It's easy to think an A/B test tells you which version is better, but actually it only tells you which one did better on the number you picked, during this particular run.
It's easy to assume a difference that shows up is a real one, but actually with too few people or too little time, it can't be told apart from a difference that happened by chance.
It's easy to think any random-looking split will do, but actually the moment a personal trait sneaks into how the split is made, there's no way to tell what caused the difference.
7One-line summary
In shortAn A/B test changes exactly one thing between two real-world versions and compares the results on a measure decided in advance, and if the split goes wrong, the result goes wrong with it.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02