Sampling Bias
A skew in data caused by who got picked
- Sampling bias happens when the way a portion gets picked from a whole makes that portion look different from the whole.
- The picked data itself isn't wrong. What goes wrong is the moment that result gets given the name of the whole.
- It doesn't fade with more volume. Picking more from the same spot just piles up the same skew, bigger.
- The part left out leaves no trace in the data at all. Staring at the table alone won't reveal what's missing.
- When the data is about people, the skew lands as a cost on whichever group is underrepresented, and the gap widens over time.
Contents
1The analogy
Picture wanting to know how well a bag of seeds sprouts. Planting all hundred isn't practical, so twenty get pulled out, laid on a damp cloth, and watched for a few days. Eighteen sprout, so the bag gets marked down as a good one.
The question is how those twenty got picked. Reach in from the top without shaking the bag first, and only the plump, well-formed seeds get grabbed. The small, shriveled ones have settled to the bottom and never get touched. Of course twenty seeds picked that way sprout well — it isn't that the bag is good, it's that the hand only grabbed the good part of it.
Sampling bias is that hand. Wanting to know about the whole, only a portion gets picked, and the way it gets picked makes that portion look different from the whole.
2In detail
Where the hand reaches in decides the result
Reach in from the top only, and only the big ones come out. Dig around just one corner of the bag, and only that corner's seeds come out. Collecting data works the same way. Take opinions only from inside an app, and only the opinions of people who use that app pile up. Call only during weekday afternoons, and only the answers of people free to pick up a call at that hour get collected.
The data that comes out of this isn't lying. Those twenty seeds really did sprout well. What goes wrong is the label attached to the result. The moment "this is a good bag" gets written down, it's speaking for the eighty seeds that never got touched too.
Piling up more data doesn't thin this skew out either. Grow twenty seeds into two hundred from the same spot and it's still the top of the bag. The skew isn't a matter of count — it's a matter of method.
What's missing leaves no trace
The hardest part is that the unpicked side leaves no mark at all. A seed sitting at the bottom of the bag never makes it onto the cloth, so it never even gets counted. Looking only at the data that got collected, there's no way to tell where the gap is — staring at the table won't reveal an empty seat.
That's why the skew shows up when the collection method gets examined, not when the data itself gets examined. Without a record of where, when, and through which channel the data was gathered, there's no way to trace it back later. Keeping a record of the collection process matters just as much as checking the data itself.
A model learns from only the picked side too, and then answers without hesitation about the side it has never once seen.
When it's about people, the cost comes back around
With seeds, buying a new bag settles it. When data about people is skewed, the consequences land on people. Whichever group is thin in the data gets judged more clumsily, and that clumsy judgment gets used exactly where it hurts — in resume screening, in automatic routing for support.
The feedback loop makes it worse. A group the system handles poorly finds the service less useful and uses it less; using it less means even less data piles up about them; less data means it keeps getting handled poorly. A small early skew widens as time goes on.
That's why a strong overall score isn't reassuring on its own. A single number that lumps everyone together gets pulled toward whichever group makes up the bulk of the data, which hides just how badly the smaller group is doing. Splitting the score out by group is the only way that gap becomes visible.
Ways to cut down the skew
The first step is shaking the bag. Picking evenly from anywhere in it, with no favored spot, is the simplest and sturdiest approach. Dividing the bag into top, middle, and bottom ahead of time and pulling a set amount from each section works too — that way no section gets skipped entirely.
Not relying on a single channel helps as well. Running an app, a phone line, and a paper form together means whoever one channel misses, another one catches. Whatever skew is left after that can be corrected afterward by weighting — counting the underrepresented group's data more heavily.
Whatever gets done, the first step is the same: writing a sentence describing who this data includes and who it leaves out. Putting it into words is what finally makes the missing side visible.
3More precisely
Sampling bias is the skew that appears when the chance of being picked differs from one member of the whole to another, during the process of selecting a portion. When a whole group never had any chance of being picked at all, that's called a coverage problem, treated as its own category. When someone was picked but never responded, leaving only the responders in the data, that's called nonresponse. Survivor bias — drawing conclusions about a whole from only the survivors — comes from the same root.
The analogy has a limit. A bag of seeds can be fully emptied out and counted if someone really wants to. In real data, nobody usually knows what the whole actually looks like. With no ground truth to compare against, the size of the skew can't be measured exactly — it can only be estimated, and only for the categories, like age or region, where a trustworthy reference number happens to exist.
It's worth telling apart from the broader term data bias, too. Sampling bias is one branch of that, specifically the kind that arises at the picking stage. A labeler's own leanings, or a bias already present in the world simply carrying over into the data, are separate branches of the same tree.
4Try it yourself
5Common misconceptions
It's easy to think collecting more data thins out the skew, but actually pulling more from the same spot in the same way just makes the same skew more solid.
It's easy to think a skewed dataset means the collected data is wrong, but actually the collected data can be perfectly accurate — the problem is reading that result as the whole story.
It's easy to think a careful look at the table will reveal the skew, but actually the missing side isn't in the table at all, so it only shows up by retracing how the data was collected.
7One-line summary
In shortSampling bias is the skew created by whichever hand did the picking, and the moment that picked portion gets given the name of the whole, the part left out disappears entirely.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02