Exploration vs Exploitation
The balance between using what you know and trying something new
- Exploration is trying the side you don't know yet; exploitation is using whichever side has worked best so far.
- Only one can be chosen at a time. The moment you try something new, you give up using whatever you already know is good.
- Exploit only, and you get stuck on whatever happened to work early on. Explore only, and you never once get to use what you've found.
- The common fix is picking randomly on purpose now and then, with that rate set high at the start and gradually lowered.
- Giving an unfairly generous value to the side that's been tried less, so it naturally gets picked more often, is a widely used method too.
Contents
1The analogy
Settle into a spot at a fishing hole, and after an hour or so you start to notice which spot your float keeps bobbing at. Keep casting there, and that's the best choice you know of for today. But a patch of reeds a bit further off has never once had a line in it. It might be much better, or it might be a complete waste of time. There's no way to know without casting there. The catch is that casting toward the new spot means leaving the known spot empty the whole time. Time is limited, and there's only one line. Deciding how to split your casts between the spot you know and the spot you don't is the balancing act of exploration and exploitation.
2In detail
Only one can be chosen at a time
What makes this problem hard is that both things can't be done at once. Every cast toward a new spot is a cast that didn't go toward the spot you already know. So trying something new always comes with a cost. What you buy with that cost is information — you find out whether that spot is good or bad, and that finding changes the decision that comes next.
This isn't a problem unique to reinforcement learning either. Sticking with the usual route or trying a new one, keeping a product on the shelf because it sells or bringing in something new — they all share the same shape. In reinforcement learning, though, this balancing act repeats tens of thousands of times, so how it's split ends up deciding whether the whole training run succeeds.
Lean too far either way
Start with pure exploitation. Land a couple of fish by chance at the first few spots you try, and that spot starts to look like the best one there is. From then on, every cast goes there, and no other spot ever gets checked. A better spot right next door can go undiscovered the whole time. Whatever luck showed up first ends up deciding the entire day.
Pure exploration runs into trouble too. Cast toward a new spot every single time, and information about spots piles up fast, but no time ever actually goes toward the spot that's good. In training, that's the same as moving purely at random — a score is barely ever earned, so there's almost nothing solid to learn from either.
Picking randomly, on purpose, now and then
The most widely used method is simple. Nine times out of ten, go with whichever has worked best so far; one time out of ten, pick at random. All that needs setting is the rate of picking randomly, which makes it easy to use, and it works, at least a little, on just about any problem.
That rate doesn't stay fixed — it gets lowered over time. Early on, with nothing known yet, close to half the picks might go to random choices; as more rounds pass and the values settle, that rate gets brought down bit by bit. By the end, maybe only one pick in a hundred goes toward trying something new. The flow is wide exploration at first, narrowing down as it goes.
Giving an unfairly generous value to the untested side
Picking at random has a flaw: it casts toward a spot already confirmed to be bad with the exact same odds as anywhere else. So a better method came along. When scoring each spot's value, a bonus gets tacked on for the spot that's been tried less — the fewer the tries, the bigger the bonus.
This naturally pulls the less-confirmed spots up toward the front, where they get picked. The bonus shrinks the more a spot gets tried, so a bad spot naturally falls back on its own, while a spot that might be good but hasn't been confirmed yet keeps rising to the top. It's a far more efficient way to gather information than casting at random.
Once training is finished
Once training is done and the policy is actually put to work, exploration usually gets switched off. Only whichever choice was judged best gets picked from there on — there's no reason to deliberately pick a worse option out in the real world. That said, in places where conditions shift with the seasons the way the water does, a sliver of exploration gets kept even then, because there's no guarantee yesterday's best is still today's best.
3More precisely
The balance between exploration and exploitation is a problem shared by reinforcement learning and the bandit problem. Picking a random action with a set probability is called the epsilon-greedy method, and lowering that probability as training runs is called decay. Adding a bonus for an under-tried action proportional to its uncertainty is called an upper confidence bound method, and treating each action's value as a probability distribution and sampling from it for comparison is called Thompson sampling. The gap that piles up between what was learned and the best choice missed along the way is called regret, and a good method keeps that value growing slowly.
The analogy breaks down in a few places. At a fishing hole, moving to a new spot leaves the old one sitting there unchanged, but in reinforcement learning, a chosen action often changes the very next situation itself, with no way back. So the cost of exploring doesn't end with that one loss — it can echo forward for a long time. And a fishing hole's spots can be counted by eye, while in problems worked out from raw screen images, the situations never yet visited are effectively endless, so picking at random alone can't cover the ground.
4Try it yourself
5Common misconceptions
It's easy to think exploration is only needed early in training, but actually where conditions keep shifting, a little needs to be kept going right through to the end.
It's easy to think trying more things at random always leads to better learning, but actually with no experience using the good option, its value never settles into place.
It's easy to think exploration and exploitation is a matter of picking one or the other, but actually it's a question of how to shift the balance between them over time.
7One-line summary
In shortExploration versus exploitation is the balancing act between casting where you know and casting where you don't, and the usual approach is to start wide and narrow it down as you go.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02