Q-Learning
Writing down an expected score for every action and refining it
- Q-learning builds a table with an expected score for every situation and every action, and keeps refining those numbers.
- A single number in the table means "pick this action in this situation, and here's roughly what you'll earn from here on out."
- Updating a number means taking the score just received, adding the best number in the row for the next situation, and blending that into the old number a little at a time.
- Once the table is filled in, the policy falls right out of it. Just pick whichever square holds the biggest number for the situation you're in.
- Once the number of situations and actions grows large, a table can't keep up. Having a neural network remember that table instead is what's called DQN, short for Deep Q-Network.
Contents
1The analogy
A scheduling board hangs on the wall of a movie theater's office. Time slots run down the side, genres of film run across the top, and every square has a number penciled in. The seven o'clock row has an 8 in the action square and a 5 in the family square; the two o'clock row has the order flipped around. These numbers are guesses at how well that film would do at that time slot. Once yesterday's actual attendance comes in, the number in that square gets erased and rewritten a little higher or a little lower. After a few months the board gets pretty trustworthy, and scheduling becomes simple — just pick the biggest number in that time slot's row.
2In detail
What one square means
A single square in the table is where one situation meets one action. The number written there isn't the score you'd get right away — it's a guess at the total score you'll end up earning if you take this action and follow through to the end. That's why a square can carry a high number even when the action costs something right now, as long as it pays off later.
Every square starts out at zero, or filled with some arbitrary number. Starting from a table that's completely wrong is fine. Little by little, adjusting each square based on the score actually received, the numbers settle into their proper place. Once the table has settled, there's no separate policy left to learn — picking the biggest number in the current row is the policy.
How the numbers get corrected
Every time an action is taken, four pieces of information land in your hands: what situation it was, what got picked, how much score came back, and what situation it led to. These four correct the number in the square that was just picked.
First, a new guess gets built: the score just received, plus the biggest number in the row for the situation it led to, shaved down a little. Then that new guess gets blended with the number that was already written there. It's blended a little at a time rather than swapped wholesale, because a single result always has some luck mixed into it. How much gets blended in each step is called the learning rate.
Why pull in the best number from the next row
Picture a problem where the score only comes at the very end. At first, only the very last square's number goes up. But correcting the square right before it means the situation it led to is the row containing that last square. The best number in that row gets pulled in, so the number in the earlier square rises too. Good news spreads backward this way, one square at a time, all the way from the end.
There's an important property hiding in this. What gets pulled in is the best number in the next row, not whatever was actually picked next. So even trying out plenty of random, off-track actions during training doesn't stop the table from still growing toward the best policy. That's how it manages to learn the best path while wandering freely.
The table has to be tried to get filled in
A square that's never once been picked keeps its starting value forever. Always picking whichever square looks biggest means only whatever happened to start out high keeps getting picked, and the better square sitting right next to it never gets checked at all. So during training, some squares get picked at random on purpose, now and then.
A common approach starts with a high rate of picking squares at random, then lowers it as the table settles. Enough wandering has to happen for the table to fill in evenly, and only then can the biggest number chosen at the end actually be trusted.
When the table gets too big to handle
The limit of Q-learning is the size of the table. The number of squares grows as the number of situations multiplies by the number of actions. For a twenty-square grid with four directions to pick from, eighty squares is manageable — but treat the raw pixels of a screen as the situation, and the square count grows past counting.
On top of that, every square stands alone. However similar a neighboring square's situation might be, nothing learned there carries over. Between these two problems, the table approach collapses the moment a situation gets even a little complicated. So instead of writing down one square at a time, a neural network took over — one that, given a situation, guesses the number for each action on its own. That's DQN, short for Deep Q-Network.
3More precisely
Q-learning learns an action value for every state-action pair. The update takes the difference between the current value and a target built from the reward received plus the discounted best action value at the next state, and applies a fraction of that difference, set by the learning rate. Because the target is built using the best possible value rather than the action actually taken next, it's classed as an off-policy method. It's been proven that visiting every pair often enough, while shrinking the learning rate at the right pace, makes the table converge to the optimal action values.
The analogy breaks down in a few places. A scheduling board is built for a person to glance at, but a Q-table's square count quickly grows past anything a person could scan by eye. A scheduling board's number also only carries that one time slot's outcome, while a Q-value has every result that follows, discounted, folded into it as well. And a theater just writes down last season's real numbers, but Q-learning corrects itself based on its own guess, one that hasn't been confirmed yet — which is why its early numbers stay considerably off for a while.
4Try it yourself
5Common misconceptions
It's easy to think a Q-value means the score you're about to receive, but actually it's a guess at the total, discounted sum of everything that follows that action too.
It's easy to think the table has to be complete before it can be used, but actually it gets consulted and corrected at the same time, all the way through the process of filling it in.
It's easy to think Q-learning and DQN are different methods, but actually DQN is the same update rule, just with a neural network standing in for the table.
7One-line summary
In shortQ-learning writes down an expected score for every situation and every action, then refines those numbers a little at a time based on what actually happens when it tries them.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02