Reinforcement Learning

All that comes back is a score

Reinforcement learning has three players: the one acting, the stage that receives the action, and the score the stage hands back. In bowling, that's the bowler, the lane and pins, and the number of pins knocked down.

In supervised learning, every roll would need an answer attached — "that stance was correct this time." Reinforcement learning has no such answer. Only a single score comes back, and stance, angle, power, and starting position are all bundled together inside it. Figuring out which part of that raised the score is itself part of what has to be learned.

How the scoreboard is written decides the whole outcome. Score only on pins knocked down, and a habit of throwing as hard as possible sets in. Dock heavily every time it hits the gutter, and a cautious throwing style sets in instead. The scoreboard is the goal.

The score can arrive a long time later

A bowling frame is two balls. Where the first ball lands decides whether the second ball can pick up a spare. Leave only a corner pin standing with the first ball, and there's almost nothing the second ball can do about it. The first ball's score, eight, wasn't bad on its own — but looking at the whole frame, it was a costly choice.

The problems reinforcement learning deals with tend to have this same shape. The value of a choice made now often only shows up several steps later. That's why the choice gets made by weighing not just the immediate score but the score expected further ahead too. Future score gets counted at a slight discount, which is what lets it be weighed against a score that's certain right now.

Whether to stick with what's worked, or try something new

Find a stance that reliably knocks down seven pins, and it pays to keep rolling from there. But if a spot two steps over would land a strike, sticking to the known stance means you never find out. Roll from a brand-new spot every single time instead, and the score never stops bouncing around.

This balancing act is called exploration and exploitation. Early on, learning leans heavily toward trying new things; as it gets more experience, it shifts toward whatever has worked well. Tuning that balance without tipping too far either way is the part of reinforcement learning that takes the most care.

What settles in the body becomes the policy

After dozens of games, what's left behind isn't a record of scores — it's a rule: "in this situation, roll it this way." Which spot to stand on when a right-side pin is left standing, how much power to use when the lane is slick, things like that. This bundle of rules is called a policy.

Once the learning process ends, this policy is what actually gets used. Feed it a situation and out comes a move. A well-tuned policy can still roll reasonably on a lane it's never faced before, but if that lane is too different from what it trained on, it ends up in the gutter again, just like the very first ball.

In practice, it rolls hundreds of thousands of times

A person can pick up the feel of the game in a few rounds, but a machine can't. A single score carries so little information that reaching a workable policy takes hundreds of thousands, sometimes hundreds of millions, of attempts.

So most training happens inside a simulated stage rather than the real world. Inside a simulation, millions of games can run in a single day, with no risk of a ball ever actually breaking. The catch is that even a small gap between the simulation and reality can leave a learned trick useless. Closing that gap remains one of the biggest open problems in robotics.