What is Hand Tracking? — one analogy

1The analogy

Everyone's had to check out a room with just a flashlight during a blackout. The circle of light only covers a palm's width, and if whatever you want to keep an eye on keeps moving, the flashlight has to keep swinging to follow it — watching where the light sits now while nudging the wrist a little ahead, guessing where it'll go next.

The trick is not sweeping the whole room every time. Checking just around where the light already sits keeps the arm relaxed and the response quick. But the trick stops working the moment the target ducks behind furniture or out the door — then the beam has to widen and sweep the whole room again from the start.

Snapping the flashlight around fast doesn't help either. A jerky circle of light is hard to read, so it's better to ease it along smoothly even if that's a touch slower. Hand tracking is that flashlight work.

2In detail

It handles a flow, not a single frame

Marking hand-joint points in one photo and following a hand through video aim at different questions. The first answers "where is it right now," and the second answers "where did that hand from before end up now." The second one only works by remembering the previous frame's result.

One hand commonly uses twenty-one spots: one wrist, plus four joints per finger from base to tip across all five fingers. These twenty-one points get marked fresh every frame, then linked to the previous frame's points to build a trail.

The numbering stays fixed. Whichever point is the thumb tip is always the same one, so today's thumb tip and this frame's thumb tip can be compared directly. That subtraction is what produces something like "the thumb moved up by this much."

Knowing the last frame narrows the search

Searching the whole screen for a hand is expensive. Knowing the box the hand was in last time, though, means the next frame only needs to check a slightly widened version of that same box — the hand is almost always still inside it. A human hand doesn't teleport to the opposite side of the screen between frames.

So real tools split the work into two steps. The heavy step, finding the hand for the first time, runs only occasionally; the light step, marking joint points, runs every frame. While tracking is going well, the heavy step can be skipped entirely.

That structure is what lets a web browser or a phone, without much computing power to spare, process dozens of frames per second. It's why demo tools take a beat to first recognize a hand and then run smoothly after that.

Losing it means starting the search over

Tracking can break at any time. A hand leaving the screen, getting covered by the body, moving too fast to leave more than a blur, or overlapping another hand — the trail snaps in any of these moments. What's needed then is a way to notice the break.

The way to notice is the confidence value. When the values across every joint point drop all at once, it means there's no hand where it's currently looking. That's the cue to drop the narrowed search area and go back to scanning the whole screen.

Skipping that reset causes strange behavior — the system keeps combing near the last known spot even after the hand has already left, and a point ends up stuck to a wallpaper pattern. Reset too eagerly, on the other hand, and every brief blur sends it back to searching from scratch, stuttering the display.

Smoothing the jitter adds lag

Even a hand held perfectly still gets points that shake a little. Each frame gets computed separately, so a value drifts a few pixels one way or another, and that drift reads as jitter on screen. For a feature that draws with a fingertip, the line comes out messy.

So the previous frame's location and the current one get blended together to smooth the motion. Weight the previous frame more and the display gets very smooth, but a fast hand movement makes the point feel like it's lagging behind. Weight the current frame more and the response is quick, but jitter stays.

Which way to lean depends on the use. A drawing feature leans smooth; a swipe-to-navigate gesture leans fast. If a demo tool feels like the point keeps gliding a bit after your hand suddenly stops, that's this smoothing at work.

3More precisely

Hand tracking outputs a horizontal and vertical location plus a confidence value for every joint, and many tools also report whether the palm faces forward or back, and whether it's a left or right hand. Some tools guess depth too and handle the hand in three dimensions, though that value wobbles especially hard from a single camera.

Accuracy needs splitting into two separate things: how far a single point drifts from the real joint, and how long tracking keeps the hand without breaking. A tool can be strong on the first and still be hard to use if it keeps losing the hand, and the reverse holds too.

The analogy breaks down somewhere too. A flashlight only has to move one circle, but hand tracking follows twenty-one joints at once while checking whether those points form a layout a real hand could actually produce. And a flashlight beam shows exactly what it's pointed at, but hand tracking has to recompute inside its narrowed search area and guess the location fresh each time. A hand can suddenly change direction or fold up entirely, so that guess misses often — which is why the search area always gets kept generously wide.

4Try it yourself

5Common misconceptions

It's easy to think hand tracking also figures out what a gesture means, but actually it only outputs joint locations — deciding something is "a wave" or "a peace sign" is handled by gesture recognition afterward.
It's easy to think once it recognizes a hand it keeps following it forever, but actually getting covered or leaving the screen breaks the trail and sends it back to searching from scratch.
It's easy to think jittery points mean poor performance, but actually that jitter always shows up when every frame gets computed separately — smoothing reduces it, at the cost of trading away some response speed.

7One-line summary

In shortHand tracking marks points on hand joints and re-marks them every frame to link them together, staying light and fast by remembering the previous frame.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02