NeRFNeural Radiance Field

Taming the booth with dozens of photos

Dozens of photos of the same subject, taken from many angles, are the raw material. For each photo, the system first works out where the camera stood, then stretches a line of sight from that spot through every point on the image.

At first the booth just makes things up. Build a picture from the answers gathered along each line of sight, and it looks nothing like the original photo. The booth's habit of answering gets nudged, little by little, by exactly that gap.

Repeat this hundreds of thousands of times, and the booth ends up giving answers — for the spots and directions the photos covered — that nearly match the photos themselves. From there, asking about a spot no photo ever covered also gets a plausible answer. It's filling in the empty angles between one photo and the next.

The booth answers with color and density

A question hands over two things: a spot in space, and the direction you're viewing it from. The answer hands back two things too: the color of the light coming from that spot, and how solid that spot is.

Density works like fog. Near zero, it's empty air and the line of sight passes right through; high, it's a solid surface and the line of sight stops there. Nobody has to point out where the surface is — wherever density suddenly spikes, that's the surface.

Stacking answers along a line of sight

Coloring a single point on the screen means stretching a line of sight from that point into the scene. Dozens to hundreds of spots get placed along that line, and the booth gets asked at every one.

The answers get stacked front to back. If a dense answer already showed up near the front, the color behind it gets mostly buried; if the front is empty, the color behind comes through as-is. The color gathered this way becomes that point's final color.

Making one picture means doing this for every point on the screen. With hundreds of questions per point, the total count is enormous — which is exactly why people call NeRF slow.

The answer changes with direction

The same spot can show a different color depending on where you view it from — a reflection off a glass, a glint on metal, light bouncing off a wet floor. That's why direction rides along with every question.

Drop direction and ask about the spot alone, and a shiny surface comes out looking like flat paint. Add direction back in, and the reflection seems to follow you as you turn your head. But at angles the photos never covered, that same freedom works against you — hazy, foggy smudges can appear floating in empty space.

Where it splits from Gaussian splatting

Gaussian splatting does the same job from several photos. Where the two split is what they hold the result as. Splatting holds a visible pile of pieces scattered through space; NeRF holds a computational rule that answers a question.

So splatting only has to press its pieces onto the screen, which makes it fast, while NeRF has to ask hundreds of questions along every line of sight, which makes it slow. In exchange, NeRF is strong at rendering things with no clear surface, like fog or translucent glass, smoothly.