3D Generation

A flat photo has no depth

A photo flattens the world onto a single plane. Whether that cup sits 30 centimeters away or 3 meters away gets erased the moment it's captured. So pulling a 3D shape back out of a photo means recovering information that's already gone.

Photograph the same thing again from a different spot, and a clue appears. Things nearby seem to shift a lot; things far away barely move. Gather that difference across several photos and how far away something is starts narrowing down — the same reason having two eyes gives you a sense of depth.

But this trick only works where a camera actually pointed. A side that was never photographed leaves no clue behind, and no amount of calculation can recover it. That side isn't being reconstructed — it's being made up.

Piecing it together from many photos

Building from a stack of photos is the older, established approach. The order usually runs like this: first, work out where and in which direction each photo was taken. Matching points that show up in common across several photos narrows down the camera positions.

Next comes filling the empty space. For any point in space, the system asks: if something of a certain color and density sat there, would every photo show what it actually shows? Where the guess is off, it gets nudged a little, over and over, until everything lines up with every photo.

This approach is very accurate wherever a camera pointed. But it's a one-off calculation for that one scene, so it can't carry over to a different object, and the photos have to be spread out evenly. Cover only one side heavily and the other side stays empty.

Building from one photo or a line of text

Often there's only a single photo, or just a line of text. There's nothing to observe here, so the gap gets filled by generation instead. The method turns out to be surprisingly simple: produce several pictures of "this object seen from the side," "seen from the back," and treat those pictures as if they were photos taken from several angles, hardening them into a 3D shape.

So the quality of the result depends on how well the imagined pictures agree with each other. If the handle visible from the front is missing from the side view, it gets mangled when the shape is hardened. Making several viewpoints at once without letting them contradict each other is the central challenge here.

The invented back is only ever a plausible back. It can differ from the object's real back, and running it again produces a different guess. This approach shouldn't be used for work that has to reproduce a real object precisely.

What holds the result

Storing results as a shell is the older format. The surface gets built from small connected triangles, with a texture image wrapped over it. Games and video tools have used this format for a long time, so it drops right in and the files stay light.

There's also a way of filling space itself. Instead of a surface, it records how strongly each point in space shows some color. This captures even things with fuzzy edges, like glass or fur, and looks realistic — but without a surface, it's hard to use directly for physics or editing.

Scattering small blobs through space has become widely used lately. Millions of blobs, each with its own position, color, and size, get stored and splashed onto the screen when needed, which draws extremely fast. Which format gets used depends on the goal, and the two are sometimes converted into each other.

Where it's still weak

The raw output is often hard to use as-is. Surfaces can be bumpy or made of far too many triangles, and texture images blur enough to show under close inspection. Places that should be hollow sometimes come out solid, and the reverse happens too.

Thin, transparent, and shiny things are also weak spots. Things like wire mesh or hair come out clumped together, and glass or metal can mistake a reflection for a surface pattern and bake it right in — which is why changing the lighting makes the result look wrong.