Text-to-Video

Not one frame, but many flowing into each other

Even a short video is hundreds of still frames flashing past quickly. A few seconds of footage needs hundreds of them. Generate those hundreds separately, though, and it never reads as a video — run the same prompt several times and stitch the pictures together, and every frame belongs to a slightly different world.

So video generation treats the whole clip as one block from the start. Instead of making the first frame and then the next, it starts the entire span as one hazy blob and sharpens it all together. Looking forward and backward at the same time while refining is exactly what lets the frames connect.

Handling time as its own dimension

Image generation looks across two directions, width and height. Video adds a third: time. A spot in the upper-left corner of the frame relates to the spot next to it, but it also relates to the same spot a moment earlier and a moment later. Getting that time-direction relationship right is what makes motion look natural.

Add one more dimension and the amount of data to handle multiplies. Stretch the same quality footage five times longer and the effort goes up by more than five times. That's exactly why it takes a while to generate and why a clip tends to come out short.

So the common approach splits the job up: build a small, coarse skeleton first, fill in the frames between to smooth it out, then raise the resolution last. Nobody tries to do it all in one pass.

Where continuity breaks down

The most common failure is what something actually is. A blue cup held early on turns green a few seconds later, or an outfit changes mid-stride. Since every instant gets redrawn fresh, forgetting what came before shows up immediately.

Backgrounds drift too. Pan the camera sideways and back, and the spot it just passed often comes back looking different. Anything that leaves frame and re-enters is especially fragile — what left and what comes back rarely match.

Text is a weak spot as well. Lettering on a sign is already hard to render in a single still frame; across a moving sequence, the strokes keep wobbling. That's why videos where text actually matters usually add it on top of the generated footage separately, as its own layer.

It still gets physics wrong often

Water flows uphill, a dropped object lands without bouncing, or shattered glass reassembles itself. That's because the model never learned a rule for how the world actually works — it learned to string together scenes that look plausible in sequence.

Anything where two things touch and change state — handing off an object, pouring something — is especially prone to breaking. Cutting clips short and steering around difficult motion hides most of the trouble, and there's active work aimed at teaching models an actual sense of how the world behaves, tackling the problem head-on rather than working around it.

What you can put in a request

The same fields used for a still image carry over: what's in frame, the mood, the color palette. Video adds its own fields on top — pushing the camera slowly in, panning sideways, holding it still.

Handing over a starting image is common practice too. Generate one frame you like first, then say "start from this and move like this." Starting from a fixed point makes the outcome far easier to predict, which is why this approach is more common in actual practice.