Video generation models: Sora, Veo, Runway, and what they actually do

Generating a single image is hard. Generating a video means generating many images that are also temporally consistent — the same character has the same face frame to frame, the cup stays on the table, the camera moves smoothly.

The naive approach — generate each frame independently — produces flickering chaos. The frontier models all share the same fix: treat video as a single 3D tensor (height × width × time) and run diffusion across all three axes at once. That's why these models need orders of magnitude more compute than image models, and why outputs tend to be 3–10 seconds long: the tensor grows linearly with duration but quality degrades fast past that.

Almost every modern video model is a latent diffusion model. The recipe:

1. Train an autoencoder that compresses a video clip into a much smaller latent tensor.
2. Add Gaussian noise to that latent in $T$ small steps until it's pure noise.
3. Train a network to reverse one step of noise, conditioned on a text embedding.
4. To generate: start from noise, iteratively denoise for $T$ steps using your prompt as the condition.
5. Decode the final latent back to pixels.

The text embedding (often from a CLIP-like model or T5) steers the denoising toward content that matches the prompt. The 3D structure means each step considers spatial and temporal context.

Frontier text-to-video as of mid-2025:

• OpenAI Sora — long-form (up to ~60s), strong physics, closed.
• Google Veo (Veo 2/3) — competitive quality, exposed via Vertex AI and Gemini API.
• Runway Gen-3 / Gen-4 — earliest commercial offering, mature control surface.
• Kling — Chinese model, particularly strong on human motion.
• Luma Dream Machine — fast, web-first.
• Open source — HunyuanVideo, Mochi, CogVideoX, LTX-Video — viable on a single GPU at 5–10s clips.

All of them produce noticeably different aesthetics. "Try several" is genuinely good advice; outputs differ more than for text models.

Three input modes you'll see:

• Text-to-video (T2V): just a prompt. Maximum creative freedom, minimum control.
• Image-to-video (I2V): a starting frame + a prompt describing motion. You control the look; the model controls the motion. Far more reliable for character/brand consistency.
• Video-to-video (V2V): a reference clip + a prompt. The model preserves motion or composition while restyling. Used heavily for music videos and visual effects.

I2V is what most production teams actually use. Generate the look with an image model (where iteration is cheap), then animate the result. It sidesteps the "every clip looks like a different character" problem.

Cheap iteration on the look, expensive only once.

Video prompts are structured. A bare phrase like "a man walking" wastes the model's capacity. Good prompts cover four axes:

1. Subject: who/what — including specific traits ("a tall woman with curly red hair in a navy raincoat").
2. Action: what they're doing, including pace ("jogging slowly along a wet boardwalk").
3. Setting: where, time of day, weather, mood.
4. Camera: shot type (close-up, wide), motion (static, slow dolly forward, handheld), lens (35mm, telephoto).

Most models also respect aesthetic keywords ("cinematic", "shot on 16mm", "in the style of Wes Anderson") and can be steered hard by a single strong reference image when in I2V mode.

Known and recurring failure modes — budget for these:

• Hands and faces: still fragile, especially in motion. Faces "morph" between frames.
• Text: signs, screens, T-shirt logos. Comes out as gibberish almost always.
• Object permanence: a glass that goes off-screen and returns isn't the same glass.
• Physics: cloth, water, hair occasionally violate basic causality.
• Long shots: past 10–15 seconds, drift compounds and content slowly mutates.
• Compositional prompts: "a dog under a table next to a blue chair" — models conflate attributes.

Workflow trick: generate clips in 4–8s pieces, then edit them together. Avoids most of the long-shot drift.

Video is expensive. Rough ballpark for a 5-second 1080p clip from a frontier model: $0.50–$3.00 per generation, and 30–120 seconds of wall time. Multiply by however many iterations you need.

Budget at least 5x your final clip count in generations — the model misses on the first try more often than image models do, and prompt iteration is part of the workflow rather than the exception.

For cost-sensitive use: an open-source model on a single H100 can produce a 5s clip in ~2 minutes at near-zero per-unit cost, if you have the GPU sitting around. For one-off production, the API is cheaper than hardware. For ongoing volume, it flips.