Speech and video generation

Speech and video use the same generative ideas as images, with different bottlenecks. Speech is often turned into a sequence of discrete audio tokens that a language model can predict. Video is harder, because every frame adds to the work, and the model must keep objects consistent across time.

Audio: codecs and tokens

A neural audio codec, such as SoundStream or EnCodec, compresses a waveform into discrete tokens at a low bitrate, and decodes them back to sound. Once speech is tokens, an autoregressive language model can generate it the same way it generates text. This is the approach behind AudioLM-style systems.

Text-to-speech systems use different designs. Some use autoregressive codec models. Others use diffusion on spectrograms, or a hybrid pipeline. Latency needs differ from chat. A voice conversation has a tight budget for the first audio, because silence feels broken.

Speech recognition is a different job. Whisper-style models are encoder-decoder transformers trained on large amounts of weakly labeled audio. They turn speech into text, so they recognize speech rather than generate it. They share encoder ideas with the generative models, but they are not generative in this sense.

Video: more frames, more work

Video multiplies the number of tokens by the number of frames. Common approaches include:

• Cascades. A low-resolution, few-frame base model, followed by temporal upsampling.
• Latent video. Compress frames with the same kind of autoencoder used for images, then apply attention over space and time.
• Image-conditioned video. Animate a still image, which reduces how much the model must invent.

Failure modes are specific to video: identity drift, where a face slowly changes; flickering textures; and broken physics. Evaluation is still partly human preference, because automatic metrics lag behind what viewers notice.

Speech in a product

A voice assistant chains several models: speech in, recognition with ASR, a language model, then speech out with TTS. Each stage has its own latency budget and its own batching behavior. Knowing which stage is diffusion, which is autoregressive, and which is recognition stops you from applying LLM batching advice to the wrong part.

Common mistakes

• Applying LLM batching to audio codecs or video models. Their shapes and bottlenecks differ.
• Scoring video with text-style metrics. Judge it with human preference and task checks.
• Ignoring the latency of the first audio. Users notice a slow first response more than a slow total time.

Check yourself

1. What does a neural audio codec produce? Discrete audio tokens that a language model can predict, and that the codec can decode back to sound.
2. Why is speech recognition not a generative model in this sense? It turns audio into text. It does not generate new audio.

Key takeaways

• Speech can be modeled as discrete tokens, using a neural codec.
• Text-to-speech and recognition use different designs, and latency matters most for voice.
• Video multiplies the work by the number of frames, and its failure modes include identity drift and flicker.
• In a voice product, each stage has its own model family and latency budget.

Further learning

The topic overview lists the other families and the cross-cutting pitfalls: generative models beyond text.