Prompting and context
Audio and video
Audio is either speech-to-text plus a text model, or native audio tokens. Video is sampled frames. Cost scales with duration and resolution.
Audio and video look like one product feature ("accept a clip") and are two different pipelines. Audio is either a stack you can inspect or a single native model. Video is almost never "the whole file." It is a sample of frames, sometimes plus an audio track.
Concept
Before you start: voice agents are the interactive case, where turn-taking and latency dominate. This page is about the input itself: how a memo or a clip becomes tokens the model can read.
The big idea
Two audio architectures dominate:
| Approach | Pipeline | Strengths | Weaknesses |
|---|---|---|---|
| STT → LLM → TTS | Whisper-class ASR, text model, speech synth | Debuggable, swappable parts, cheap text LLM | Latency stack. ASR errors poison the LLM |
| Native audio models | Audio tokens in and out of one model | Better prosody and overlap handling | Cost, less inspectable mid-pipeline |
For a 4-minute voice memo ("summarize this"), STT then an LLM is usually enough and easier to eval. You can grade the transcript separately from the summary. For a live voice agent, native audio can be worth the opacity. See voice agents.
Video is sampled as frames, plus an optional audio track. Cost scales with frames × resolution × duration. Practical systems sample 1 frame per second or at scene cuts, caption keyframes, and only send the full clip to an expensive model when retrieval says it matters.
Ask whether you need temporal reasoning ("what happens after the red light?") or just a representative still. Many "video" tickets collapse to three well-chosen frames.
Walking through a clip
A 60-second warehouse video at 1 frame per second is 60 images. If each frame is billed like a ~1k-token photo, you are in the tens of thousands of image tokens before the question. Scene-cut sampling might keep 8 frames. Three stills at the moments a human would screenshot may answer "is the pallet labeled?"
A 4-minute memo through STT might produce ~600 words of text, then a cheap text summary. The same memo through a native audio model is one call with better overlap handling and a bill you cannot split into "transcript vs summary" when something is wrong.
Vision prefills are heavier than text-only prefills with the same token count because of the vision encoder path. Product implications that apply to frames too:
- Run async extraction for uploads. Email the user when done.
- For interactive "explain this screenshot" chat, stream text and accept a higher time to first token.
- Cache identical binaries by content-hash. Users resend the same PNG or clip.
Common mistakes
- Sending the whole video. Sample first. Most tickets do not need every frame.
- Native audio when you need to debug a wrong summary. You cannot tell whether the ears or the brain failed.
- Treating video as one modality with one price. Duration and resolution dominate.
Check yourself
- A 4-minute memo that must be summarized overnight: which audio path? STT then an LLM. You can grade the transcript and the summary apart.
- Do you need the full clip to answer "is the pallet labeled?" Usually no. A few stills at the labeled faces are enough.
Key takeaways
- STT → LLM is debuggable. Native audio shines for real-time voice.
- Video cost is frames times resolution times duration. Sample on purpose.
- Many video questions collapse to a few stills.
Further learning
Next, put the pieces into a pipeline you can eval in multimodal evals.
Related
Document understanding
PDFs, tables and charts: prefer a native text layer when it exists, and use vision for the parts text cannot see.
Multimodal evals
Grade real photos and screenshots, not clean demos. Schema validity is not a correct total, and production lighting is part of the fixture.
Voice agents
Real-time speech AI: STT→LLM→TTS pipelines vs speech-to-speech models, latency budgets and turn-taking.