Streaming for perceived latency

Streaming is the highest-leverage UX fix for chat-style products. Send tokens to the client as they arrive (stream: true on most APIs). The model still takes the same total time to finish, but TTFT becomes the thing users feel.

Beyond the stream

• Optimistic UI. Show the user's message immediately, reserve a reply skeleton, keep the input disabled until the first token or an error.
• Progress for multi-step work. Agents should surface "searching docs…" and "checking order…", not a blank spinner for eight seconds.
• Chunked post-processing. If you moderate, translate or reformat the reply, process the stream in chunks instead of waiting for the full completion.
• Don't stream secrets. Tool arguments and chain-of-thought you do not want the user to see stay on the server. Stream only the final channel.

Walking through a slow agent

The agent still runs retrieve, then getAccount, then generate, all in series. Streaming the final answer does not hide the 2.4 s of tools.

Independent LLM or tool calls should run concurrently. Classic pattern: retrieve docs and fetch account state in parallel while a small model classifies intent, then merge into one generation call.

const [chunks, account, intent] = await Promise.all([
  retrieve("Where is order 881?"),
  getAccount("user-17"),
  classifyIntent("Where is order 881?"),
])
console.log(intent, chunks.length, account.plan)

This is a sketch of the orchestrator, not a full SDK. The win is wall-clock: three 800 ms calls become about 800 ms, not 2.4 s, and then the stream can start.

Speculative execution starts the likely next step before you know you need it, such as prefetching retrieval while the user is still typing. It wastes money on wrong paths. Use it only when the speculative call is cheap relative to saved latency. Never speculate a refund. Parallel tool calls that mutate state need idempotency keys and clear ownership.

Common mistakes

• Streaming while tools still run serially first. Fix the critical path, or stream a status line from the orchestrator before the model speaks.
• Streaming tool arguments to the browser. Keep them on the server.
• Streaming a batch job. Nobody is watching. Optimize throughput instead.

Check yourself

1. First token appears at 400 ms, full answer at 4 s. Silent wait of 2.5 s for a shorter answer. Which feels faster? The streamed one. Users judge time to first token.
2. Should a nightly extraction job stream? No. Optimize cost per document and throughput.

Key takeaways

• Stream interactive chat. Users feel TTFT.
• Show progress for multi-step agents.
• Parallelize independent tools before generation.
• Reserve speculation for cheap, safe work. Never speculative mutations.

Further learning

How the API actually emits chunks is in streaming and stop reasons. Next, cut the tokens that make both TTFT and the bill worse: reducing LLM tokens.