TTFT and token streaming

An LLM chat product streams its answer one token at a time, so text appears before the answer is finished. A token is a small piece of text, often part of a word. Two numbers describe the experience. Time to first token (TTFT) is the wait before anything shows. Inter-token latency is how steadily the rest arrives after that.

The big idea

Streaming changes what the user feels. A full answer that arrives after 4 seconds of silence feels slow. The same answer that starts in 0.8 seconds feels fast, even if the total time is similar. The first token is the moment the user decides the product is responding.

Total time still matters for capacity and cost. Streaming hides latency from the user, but the GPU still runs for the full decode. Plan capacity with total tokens, and judge the experience with TTFT.

Walking through it

Trace one request:

t=0.00 s  request arrives at the orchestrator
t=0.00 s  queue wait: no free GPU slot yet
t=0.35 s  prefill: the model reads the 3,000-token prompt
t=0.80 s  first token sent to the client        <- this is TTFT
t=0.80 s  decode: one token at a time until the answer ends
t=4.10 s  final token sent, stream closed

Three levers move the numbers:

• Longer prompts lengthen prefill, so TTFT rises.
• A deeper queue adds wait before prefill, so TTFT rises even with a short prompt.
• Longer answers lengthen decode, so total time rises, but TTFT doesn't change.

The streaming path must not buffer. The model server emits tokens as they arrive, the orchestrator forwards them, and the gateway and any proxy pass them through without holding the response. Server-sent events are a common format, because each token becomes a small event the browser handles as it arrives.

Common mistakes

• Reporting only total latency. A product can look fine on total time and still feel slow.
• Buffering in a proxy. Turn off response buffering on the gateway and any reverse proxy.
• Mixing queue wait with model speed. If the queue is 2 seconds and prefill is 0.3 seconds, the fix is capacity, not a faster model.

Check yourself

1. A 3,000-token prompt has a TTFT of 0.8 seconds. The user asks for a longer answer. Does TTFT change?

   No. Longer answers lengthen decode and total time. TTFT covers queue wait and prefill, which happen before the first token.

2. A proxy buffers the whole response. What does the user see?

   Nothing until the full answer is ready, so the streaming benefit disappears.

Key takeaways

• TTFT is the number users feel most. Track it on its own.
• TTFT is queue wait plus prefill. Decode affects total time, not the first token.
• Streaming works only if every hop forwards tokens without buffering.

Further learning

Next: LLM capacity math, which turns token load into GPU counts.