LLM fundamentals
Streaming and stop reasons
Tokens arrive as events. The model always stops for a reason. Branch on that reason. Do not assume success.
Interactive UIs should stream. Instead of waiting for the full completion, the API sends server-sent events (SSE) or a similar chunked protocol. You render partial text as it arrives. Perceived latency drops to TTFT even when the full answer takes seconds.
The model also stops for a reason. Branch on it. Do not assume success.
Concept
Before you start: the request is a list of messages and roles. Streaming is how the response arrives. The stop reason is why the model stopped emitting.
What the loop must handle
- Text deltas: append to a buffer, flush to the UI
- Tool-call deltas: arguments often arrive as partial JSON. Buffer until complete
- Final metadata: usage tokens, stop reason
- Mid-stream errors and client disconnects: cancel upstream when the user navigates away
Tip
Buffer tool-call argument fragments until the provider marks the tool-use block complete. Parsing half-built JSON on every delta is a common source of flaky tool execution.
Stop reasons
| Stop reason (typical names) | Meaning | What you should do |
|---|---|---|
end_turn / stop | Natural completion | Persist the assistant message. Show it |
max_tokens | Hit the output cap | Continue, raise the limit, or summarize |
tool_use / tool_calls | Model wants tools | Execute tools, append results, call again |
stop_sequence | Hit your custom stop | Parse delimited output carefully |
content_filter / refusal | Safety system intervened | Show a safe fallback. Log for review |
The tool loop is the heart of agents. Details in tool use. Always bound the number of round trips (for example max 8) and the wall-clock time.
Long jobs should not be one giant max_tokens=16_000 call. Prefer map-reduce over chunks, then a merge call. Each tool round is a natural checkpoint. This keeps failure domains small, improves cache reuse on shared instructions, and avoids a single timeout taking down a 10-minute generation.
Streaming connections die. Users navigate away. Tools hang. Propagate cancellation from the HTTP client to the provider stream when the user disconnects, otherwise you pay for tokens nobody reads. If a tool times out, return a structured tool error to the model ("invoice service timeout") rather than leaving the loop hanging. If the stream fails after 80% of the answer, decide product policy: show partial with a retry button, or hide and retry once server-side.
Common mistakes
- Assuming every stop is
end_turn.tool_useandmax_tokensneed different handling. - Parsing tool JSON on every delta. Wait until the block is complete.
- Leaving the provider stream running after the user leaves. You still pay.
Check yourself
- The response stops with
tool_use. What next? Execute the tools, append the results as tool turns, and call the model again. - Stop reason is
max_tokens. Is the answer complete? No. Continue with a follow-up, raise the limit, or summarize.
Key takeaways
- Stream for interactive UX. Perceived latency is TTFT.
- Buffer tool-call JSON until the block is complete.
- Always branch on stop reason.
- Bound tool rounds. Cancel the upstream stream on disconnect.
Further learning
Next, retries and rate limits. For the UX side of streaming, see streaming for perceived latency.
Related
Working with LLM APIs
Messages, roles, streaming, stop reasons, errors, rate limits and batch APIs. The plumbing of every LLM app.
Chat messages and roles
Each call resends the full conversation. Roles mark who said what. Pin the model id and version the system prompt.
Tool use and function calling
Letting models call your code: tool schemas, the tool loop, and designing tools models use well.