Streaming contract and cancellation

A streamed answer is a contract between server and client. The client needs to know what each event means, what to do when the connection drops, and how to stop the answer. Cancellation matters for money as well as UX: a stopped answer should stop GPU work, or the product keeps paying for tokens nobody reads.

The big idea

Define a small set of event types, with a stable schema, and version it. A typical set is message_started, token, tool_call, tool_result, message_completed and error. Every event carries the message ID and a sequence number. The client uses the sequence number to detect gaps and the message ID to reconnect.

Make cancellation explicit. The client sends a cancel request for that message ID, the orchestrator stops the stream, and it tells the model server to stop decoding. Closing the connection alone is not enough, because the backend may keep decoding until it notices.

Walking through it

A stream on the wire looks like this:

event: message_started
id: 812-1
data: {"messageId":"m_812","model":"fast-tier"}

event: token
id: 812-2
data: {"messageId":"m_812","seq":2,"text":"Hello"}

event: token
id: 812-3
data: {"messageId":"m_812","seq":3,"text":" there"}

event: message_completed
id: 812-4
data: {"messageId":"m_812","inputTokens":1930,"outputTokens":212}

Now the connection drops after event 3. The client reconnects with the last event ID it saw, 812-3. The server replays from event 4 if it still holds the buffer. If it doesn't, the client shows the partial answer and offers to regenerate. Either way, the user is not left staring at a frozen message.

On the server, save the partial answer as the stream goes, with a status of streaming. If the client disconnects and never returns, mark the message incomplete and keep the tokens already spent in billing. Don't bill twice when the user retries.

Cancellation runs in order:

1. Client sends POST /messages/m_812/cancel.
2. Orchestrator marks the message cancelled and closes the client stream.
3. Orchestrator tells the model server to stop decoding, so the GPU slot frees up.
4. Billing charges for the tokens produced so far.

If step 3 fails, the orchestrator keeps a timeout, so the GPU slot can't stay busy forever.

Common mistakes

• Closing the socket without telling the model server. The GPU keeps decoding a reply nobody reads.
• No sequence numbers. The client can't tell if it missed an event.
• Losing partial answers on disconnect. Save them as they stream.
• Changing event shapes without versioning. Old clients break silently.

Check yourself

1. The user closes the tab mid-answer. What must happen to the GPU work?

   The orchestrator must tell the model server to stop decoding, so the slot frees up. Closing the client connection alone is not enough.

2. A client gets events 1 to 3 and then disconnects. How does it resume?

   It reconnects with the last event ID it saw, and the server replays from event 4 if the buffer still holds it.

Key takeaways

• Define a small, versioned set of event types, with message IDs and sequence numbers.
• Make cancellation explicit, and stop GPU decoding, not only the connection.
• Save partial answers as they stream, and bill only for the tokens produced.
• Let clients resume from the last event ID when the server still has the buffer.

Further learning

For the serving side of batching and cancellation, see the inference serving platform.

For the API surface that most products build on, see LLM APIs.