Conversation context assembly

Every chat turn needs the conversation so far, but the model's context window is finite and every token costs money. The product stores the full history, then builds a bounded prompt for each request. Context assembly is the step that decides what goes into that prompt, and in what order.

The big idea

Store turns, not prompts. Each turn records the user message, the assistant message, the model, the prompt version and the token counts. Keep files in object storage and reference them, instead of pasting their contents into every turn.

Assemble the prompt from parts, in priority order, under a fixed token budget. The system prompt and the latest user message always fit. Older turns, retrieved passages and summaries fill the rest.

Walking through it

A conversation schema:

conversations  id, user_id, title, created_at, model_default
turns          id, conversation_id, seq, role, content, model_id,
               prompt_version, input_tokens, output_tokens, created_at
attachments    id, turn_id, object_key, mime_type, size_bytes, extracted_text_ref
summaries      conversation_id, through_seq, content, token_count

Each request assembles the prompt in this order:

1. System prompt and product rules, always included.
2. Summary of the older turns, if the conversation is long.
3. Recent turns in order, newest last, until the budget is nearly full.
4. Retrieved passages and file references, each capped.
5. Latest user message, always included.

Budget arithmetic for an 8,000-token window, with 1,000 tokens reserved for the answer:

window            8,000
reserved output  -1,000
system prompt    -  400
latest message   -  300
summary          -  500
retrieved        - 1,500  (cap)
recent turns     -> remaining 4,300 tokens for the newest turns

When the budget runs out, drop the oldest turns first. Updating the summary in the background keeps old context from disappearing. Some products re-summarize every few turns and store the result with the turn sequence number it covers.

Cost follows the context size, not the history size. A 2,000-token prompt on every turn of a 20-turn chat costs far more than trimming the history to 1,000 tokens. A rough rule: input tokens on every turn grow with the conversation, so long chats are where the bill grows.

Store the references for every reply: the model ID, the prompt version and the token counts. You need them for debugging, evaluation and billing disputes.

Common mistakes

• Sending the whole history every turn. It costs more and slows down prefill.
• Pasting file contents into every turn. Store a reference, and attach only the extracted text the turn needs.
• Dropping the latest message when the budget is full. Reserve space for it first.
• Not recording the prompt version. You can't reproduce a bad answer without it.

Check yourself

1. Why reserve tokens for the answer when computing the prompt budget?

   The model writes the answer inside the same window. If the prompt fills the window, the answer gets cut short or fails.

2. A 20-turn chat sends the full history each time. What is the fix?

   Assemble a bounded prompt from the system prompt, a summary and the recent turns, and attach files by reference.

Key takeaways

• Store turns and file references, not full prompts.
• Assemble each prompt from parts, in priority order, under a fixed budget.
• Use a rolling summary for older turns, and keep the latest message safe.
• Record model, prompt version and token counts on every turn.

Further learning

Next: LLM rate limits and overload, which covers what happens when the fleet is full.

For the API side of these calls, see LLM APIs.