Prompt structure

A prompt has parts with different lifetimes. Some rules apply to every request. Some data changes with each request. The structure of the prompt should keep these apart, so the model cannot confuse a rule with a customer's ticket. Labeled sections and a fixed order do most of that work.

The big idea

Put these in the system prompt: product rules, tone, safety limits, tool-use rules and glossary definitions. Put these in the user message: the ticket, the document and the question.

Mixing them causes trouble. If you edit the system prompt to fix one ticket, you change every request. If the ticket text sits next to the rules, the model may treat a sentence from the ticket as an instruction.

A filing cabinet is a useful picture. Policy binders stay on the shelf. Case files go in the folder for that case. The analogy stops at one point: the model reads the whole cabinet on every call, so long binders cost tokens each time.

Walking through it

Here is a ticket classifier with labeled sections:

You classify customer support tickets for a billing team.

<instructions>
Classify into exactly one category from <categories>.
Extract the invoice number if present; otherwise null.
Do not invent invoice numbers.
</instructions>

<categories>
refund_request, payment_failed, invoice_question, other
</categories>

<ticket>
{{TICKET_TEXT}}
</ticket>

Respond with JSON only matching:
{"category": string, "invoice_number": string | null, "confidence": "high"|"medium"|"low"}

The tags do three jobs. They separate rules from data. They let you tell the model to use only the facts inside <ticket>. They also let you swap one section in a test without touching the others.

For long documents, put the source material first and the question last. Generation starts with the task still fresh in the context, so the model reads the evidence before it starts answering.

Common mistakes

• Per-request text in the system prompt. One ticket's details now sit in every call.
• Unlabeled examples. A sample input can look like the live input. Wrap examples in their own tags.
• Question first, evidence after. The model may answer a generic version of the question before it reads the sources.
• Dumping the whole wiki. More tokens do not mean more understanding. Long, noisy context hurts answers, so retrieve or trim first.

Check yourself

• Why put the question after the documents? The task is the last thing in the context when generation starts, so the model has already read the evidence.
• Where does "never invent invoice numbers" belong? In the system prompt, because it is a durable rule. The ticket itself is per-request data.

Key takeaways

• Keep durable rules and per-request data in separate parts of the prompt.
• Label each section with tags so the model knows what it is reading.
• For long inputs, put the evidence first and the question and output format last.
• Keep the context lean. Extra text costs tokens and can bury the facts that matter.

Further learning

Next, see how examples teach judgment in few-shot examples.