Prompt versioning

A prompt is code. A small wording change can shift refunds, tone or the JSON shape, so it needs the same care as a code change. Prompt versioning means every production answer records which prompt version produced it. When quality drops, you can compare versions instead of guessing what changed on Tuesday.

The big idea

There are two places to keep prompts, and each one trades speed against discipline.

• Config-driven prompts live in a CMS or a config store. Product managers can iterate without a deploy. You still need version IDs, review and rollback.
• Prompts in code are typed templates in the repository. Review and CI evals come naturally. Shipping a prompt change is a deploy.

A recipe book and a recipe card are a fair picture. The card can be changed fast, but the book keeps every version someone approved. The analogy breaks at one point: a prompt change can affect the whole product at once, so the card needs the same review as the book.

A hybrid that works well: the prompt file lives in git, and the runtime config points at a version.

prompts/support_answer_v12.txt   # reviewed in git
runtime config: { prompt_id: "support_answer", version: 12, model: "..." }

Walking through a change

Suppose you want to change a support answer prompt.

1. Copy the current prompt to a new version, v13. Do not edit v12 in place.
2. Make the change, such as moving "never invent invoice IDs" next to the output schema.
3. Run the offline eval suite on the change. Compare the score with the baseline.
4. Ship the new version to 10% of traffic. Compare thumbs-down rate with the old version.
5. If it wins on both checks, raise the rollout. If not, roll back by changing the version number in config.

Every production trace carries prompt_version. That is what makes step 5 possible.

Rules that keep it honest

• Log prompt_version on every response. Without it, you cannot attribute a regression.
• Bump the version on any change. Never edit a shipped version. An edit makes old traces unreproducible.
• Run evals in CI on prompt pull requests, even a small smoke set.
• Version the parts that act like prompts too. Few-shot examples and tool descriptions shape behavior, so they go under the same scheme.

Common mistakes

• Editing a shipped prompt. Old traces can no longer be reproduced.
• Versioning the system prompt but not the tool descriptions. A changed tool description can change behavior with no prompt diff at all.
• Shipping without a baseline. Without the old score, a new score means nothing.
• Letting a CMS bypass review for a safety-critical prompt.

Check yourself

• Why not edit v12 in place when fixing a bug? Old traces then point at a prompt that no longer exists, and you cannot tell what produced them.
• Why version tool descriptions? The model reads them as instructions, so a change to them changes behavior.

Key takeaways

• Treat prompts as code: version them, review them and test them.
• Record the prompt version on every response.
• Bump the version for every change, and never edit a shipped version.
• Roll back by changing the version pointer in config.

Further learning

Next, see how the production answers are checked over time in online evals and alerts.