Production and case studies
LLMOps and observability
Tracing, prompt management, feedback loops and online monitoring for LLM applications.
Classic operations asks whether the service is up and fast. LLMOps asks whether the product is still good. Are answers faithful? Did the new prompt break refunds? Did the provider quietly update the model? You still need the metrics, logs and traces from observability. But the unit of debugging is now a generation: the inputs, the retrieved context, the tool calls, the output, the scores and the cost. An HTTP 200 tells you almost nothing about quality.
Teams that skip this layer ship on feel until a quiet quality drop shows up in support tickets or the invoice. The investment is mostly process: version IDs, a labeled set of good cases and the habit of reading traces. It does not depend on a particular product.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Tracing and logging | What do you record for each model call, and how do you see a whole request? |
| 2 | Prompt versioning | How do you know which prompt produced which answer? |
| 3 | Online evals and alerts | How do you catch quality drops in live traffic? |
| 4 | LLM gateways | Where do keys, quotas, fallbacks and spend live? |
Who owns what
Name the owners early, or traces rot unread. Feature engineers own prompt versions and offline evals for their feature. A platform owner runs the gateway, the tracing pipeline and spend alerts. Support or ops owns the queue of labeled failures. A small team can have one person wearing all three hats. Even then, write down the weekly routine so it survives a vacation.
A weekly rhythm
LLMOps fails when it is only a vendor dashboard. A light routine that works for a small team:
- Daily. Scan error rate, cost per 1,000 requests and cache hit rate. Read about 15 minutes of thumbs-down traces.
- Weekly. Review about 50 traces, tag the failure modes and add the best 5 to 10 cases to the golden set. Check that the LLM judge still agrees with a few human labels.
- Per change. A prompt or model change must show the offline eval difference, overall and per slice, plus the expected cost impact.
- Monthly. Decide whether to pin or upgrade each provider model. Remove stale prompt versions and review who can read raw traces.
Testing changes safely
Use the same discipline as a product A/B test. Randomize by user or session, and keep each conversation on one variant so it does not change mid-chat. Pick a primary metric tied to the job, such as task success or acceptance rate, and track cost and latency as guardrails. LLM output varies a lot, so small tests mislead.
For risky changes, use shadow mode: the new prompt sees the traffic, but users still get the old output. Compare the two offline, then run a real A/B test.
Responding to an incident
When quality drops but the service dashboards are green:
- Freeze prompt deploys and model alias changes.
- Compare today's traces with yesterday's on the failing slice: retrieval emptiness, tool errors and model IDs.
- Roll back the prompt version or the model pin through config. Do not wait for a full redeploy.
- Save the failing traces as eval cases before you fix anything, so the bug cannot return quietly.
Write the incident up like a postmortem. Include the eval that would have caught it.
Common mistakes
- Alerting only on latency and 5xx errors. A polite model that cites the wrong refund policy never trips those.
- One global quota. A runaway loop in a demo can starve production support.
- Pinning nothing. Aliases like "latest" move under you.
- Starting with a tool choice. Decide what you must answer about each request first, then pick the tool.
Key takeaways
- Log the inputs, outputs, tools, tokens, cost, latency and model and prompt versions, with redaction for private data.
- Version prompts like code, and record the version on every response.
- Turn thumbs-down, edits and escalations into labeled eval cases.
- Combine offline gates with sampled online checks and quality alerts.
- Centralize keys, quotas, fallbacks and spend in a gateway.
Further learning
Hamel Husain's walkthrough of error analysis on real traces is a good mental model before you add tooling.
The OpenTelemetry conventions for generative AI standardize span attributes, so your traces are not tied to one vendor.
OpenTelemetry semantic conventions for generative AIImportant GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository. This page has moved and is no longer maintained in this repository.OpenTelemetry
Start with the first concept page: tracing and logging.
Related
Observability: logs and traces
Structured logs, distributed tracing and OpenTelemetry, so you can ask new questions of a running system instead of guessing which dashboard to open.
Evaluating LLM apps
A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.
ML in production
Feature stores, training-serving skew, monitoring for drift and the lifecycle of a deployed model.