Tracing and logging LLM calls

A trace is the full tree of work for one user request: a parent span for the request, and child spans for each model call, retrieval step and tool call. With a trace, you can replay what the model saw. Without one, "the model hallucinated" is a guess. With one, you can see the empty retrieval or the tool error that led to the bad answer.

The big idea

Use hierarchy. A single chat turn may be one span. An agent may be dozens of steps: plan, search, read, edit, test. A flat log line per call tells you a step failed. A tree tells you which step it was and what came before it.

A flight recorder on an aircraft is a fair picture. It keeps the full sequence of events, so the crash can be replayed. The analogy stops at privacy: a flight recorder does not hold customer documents, but an LLM trace often does.

Walking through a trace

Here is a trace for one support question, where a tool call times out:

trace 8f2a: support question (session s-107)
├─ retrieval   top-k=5   ids: refund-v3#2, faq#11 ...   empty=false
├─ llm call    model=m-2025-06  prompt_version=12
│              tokens in=1840 out=212   stop=tool_use
└─ tool call   get_order(order_id)   error=timeout after 2 s

Read it top to bottom. Retrieval worked. The model asked for the order. The tool timed out, so the model had nothing to answer with. The fix is in the tool path, not the prompt. A flat log would have shown only three lines with no link between them.

What to log on every model call

Privacy comes first

Redact secrets, payment data and credentials before anything is saved. Hash or segment personal data, and limit who can open raw traces. Follow the same retention and deletion rules as your other production logs.

Keep a separate debug mode with richer logging for sensitive surfaces, and keep it off by default. Never log API keys. Be careful with full prompt dumps that contain customer documents under a contract.

Common mistakes

• Logging only HTTP status. A 200 with a wrong answer looks healthy.
• Flat logs with no trace ID. You cannot link a retrieval miss to the answer it caused.
• Missing stop reasons. A cut-off answer looks like a model choice when it was a token limit.
• Raw prompts everywhere. Anyone with log access can read customer data.

Check yourself

• In the example trace, why is the fix in the tool path and not the prompt? Retrieval worked and the model asked for the right thing. The tool timed out, so the model had no data to answer from.
• Why record the stop reason? A response that stopped at a token limit looks like a finished answer unless you check it.

Key takeaways

• A trace is a tree for one request. Spans show which step failed.
• Record model, prompt version, tokens, cost, latency, tool results and retrieved IDs on every call.
• Redact private data before saving, and limit who can read raw traces.
• Read real failing traces before you trust the dashboards.

Further learning

Next, see how to record which prompt version each trace used, in prompt versioning.