LLM latency metrics

Before you swap models or shorten a prompt, instrument four numbers on every production request. Averages hide the tail. Agent workloads with tool loops have long tails: one slow tool call or a 2,000-token reasoning dump ruins p99 while p50 looks fine.

The four numbers

Also track token counts (prompt, completion, cached) and success. A cheap failed call that forces a retry is more expensive than a slightly dearer first try that works.

Log percentiles (p50 / p95 / p99), not averages.

The big idea

Users judge the gap to something appearing, not total GPU time. A chat reply that streams the first token at 400 ms and finishes at 4 s feels faster than a silent 2.5 s wait, even though total generation time is longer.

That is why TTFT and end-to-end latency are different knobs. Streaming without moving tools off the critical path only paints over a slow orchestrator.

Break a slow request into network, prefill, tools and decode on a trace. Name the metric that matters for this surface: TTFT for chat, complete-job latency for a batch extraction.

Common mistakes

• Optimizing average latency. The p99 is what pages and on-call see.
• Counting only model time. Retrieval, tools and post-processing are often the wait.
• Ignoring failed calls. Retries double spend and blow the tail.

Check yourself

1. A copilot is "slow" but p50 generation is 1.2 s. What do you look at next? TTFT and end-to-end, including tools. Users feel the wait to first token, not GPU time.
2. Why log p99 as well as p50? Tool loops and long reasoning dumps hide in the tail. Averages look fine while some users wait forever.

Key takeaways

• Instrument TTFT, tokens/s, end-to-end latency and cost per successful task.
• Log percentiles and token breakdowns, including cache.
• Count success. Cheap failures are expensive.
• Pick the metric that matches the surface: first token vs job done.

Further learning

Next, make the wait feel shorter in streaming for perceived latency.