Latency budgets

A product requirement might say the home page API responds within 300 ms at p99. That is one number, shared by many services. A latency budget splits it across each hop, so every team knows how much time it can spend and still meet the target.

The big idea

Think of a trip with a fixed travel time. You allocate minutes to each leg. Legs that happen one after another use up the time together. Legs that happen side by side only cost as much as the slowest one.

That gives two rules:

• Parallel calls cost the maximum. Sequential calls cost the sum.
• Keep headroom. A budget with no slack fails on the first garbage-collection pause.

Walking through it

Here is a budget for the home page, with a 300 ms p99 target.

The parallel calls don't add up. Feed, profile and recommendations run together, so only the largest one counts. That leaves 50 ms of headroom.

Turning a chain of three 50 ms calls into one parallel fan-out saves 100 ms. Latency goes from 50 + 50 + 50 = 150 ms to max(50, 50, 50) = 50 ms.

Two more rules keep budgets honest:

• Budgets are targets, not proof. Verify the end-to-end latency with measurement and tracing.
• Propagate deadlines. Pass the remaining time along with the request. gRPC does this natively. Then a downstream service can stop working on a request the caller has already given up on.

Common mistakes

• Zero headroom. Budgets with no slack fail on the first slow hop.
• Treating budgets as exact. Percentiles don't add up cleanly, so use budgets to plan and check the total with real traces.
• Forgetting the deadline. A service that keeps working on abandoned requests wastes capacity that live requests need.

Check yourself

1. Three 50 ms calls run in sequence. Then they run in parallel. What is the new latency? 50 ms, down from 150 ms, because the parallel calls cost the maximum.
2. Why pass the remaining time to downstream services? So they stop work on requests the caller has already given up on.

Key takeaways

• A latency budget splits one end-to-end target across every hop.
• Parallel calls cost the maximum. Sequential calls cost the sum.
• Keep headroom for GC pauses, retries and queueing.
• Propagate deadlines so downstream work stops when the caller gives up.

Further learning

Use observability to measure each hop against its budget, so you can see which team is over.

Next, return to the main availability and latency overview to see how these pages connect.