Local plus global limits

A Redis round trip costs about 0.3 to 1 ms and puts Redis on your hot path. At very high volume, that is too much. The hybrid design lets each server decide locally and syncs the counts to a shared store on a timer. The price is a small, bounded overshoot.

The big idea

Each gateway keeps its own token bucket for each key. Its size is its share of the global limit: the global limit divided by the number of servers. Requests are decided in memory. Every 100 ms to 1 s, each server reports what it consumed and receives the updated global view.

Envoy works this way in practice. Its local rate limit filter needs no network call. Its optional global service, Lyft's open-source ratelimit backed by Redis, handles limits that must be shared.

The trade-off is accuracy. Suppose a client's traffic lands on only 10 of 200 servers. Each server's share is 1/200 of the limit, so the client is capped far below its real limit. If sync is slow, the client can overshoot. Most products accept a few percent of overshoot in exchange for removing a network hop.

Walking through it: a 1 million requests per second design

Requirements. An API gateway in three regions serves 1 million requests per second at peak. Limits are per API key, set by plan (for example, 100 requests per second with bursts of 500). There are about 10 million active keys. The limiter must add under 1 ms at p99 and must not take the API down if it fails.

State. A token bucket needs two numbers plus the key, about 100 bytes in Redis with overhead:

10,000,000 keys × 100 bytes ≈ 1 GB per region

Memory is not the problem.

Throughput. Traffic splits 50% / 30% / 20% by region, so the largest region sees about 500,000 checks per second. A Redis shard can handle roughly 50,000 scripted checks per second with headroom:

500,000 checks/s ÷ 50,000 per shard = 10 shards, plus a replica each

That works, but it puts Redis on every request. The hybrid design cuts that sharply.

Design.

1. Gateways decide locally with an in-memory bucket per active key. Most requests never leave the process.
2. Regional sync. Every 250 ms, each node sends one batched "key to tokens consumed" call. With 200 nodes per region, that is 800 calls per second instead of 500,000 checks.
3. Strict keys go straight to Redis. Login attempts and low-limit keys use the script on every request.
4. Cross-region splits. A key's limit is divided by recent traffic. If a 100/s key gets 70% of its traffic in us-east, us-east enforces 70/s. A job rebalances about once a second.

Overshoot. During one 250 ms sync interval, nodes can each admit their share before learning the global count is used up. The bound is the rate times the interval: 100/s × 0.25 s = 25 requests. The sync interval is the knob.

Common mistakes

• Splitting the limit evenly when traffic is uneven. A client on 10 of 200 servers gets capped near 5/s, even though its limit is 100/s.
• Treating the hybrid as exact. It is bounded, not exact. Write the bound down.
• Coordinating across regions on every request. That adds 70 to 150 ms. Nobody does that for rate limits.

Check yourself

1. A 100/s key's traffic lands on 10 of 200 servers, each holding 1/200 of the limit. What goes wrong?

   Answer: each server allows only 0.5 requests per second, so the client is capped near 5/s, far below its 100/s limit.

2. Why is the overshoot bounded?

   Answer: nodes can run ahead only until the next sync, so the overshoot is about rate × interval, here 100 × 0.25 = 25 requests.

Key takeaways

• Redis on every request adds latency and makes Redis part of the hot path.
• Local buckets sized to each server's share remove that hop.
• Periodic sync bounds the error, and the sync interval is the tuning knob.
• Uneven traffic makes even splits unfair, so watch hot keys.

Further learning

Next, see what a client should see when it is refused, and how quotas differ from rate limits, in responses and quotas.