Rate limits and provider quotas

Providers such as APNs, FCM, email services and SMS gateways each have rate limits. So do your own users. A token bucket keeps you under those limits. The bucket holds up to a fixed number of tokens and refills at a steady rate. Each send takes one token. When the bucket is empty, the send waits. It does not fire anyway.

The big idea

Limits apply at several layers, so one noisy product can't drain a shared budget:

Workers should check a distributed token bucket, usually in Redis, before they call a provider.

Walking through it

Take a flash sale: 50 million users are eligible for a push, and the provider allows 80,000 pushes per second.

50,000,000 pushes / 80,000 per second = 625 seconds, about 10.4 minutes

The campaign takes about ten minutes, and the bucket is what keeps it at that pace.

A small trace shows the bucket at work, with capacity 3 and a refill of 1 token per second:

t=0   bucket full (3)     send 1 -> 2 left
t=0   send 2 -> 1 left
t=0   send 3 -> 0 left
t=0   send 4 -> bucket empty: NACK with a delay, requeue with backoff
t=1s  one token refilled  -> send 4 goes out

The empty bucket case needs care. A worker that spins, retrying in a tight loop, burns CPU and does no useful work. A NACK (negative acknowledgement) with a delay returns the message to the queue to be picked up after the delay. When a provider returns a rate-limit error, back off for that provider as a whole, and don't retry each message at once. An immediate retry storm is how a 429 becomes an outage.

Circuit-break a failing provider adapter, as described in resilience, so that traffic shifts to a secondary provider without redeploying the business logic.

Common mistakes

• Spinning on an empty bucket. Delay the message, don't loop.
• Limiting only per user. Without a per-provider limit, one campaign can exceed the provider's quota for everyone.
• Retrying rate-limited sends immediately. Back off globally for that provider.

Check yourself

1. A flash sale sends 50 million pushes at a cap of 80,000 per second. How long does it take?

   About 625 seconds, roughly 10 minutes.

2. Why NACK with a delay rather than retry at once when a bucket is empty?

   An immediate retry spins and wastes work. A delayed retry waits for tokens to refill.

Key takeaways

• Token buckets keep sends under each provider's limit.
• Apply limits at user, template, provider and priority levels.
• An empty bucket means delay, not spin.
• Back off the whole provider after a rate-limit error.

Further learning

Next: retries and provider failures, which covers what happens after a send fails.