Retries and rate limits

Model calls fail. Rate limits hit. The question is not whether you retry. It is what is safe to retry.

RPM is requests per minute. TPM is tokens per minute, often the binding constraint. A single 100K-token prompt burns TPM fast.

What to retry

Set timeouts larger than your p99 generation time but smaller than the user's patience. Stream so the connection stays busy. For very long jobs, prefer batch APIs.

If p95 generation is 12 s and the gateway timeout is 10 s, users see failures that are actually successes truncated by your proxy. Raise the gateway timeout above p99, keep an app-level deadline, and stream so load balancers see active bytes.

Walking through backoff

This retry helper is self-contained. A fake flaky call fails twice, then succeeds.

def with_retries(fn, attempts=5):
    delay = 0.5
    for i in range(attempts):
        try:
            return fn()
        except Exception as err:
            status = getattr(err, "status", None)
            retryable = status in (429, 529) or (status is not None and status >= 500)
            if not retryable or i == attempts - 1:
                raise
            jitter = delay * 0.1  # fixed here so the output is stable
            print(f"retry {i + 1} after {delay + jitter:.2f}s ({status})")
            delay = min(delay * 2, 30)

class ApiError(Exception):
    def __init__(self, status):
        self.status = status

fails = {"n": 0}

def flaky():
    fails["n"] += 1
    if fails["n"] < 3:
        raise ApiError(429)
    return "ok"

print(with_retries(flaky))

Output:

retry 1 after 0.55s (429)
retry 2 after 1.10s (429)
ok

In production, replace the fixed 0.1 with a random draw in [0, delay] so many clients do not retry at the same instant.

Idempotency across the tool boundary

1. Model returns tool_use for refund_order.
2. You execute the refund. The provider 529s on the follow-up completion.
3. A naïve retry re-sends the prior assistant tool call without recording success. Double refund.

Fix with tool-side idempotency keys: derive a key from (conversation_id, tool_use_id) and make the refund handler a no-op on replay. Store tool results in the conversation transcript before calling the model again.

Common mistakes

• Retrying 400s. The request is wrong. Fix it.
• Retrying without jitter. You DDoS the provider on the recovery.
• Retrying the whole agent loop after a successful mutation.

Check yourself

1. You get 429. Retry immediately at full rate? No. Exponential backoff plus jitter, and shed load.
2. The refund tool ran, then the completion call 529'd. What must be true before you retry the model? The refund handler is idempotent on (conversation_id, tool_use_id), and the tool result is already in the transcript.

Key takeaways

• Retry 429 and 5xx with exponential backoff and jitter.
• Do not retry 400 or 401.
• TPM often binds before RPM.
• Tools need idempotency keys. Model calls usually do not.

Further learning

A gateway is where retries, quotas and fallbacks should live: LLM gateways. Next, batch LLM APIs for work that can wait.