LLM gateways

An LLM gateway is a proxy that sits between your applications and the model providers. Every model call goes through it. It can be built in-house or bought as a product. Its job is to hold the API keys, enforce quotas, route requests, retry safely and record the spend in one place.

The big idea

Without a gateway, every service holds its own provider key. Nobody knows which feature drives the bill, and one loop can drain the budget. With a gateway, keys live in one place, each feature has a quota, and every call is logged with a team and a feature label.

A corporate travel desk is a fair picture. Staff book through one desk that holds the budget, applies the policy and keeps the receipts. The analogy stops at speed: the desk adds a hop to every call, so the gateway must be fast and must not become the outage.

A thin internal gateway is often enough. It injects auth, emits traces, and enforces a monthly budget.

Walking through a budget

Say a feature has a monthly budget of $10,000. Set alerts at 50%, 75% and 90% of the forecast, not at 100%:

monthly budget:      $10,000
alert at 50%:         $5,000  -> notify the feature owner
alert at 75%:         $7,500  -> review the top routes by cost
alert at 90%:         $9,000  -> freeze non-critical traffic

The early alerts give the team time to act. An alert at 100% is a surprise invoice, not a warning.

Four jobs of a gateway

• Keys and quotas. One key store, with quotas per feature and per tenant tier. A demo should not starve production support.
• Routing and fallbacks. Send each request to a model. If one provider has an outage, switch to another.
• Retries. Retry with jitter, so many clients do not all retry at the same instant.
• Spend and logs. Attach the team, feature and prompt version to every call.

The rules that keep it safe

Retries and fallbacks are where incidents start.

• Use idempotency keys on mutating tools. A blind retry of "create refund" can refund the customer twice.
• Test prompt compatibility before a fallback. Switching providers can pass health checks and still fail on quality.
• Quota per feature, not one global pool. One runaway loop should hit its own limit, not everyone's.

Common mistakes

• Each service holds its own key. Nobody can say who spent what.
• One global quota. A single bad loop takes down every feature.
• Blind retries on writes. Retry only operations that are safe to repeat.
• A gateway with no logs. Then it is one more hop with no benefit.

Check yourself

• Why alert at 50% and 75% instead of only at 100%? Early alerts give the team time to change the traffic before the money is gone.
• Why does a fallback need an eval? A health check only shows the service is up. It does not show that the answers are still good.

Key takeaways

• A gateway holds keys, enforces quotas, routes, retries and records spend in one place.
• Set quotas per feature and per tenant tier, not one global pool.
• Use idempotency keys before any retry on a write.
• Test each fallback model against your evals before it takes traffic.

Further learning

Next, return to the LLMOps hub for the weekly rhythm and the incident steps.