Noisy neighbors

A noisy neighbor is a tenant whose load slows the service for everyone else. A customer importing 50 million rows, running a report across five years of data, or calling the API from a misconfigured loop can each set the latency for every tenant on the same system. Without controls, the biggest tenant sets everyone's experience. The fix is to cap each tenant's share of shared resources before one tenant can take all of them.

The big idea

Picture a checkout with one line for the whole shop. One customer unloads a cart of 500 items, and everyone behind them waits. Give each customer a line, serve the lines in turn, and a small purchase gets through quickly even while the big cart is still being scanned. That is fair queuing. The trade-off is that the big cart takes longer to finish.

Walking through it

The toolkit, roughly from cheapest to most expensive:

• Per-tenant rate limits and quotas at the API, sized by plan. See rate limiting.
• Per-tenant concurrency limits. Cap in-flight requests or database connections per tenant. A tenant running 20 slow reports at once can't take more than 20 of the pool's 200 connections.
• Fair queuing for background work. Queue jobs per tenant, and have workers take turns across tenant queues, or weight the turns by plan.
• Separate pools for large tenants. Route the biggest tenants to dedicated workers or shards, so their spikes hit only their own capacity. This is a bulkhead keyed on tenant.
• Shuffle sharding. Give each tenant a random subset of workers. A misbehaving tenant then overlaps fully with only a small fraction of the others. Resilience has the math.

Here is the worked example. A job system has 50 workers and serves 2,000 tenants. Tenant X enqueues 500,000 export jobs, each taking 10 seconds.

FIFO queue:    500,000 jobs x 10 s / 50 workers = 100,000 s = about 28 hours
               every other tenant's jobs wait behind X's

Fair queuing:  workers take turns across tenants that have pending jobs
               if 40 other tenants have work, X is one of 41 busy tenants
               X gets about 1/41 of the workers: 50 / 41 = about 1.2 workers
               others see near-normal latency; X's backlog drains slowly,
               and X can be offered a dedicated pool at a price

The FIFO queue delays everyone for about 28 hours. Fair queuing protects the others and slows X's backlog, which is the intended outcome. The cost falls on the tenant who caused it.

Common mistakes

• One FIFO job queue for all tenants. A single tenant can block everyone for hours.
• Limiting only requests per second. A few slow requests can hold a whole pool of connections. Add concurrency caps.
• Launching expensive features without limits. Imports, exports and AI features need per-tenant quotas before the first incident, not after.

Check yourself

1. With 41 busy tenants, including X, what share of the workers does X get under fair queuing?

   About 1/41, roughly 1.2 of the 50 workers.

2. A tenant holds 20 concurrent slow reports. Why is a concurrency cap needed, not just a rate limit?

   A rate limit allows many slow requests to be in flight at once. The concurrency cap keeps the tenant to 20 of the pool's 200 connections.

Key takeaways

• A noisy neighbor is any tenant whose load slows others. Cap each tenant's share.
• Fair queuing across tenant queues stops one large backlog from blocking everyone.
• Concurrency caps protect shared pools in a way that request rate limits don't.
• Dedicated pools and shuffle sharding contain the biggest tenants.

Further learning

The Amazon Builders' Library article on fairness in multi-tenant systems describes per-tenant admission and quotas, the approach behind AWS services.

Fairness in multi-tenant systems (Amazon Builders' Library)How Amazon keeps shared, multi-tenant services fair, using throttling, quotas, and workload isolation so one tenant's traffic can't starve others.aws.amazon.com

Next: moving tenants and sharding, which covers how tenants are split across databases.