Load shedding and degradation

When demand exceeds capacity, something has to give. Load shedding rejects some work early and cheaply, so the work you accept finishes within its deadline. Graceful degradation serves a worse but usable experience, such as a product page without recommendations, instead of an error. Both choose what gives, so the system does not collapse by accident.

The big idea

A server that accepts everything serves everything too slowly to be useful. Its goodput, meaning completed work that still matters, falls toward zero. A server that rejects the excess with a fast 503 keeps its goodput near its maximum.

Think of a restaurant that turns away walk-ins when the kitchen is full. Guests who get a table are served well, and nobody waits three hours. The analogy breaks in one place. A refusal is not free, because the client still spends time making the request and receiving the refusal. Shedding works only when the refusal is cheap.

Walking through it

Shedding works by checking the request before it consumes real resources. Four common techniques:

• Concurrency limits. Cap the requests in flight per server. Above the cap, reject immediately. Adaptive limiters, such as Netflix's concurrency-limits library, adjust the cap from latency signals.
• Queue-time limits. Record how long each request waited before a worker picked it up. If it waited longer than the client's timeout, drop it, because the client has already given up.
• Prioritization. Netflix's API gateway classifies requests by priority. Playback sits far above prefetching and logging, and the lowest priority is shed first under stress.
• Client-side throttling. Clients track how often the backend rejects them, and drop some requests locally, so rejected requests cost the backend almost nothing.

A queue-time check with an illustrative 2-second client timeout:

Request A: waited 0.3 s in queue → run it, answer arrives at 0.8 s   ✓ still useful
Request B: waited 2.6 s in queue → drop it, client gave up at 2.0 s  ✗ work saved

Request B would have cost a worker time and produced an answer nobody read.

Graceful degradation is the product side. Decide ahead of time which features are critical, such as checkout, login and playback. Wire the optional ones, such as personalized recommendations or spelling correction, behind flags that can switch them off. Serving a feed from a cache that is 5 minutes stale can be a good trade.

Common mistakes

• Accepting everything during overload. Every request then misses its deadline.
• Shedding without priorities. Playback and logging should not be treated the same.
• Untested fallbacks. Test them under load, or they will fail the first time they are needed.
• Degradation with no switch. Without a flag, you cannot turn an optional feature off in an incident.

Check yourself

1. Why is a fast 503 better than accepting a request you will answer late?

   Answer: a late answer after the client has timed out uses capacity and delivers nothing, so the server's useful output drops.

2. Why is "read from the database if the cache fails" risky?

   Answer: the database is sized for the cache's misses, not for all reads, so the fallback can overload it and take it down too.

Key takeaways

• Shed load early, cheaply and by priority, so accepted work still meets its deadline.
• Drop requests whose client has already given up, using queue-time checks.
• Plan degraded modes for optional features, and wire each one to a flag.
• Treat fallbacks as risky code, and test them as you would the primary path.

Further learning

Netflix engineers explain how prioritized load shedding in their API gateway keeps playback working, with production graphs:

Google's SRE book covers the client-side side of overload, and the Amazon Builders' Library covers load shedding and fallback:

Using load shedding to avoid overload (Amazon Builders' Library)How Amazon uses load shedding to stay available under overload, rejecting excess work early so the system keeps serving requests it can handle.aws.amazon.com

Avoiding fallback in distributed systems (Amazon Builders' Library)Why Amazon often avoids fallback logic in distributed systems, since untested fallback paths add risk, and what reliable alternatives to use.aws.amazon.com

Next, see how architecture limits the damage of any failure, in limiting blast radius.