Bulkheads and isolation

A bulkhead is a limit on how much of a shared resource one part of the system can use. Ships have watertight compartments, so a breach floods one compartment, not the whole hull. A software bulkhead gives each dependency or workload its own bounded pool of threads, connections or slots. When one pool fails, the others keep working.

The big idea

Revisit the thread pool from cascading failures. If the recommendations client can use only 20 of the 200 threads, a recommendations outage costs 10% of capacity at most. Checkout keeps running on the other 180 threads. Only requests that need recommendations are affected.

Bulkheads appear at every scale:

The trade-off is utilization. A reserved pool cannot lend its idle threads to another pool. You need more total capacity than a single shared pool would need. That cost is usually worth paying on critical paths.

Walking through it

This sketch uses a semaphore as the bulkhead. It allows two recommendation calls at a time, where production would use 20. Checkout does not need the semaphore:

import asyncio, time

recs_slots = asyncio.Semaphore(2)  # bulkhead: at most 2 recommendation calls at once

async def recommendations(i):
    async with recs_slots:
        await asyncio.sleep(1)     # the slow dependency

async def checkout(i):
    await asyncio.sleep(0.1)       # does not need recommendations

async def main():
    start = time.monotonic()
    async def timed(name, coro):
        await coro
        print(f"{name} finished at {time.monotonic() - start:.1f}s")
    await asyncio.gather(
        *(timed(f"recs {i}", recommendations(i)) for i in range(4)),
        *(timed(f"checkout {i}", checkout(i)) for i in range(2)),
    )

asyncio.run(main())

Output:

checkout 0 finished at 0.1s
checkout 1 finished at 0.1s
recs 0 finished at 1.0s
recs 1 finished at 1.0s
recs 2 finished at 2.0s
recs 3 finished at 2.0s

Checkout finishes in 0.1 seconds, even though four recommendation calls are queued behind the bulkhead. Without the bulkhead, those calls would hold every slot the checkout needs.

Common mistakes

• One shared pool for everything. One slow dependency can consume every thread.
• Sizing bulkheads too small. Capacity sits idle while the critical path waits.
• Not monitoring pool saturation. A full bulkhead is a signal, so alert on it.

Check yourself

1. A 200-thread service gives recommendations a pool of 20. How much capacity does a total recommendations outage cost?

   Answer: 10% (20 of 200 threads). Only requests that need recommendations are affected.

2. Why do bulkheads usually need more total capacity than a shared pool?

   Answer: a reserved pool cannot lend its spare capacity to another pool, so idle threads in one pool cannot help another.

Key takeaways

• A bulkhead caps how much of a shared resource one dependency or workload can use.
• One failing dependency then costs only its own share of capacity.
• The price is lower utilization, because reserved capacity cannot be shared.
• Put bulkheads around non-critical dependencies first, so their failure degrades a feature, not the service.

Further learning

Bulkheads stop a failure from spreading, but the service still has to choose what to drop when demand is too high. Read load shedding and degradation next.