Backpressure and bounded queues

When producers send work faster than consumers can handle it, something has to give. An unbounded queue hides the problem. Work piles up in memory until the process runs out of RAM, and then it crashes, often long after the cause. Backpressure is the signal from the consumer back to the producer that says "slow down". A bounded queue is the simplest way to create that signal.

The big idea

A bounded queue has a fixed capacity. When it is full, a put either waits, is rejected, or drops work. The producer feels that delay or rejection right away, so it can slow down, retry later, or shed load. The limit moves the problem from "the server died at 3 a.m." to "one caller saw a slow request".

The capacity is a design choice. A useful rule: the queue should hold a few seconds of work at normal throughput. Anything bigger hides a capacity problem, and anything smaller makes normal bursts fail.

Walking through it

Python's asyncio.Queue with maxsize=2 makes the waiting visible. The producer must await each put, so it stops when the queue is full:

import asyncio

async def producer(q):
    for i in range(4):
        await q.put(i)                    # blocks when the queue is full
        print("put", i)
    await q.put(None)

async def consumer(q):
    while (item := await q.get()) is not None:
        print("got", item)
        await asyncio.sleep(0.01)         # the slow part

async def main():
    q = asyncio.Queue(maxsize=2)
    await asyncio.gather(producer(q), consumer(q))

asyncio.run(main())

Output:

put 0
put 1
got 0
put 2
got 1
put 3
got 2
got 3

The producer wrote 0 and 1, then waited. It couldn't put 2 until the consumer took 0 off the queue. The queue never holds more than two items, so memory stays fixed however fast the producer runs.

What to do when the queue is full

When backpressure can't be absorbed, pick one of three policies on purpose:

Blocking is the default in most languages. It is safe, but it can cascade. If a worker thread blocks on a full queue, its own callers wait, and so on up the chain. Rejecting with a clear error, such as HTTP 429 or 503, lets the system degrade one layer at a time.

Backpressure across services

The same idea scales to services. A worker pulling from a broker can limit its prefetch count, so it takes only as many messages as it can process. The broker then holds the rest, where they are durable and visible. The backpressure system design page covers this across services, including rate limits and load shedding.

Little's law helps size the bound. If the consumer handles 100 items per second and each waits up to 2 seconds in the queue, you need room for about 200 in flight. A bound far below that will reject normal bursts.

Common mistakes

• Using an unbounded queue, then discovering the memory problem during an incident.
• Setting the bound from guesswork, with no tie to throughput or latency.
• Dropping work silently, so the producer believes it succeeded.
• Blocking an entire thread pool on one slow downstream, which stalls unrelated requests.

Check yourself

• Why does a bounded queue keep memory fixed when the producer is faster? Answer: once the queue is full, the producer must wait or be rejected, so the queue never holds more than its capacity.
• Why is rejecting with an error often better than blocking? Answer: rejection is visible to the caller now, so the problem stays local instead of spreading up the chain as waiting threads.

Key takeaways

• Unbounded queues hide overload until memory runs out.
• A bounded queue turns overload into a visible wait or rejection.
• Choose block, reject or drop on purpose, and count every drop.
• Size the bound from throughput times acceptable wait.

Further learning

Next, return to the async I/O overview to see how these queues fit with event loops and goroutines.