Backpressure and autoscaling

A queue can hide a problem for hours. Work keeps arriving, the queue keeps accepting it, and nothing looks broken until the oldest message is so old that nobody wants it. The question is how to notice the backlog growing, and what to do about it.

The big idea

Use the queue as a shock absorber for short spikes. It can't absorb a long gap in capacity. If work arrives faster than it's processed for long enough, the backlog grows forever, and message age grows with it.

Two things follow:

1. You must add workers when the backlog grows, or the system never catches up.
2. When you can't add workers, because the bottleneck is a shared database or a rate-limited API, you must slow the producers or drop work.

Choosing the scaling signal

CPU is a poor signal for queue workers. An I/O-bound worker waiting on a database looks idle, but it is busy. Scale on the queue instead.

• Backlog per worker. Divide visible messages by running workers. If each worker can handle 32 messages per second and you can tolerate 60 seconds of delay, a worker should not have more than 60 × 32 = 1,920 messages waiting for it. A backlog of 200,000 needs about 200,000 / 1,920 ≈ 105 workers.
• Age of the oldest message. This is the best service-level signal, because it matches what users feel. "Emails go out within 2 minutes" is a statement about age, not depth. In SQS it is ApproximateAgeOfOldestMessage.

Walking through a backlog

Traffic is 2,000 messages per second. You run 90 workers, each handling 32 messages per second at 0.25 s per message, which is 2,880 per second in total. Then a dependency slows down, and each message takes twice as long.

Before the slowdown:  90 workers × 32/s = 2,880/s  (more than the 2,000/s arriving)
After the slowdown:   90 workers × 16/s = 1,440/s  (each message now takes 0.5 s)
Arrivals:             2,000/s
Net backlog growth:   2,000 − 1,440 = 560/s  -> 33,600 more messages per minute

Adding workers helps only if the dependency can take the extra load. If the database is the slow part, 100 workers just means 100 workers waiting on the same database. The queue moves; it doesn't disappear.

What to do when you can't add workers

• Slow the producer. Apply rate limits so new work enters at the rate you can handle.
• Reject with a clear error. Return "try again later" to the caller instead of accepting work you will never finish.
• Shed low-priority work. Drop or defer the least valuable messages first, such as analytics refreshes, to keep the important ones moving.
• Put a TTL on messages that go stale. A push notification about yesterday's game score is worse than none.

Common mistakes

• Scaling on CPU. Workers look idle while they wait on I/O, so no scaling happens.
• Scaling without checking the bottleneck. More workers hitting the same slow database make the outage worse.
• No producer-side limit. The queue grows without bound, and the only signal is a full disk or an expired message.

Key takeaways

• A queue buffers work but does not create capacity. A long gap in capacity means a growing backlog.
• Scale workers on backlog per worker and oldest-message age, not CPU.
• Use Little's law to size the pool, then check the bottleneck before adding workers.
• When capacity is fixed, apply backpressure: slow producers, reject with clear errors, or shed low-priority work.

Further learning

Next, see how the most common queue products compare in choosing a queue product.