Acks and visibility timeouts

A worker takes a message off a queue. What stops another worker from taking the same message while the first one is still working? And what happens if the first worker dies? The answer is a lease: the broker lends the message to one worker for a limited time.

The big idea

Brokers handle in-flight messages in one of two ways.

• Hold until ack (RabbitMQ style). The message stays assigned to the consumer until it acks, or until the consumer's connection closes.
• Lease with a timeout (SQS and Pub/Sub style). When a worker receives a message, it becomes invisible for a visibility timeout (SQS) or ack deadline (Pub/Sub). If nobody deletes or acks it before then, it reappears for another worker.

The second style is more common in cloud queues, because workers can crash without the broker having to track a live connection.

Walking through a toy queue

Here is a tiny queue that hides a message for 30 seconds after someone receives it. It's a simplified model, not a real SDK.

class ToyQueue:
    """A queue where a received message hides itself for `timeout` seconds."""

    def __init__(self, timeout):
        self.timeout = timeout
        self.visible_at = {"order-17": 0}  # message id -> next time it can be received

    def receive(self, now):
        for msg, at in self.visible_at.items():
            if at <= now:
                self.visible_at[msg] = now + self.timeout  # hide it while a worker holds it
                return msg
        return None

    def delete(self, msg):
        del self.visible_at[msg]


q = ToyQueue(timeout=30)
print("t=0   worker 1 gets", q.receive(now=0))
print("t=10  worker 2 gets", q.receive(now=10))
print("t=30  worker 2 gets", q.receive(now=30))

Output:

t=0   worker 1 gets order-17
t=10  worker 2 gets None
t=30  worker 2 gets order-17

At t=10, the message is still hidden, so worker 2 gets nothing. At t=30, the lease expires and worker 2 receives it. If worker 1 is still working, the message is now being processed twice. That is the failure mode in one picture.

Setting the timeout

Set the visibility timeout above your worst-case processing time. A common starting point is 3 to 6 times the 99th percentile.

• Too short: the message reappears while the first worker is still busy. Duplicates appear exactly when the system is slow, which is the worst time for them.
• Too long: when a worker really crashes, the message stays hidden for the full timeout. Recovery is slow.

For jobs with unpredictable length, have the worker extend its lease while it works. In SQS that is ChangeMessageVisibility; in Pub/Sub it is modifyAckDeadline.

Common mistakes

• A stale delete. Worker 1 finishes after its lease expired and tries to delete the message. Real brokers reject this, because worker 2 now holds a newer lease. Don't assume the delete will succeed. Make the work idempotent so a duplicate is harmless.
• Acking before saving the result. The ack says "done". If you ack first and then the database write fails, the message is gone and the work is lost.
• Using CPU to decide if a worker is alive. A worker waiting on a database looks idle but is working. Use lease extensions or heartbeats instead.

Key takeaways

• A broker lends a message to a worker for a limited time, called a lease or visibility timeout.
• If the worker doesn't ack in time, the message is delivered again, possibly to another worker.
• Set the timeout above worst-case processing time, or extend the lease while work continues.
• Assume duplicates are possible, and make the handler safe to run twice.

Further learning

Next, see what happens when a message keeps failing in retries and dead-letter queues.