When Queues Stop Working: A Taxonomy of the Failure Modes That Matter

The broker being down is not the interesting case.

When RabbitMQ stops answering, or an SQS endpoint starts returning errors, everything about the situation is legible: connections fail, metrics flatline, alerts fire, and the fix is a known operational procedure. Unpleasant, but not confusing.

The failures worth having a taxonomy for are the ones where every dashboard is green. The broker is accepting publishes, consumers are connected, depth is low, and the system is nevertheless producing wrong answers. In every one of those cases the queue is doing precisely what its documentation says it will do.

Duplicate delivery, which is not a bug

At-least-once delivery means what it says. A message can arrive twice, and the most common route is not a broker fault at all: the consumer processed the message successfully and then failed to acknowledge it: the process died in the gap, or the ack timed out, or the network dropped it. The broker, having heard nothing, does the correct thing and redelivers.

So the duplicate is a feature of the contract, and the bug is always in the handler. Which makes the question concrete: is this handler idempotent, and against what key?

The answer is frequently “yes, mostly”, which means no. INSERT ... ON CONFLICT DO NOTHING on a natural key is idempotent. UPDATE balance SET amount = amount + 50 is not. Sending an email is not, and no amount of care in the queue configuration will make it so. The only fix is a record of “already sent” written in the same transaction as the send, which for an external provider means an idempotency key the provider honours.

A rule that survives contact with production: for every consumer, name the idempotency key out loud in review. If nobody can name it, there isn’t one.

Out-of-order delivery

Most queues make no ordering promise across a whole topic, and the ones that do make it per partition or per group only. Two messages published in sequence can arrive in either sequence if they land in different partitions or are handled by different consumers.

This is fine for most work and catastrophic for a specific shape: state transitions. user.updated followed by user.deleted, applied in the wrong order, resurrects a deleted user. The usual first attempt at a fix is to enforce global ordering, which works and costs you all your concurrency.

The better shape is to make handlers order-insensitive by carrying a version or a timestamp in the payload and discarding anything older than what has already been applied. It is a small amount of extra state per entity and it turns an ordering requirement into a comparison. Not free: the comparison needs a monotonic source, and wall clocks on different producers are not one.

The stall, which is the one that hides

Here is the failure mode that consistently takes longest to diagnose.

An ordered partition is assigned to exactly one consumer at a time; that is how ordering is delivered. If that consumer is alive but not progressing — blocked on a lock, waiting on an HTTP call with no timeout, stuck in a retry loop — then nothing behind the message it holds can be delivered to anyone. The broker sees a healthy connected consumer. The depth metric is a small number. The partition has not moved in forty minutes.

Nothing in the queue’s own health signals says anything is wrong, because from the queue’s point of view nothing is. The signal that catches it is consumer-side and specific: age of the oldest unacknowledged message, per partition. Depth does not catch it. Throughput does not catch it, if the other partitions are busy. Lag in the aggregate does not catch it either.

If a single metric had to be added to a queue-backed system, it would be that one.

An aside on timeouts nobody set

The proximate cause of a stall is almost always an HTTP client with no timeout configured. Python’s requests waits forever by default. Go’s http.DefaultClient has no timeout. Several JVM clients default to minutes. Consumers inherit those defaults and then hold a partition hostage for as long as some third party is willing to keep a socket open.

Whether the fix belongs in the client, in a middleware, or in a supervisor that kills a handler exceeding its budget is an argument with three defensible answers, and different teams have landed in different places. What is not defensible is the default.

Redelivery amplification

The failure that turns a small problem into an outage. A downstream dependency slows down; handlers start exceeding the visibility timeout; the broker redelivers; now there are two handlers per message hitting the slow dependency; it slows further; redelivery accelerates. The queue is functioning perfectly and is now a load amplifier pointed at the thing that was already struggling.

Three things break the cycle, and all three are needed: a visibility timeout larger than the slowest successful run, a bounded retry count with exponential backoff and jitter, and a dead-letter queue with an alert on non-zero depth. The third is the one that gets built and then never wired to anything, which is how work quietly accumulates in a place that feels like it was handled.

Where the taxonomy runs out

Every failure above is a single-message failure, and the fixes are all local to one handler. What none of them addresses is the case where the unit of work is not a message but a process: the shape a status-column table grows into: step four of seven, with three side effects already committed and a payload that has grown a status field, a retry_count and a last_step because the queue had nowhere else to put them.

That is the point where a queue is being asked to remember something it was explicitly designed to forget. Worth recognising early, because the alternative to recognising it is writing an orchestration engine one payload field at a time without ever deciding to.