Message Queues

A message queue is an in-between buffer for producers that emit messages and consumers that process them. Three things you get from inserting one:

• Decoupling: producer doesn't need to know who consumes, or even if anyone is listening right now.
• Smoothing: bursts get absorbed by the queue instead of overwhelming consumers.
• Retries: a consumer that fails can simply re-receive the same message later.

What you give up: synchronous responses. The producer is told "message accepted", not "work done". For request-response semantics, queues are the wrong tool — use a synchronous API. For background work, async fan-out, and inter-service events, they're the right one.

Two delivery topologies:

• Point-to-point (work queue): each message is processed by exactly one consumer in the pool. Adding consumers spreads the work. Used for job processing ("send this email", "resize this image").
• Publish-subscribe (broadcast): each message is delivered to every subscriber. Used for events ("order placed", "user signed up") where multiple downstream services need to react.

Most modern systems (Kafka, RabbitMQ, NATS) support both with different topology configurations. The distinction matters because it's easy to confuse them and break things — sending a user_signed_up event point-to-point means only one downstream service sees it instead of all six that needed to.

When the network is unreliable, a queue can guarantee one of three semantics:

• At-most-once: messages might be lost; will never be duplicated. "Best effort" — used for high-volume telemetry where dropping a few is fine.
• At-least-once: messages will be delivered, but might be delivered more than once on retry. The most common default — used everywhere consumers can handle duplicates.
• Exactly-once: every message processed exactly once. The gold standard. Hard. Some systems (Kafka with transactions, e.g.) offer it at extra cost and only within their own boundaries.

In practice you almost always live with at-least-once + idempotent consumers. Designing consumers so reprocessing the same message produces the same result lets you tolerate the duplicates.

A message fails to process. Retried. Fails again. Retried. Fails again. Now what?

Without intervention you've built a poison-message loop: that one message starves the consumer of bandwidth for every other message. A dead letter queue is the standard fix: after N failed attempts, the broker moves the message to a separate queue. The consumer keeps processing; the bad message waits for human (or automated) inspection.

What to do with the DLQ:

• Alert on non-zero depth — "there are 47 messages we couldn't process" is something humans need to see.
• Inspect for bugs (often the producer sent malformed data).
• Replay after a fix if the root cause was transient.

DLQs are 10 minutes of configuration that save you from middle-of-the-night incidents.

Failures retry up to N times, then sideline.

Three systems you'll meet most often:

• Kafka: a distributed log. Messages persist; consumers track their position. Optimised for high throughput (millions/sec), event sourcing, replay. Heavier to operate. Good for: event streaming, audit logs, stream processing.
• RabbitMQ: a traditional broker with rich routing (exchanges, bindings). Lower throughput than Kafka but more flexible per-message routing. Good for: classic work queues, RPC-over-MQ, complex topologies.
• AWS SQS: managed; no ops; FIFO and standard variants. Less powerful (no real pub/sub without SNS in front) but operationally invisible. Good for: AWS shops needing decoupling without running infrastructure.

Decision rule: Kafka for event firehose, RabbitMQ for fan-out and routing, SQS when you want to outsource the problem.

Global message ordering across a queue is expensive. Most systems offer it within a subset (Kafka: per-partition; SQS: per-FIFO-message-group) but not globally — global ordering forces a single-consumer-at-a-time bottleneck.

How to design:

• Most events don't need ordering. Image-resize jobs, email-send jobs — process them in any order.
• When ordering matters, choose a partition key that aligns with the ordering boundary. "All events for user 42 must be ordered" → partition by user_id. Different users can interleave freely.
• Be deliberate about idempotency when you let things interleave. Two update_balance events processed in the wrong order can produce different totals; two set_balance_to_X events cannot.

If your design says "we need strict global order", you're probably looking for a database, not a queue.

Because at-least-once + duplicates is your daily reality, make every consumer idempotent. The minimum viable pattern:

def handle(message):
    # store a key per message; do work only if not seen
    if seen_table.exists(message.id):
        return ack(message)
    with db.transaction():
        do_the_work(message)
        seen_table.insert(message.id, ttl=7_days)
    ack(message)

Key points:

• The seen-set lives in the same DB as the work, so the insert is atomic.
• TTL matches your broker's max retry window — keep entries long enough that no legitimate retry beats them.
• For multi-step flows, give each step its own idempotency key.

Most outages of message-driven systems trace back to a consumer that wasn't idempotent and double-processed a payment / sent an email twice / decremented inventory twice. Adding this pattern early is cheap insurance.