What Is a Message Queue? – Definition and How It Works
Learn what a message queue does, how delivery guarantees and visibility timeouts work, and why crawlers use one.
A message queue holds work between a producer that creates it and a consumer that performs it, decoupling the two in time, rate, and failure.
1. What Decoupling Buys
Without a queue, a producer calls a consumer directly, so it waits for completion, fails when the consumer is down, and must match its rate.
With a queue: the producer enqueues and moves on, work survives a consumer restart, consumers scale independently, and a burst is absorbed as queue depth rather than as failures.
For a crawler this is the difference between a script that dies with its process and a system that resumes.
2. Delivery Guarantees
| Guarantee | Meaning | Reality |
|---|---|---|
| At most once | Never duplicated, may be lost | Rarely acceptable |
| At least once | Never lost, may be duplicated | The practical default |
| Exactly once | Neither lost nor duplicated | Only within narrow constraints |
Most systems provide at-least-once, and "exactly once" processing is achieved by pairing it with idempotent consumers. Any consumer must therefore tolerate seeing the same message twice — the duplicate is not an anomaly, it is the contract.
3. Visibility Timeout and Acknowledgement
The mechanism behind at-least-once: a consumer receives a message, it becomes invisible to others for a timeout, and the consumer acknowledges when done. No acknowledgement means the message reappears.
The failure mode is a timeout shorter than the work. The message reappears, a second consumer starts it, and both run — visible as duplicated work and as a message processed endlessly.
Two fixes: set the timeout above the realistic worst case, and extend it periodically from a long-running consumer.
4. Dead-Letter Queues
A message that fails repeatedly must go somewhere, or it blocks the queue forever.
A dead-letter queue receives messages exceeding a retry count. Two things make it useful rather than a landfill: something must alert on its depth, and the messages must carry enough context to diagnose the failure.
An unmonitored dead-letter queue is silent data loss with extra steps.
5. Queues Versus Streams
| Queue | Stream | |
|---|---|---|
| After consumption | Message removed | Record retained |
| Consumers | Compete for messages | Each reads independently |
| Replay | No | Yes, from an offset |
| Ordering | Usually per-queue, weak | Per-partition, strong |
| Suits | Task distribution | Event history, fan-out |
A crawl frontier is a queue: each URL should be fetched by exactly one worker. Collected results are better as a stream, so several consumers — storage, indexing, alerting — each read all of them.
6. A Crawl Frontier as a Queue
seeds → [url queue] → fetch workers → [result stream] → parse → store
↑ │
└──── discovered links ────┘
The loop is the point: consumers are also producers. Two consequences follow.
Deduplication must be external. The queue will happily hold the same URL a thousand times. A shared seen-set, checked before enqueueing, is what stops that.
Per-domain politeness cannot come from the queue. A single queue with many workers will hit one domain concurrently. Either shard queues by domain or enforce a per-domain concurrency limit in the workers.
7. Choosing One
| System | Model | Notes |
|---|---|---|
| Redis lists or streams | Both | Simplest if Redis is already present |
| RabbitMQ | Queue | Rich routing, mature |
| SQS | Queue | Managed, no operations |
| Kafka | Stream | High throughput, replay, partitions |
| Postgres table | Queue | SELECT ... FOR UPDATE SKIP LOCKED is genuinely fine at modest scale |
The last row deserves emphasis: a database-backed queue with SKIP LOCKED handles more load than most projects generate, and avoids operating another system.
8. Summary
A message queue decouples producers from consumers in time, rate, and failure, delivering at-least-once with a visibility timeout and acknowledgement, and routing repeated failures to a dead-letter queue.
For crawling, use a queue for the URL frontier and a stream for results — and remember the queue provides neither deduplication nor per-domain politeness, so both need a shared seen-set and a worker-side concurrency limit.