Message Queuing

Message queuing separates the application that creates work, called the producer, from the application that performs it, called the consumer. Instead of calling a worker directly and waiting for the task to finish, the producer sends a message to an intermediary queue. The consumer retrieves the message when it has capacity. Successful acceptance by the queue means the work has been submitted, not that the business operation has completed.

For example, an image-upload application can place a message containing an image identifier and storage location onto a queue. Background workers retrieve these messages, generate thumbnails, and record the results. Upload requests can finish without waiting for image processing. During a traffic spike, the queue holds pending work while additional workers start. Storing a reference rather than the image itself also avoids unnecessary message size and transfer overhead.

Queues provide temporal decoupling: producers and consumers do not have to be available at the same moment. Durable queues can retain accepted messages during a worker outage, subject to their configured retention and service guarantees. However, a queue does not create processing capacity. If messages arrive faster than workers can handle them for a sustained period, the backlog and processing delay grow. Engineers should monitor message age, backlog size, processing failures, and downstream capacity rather than scaling workers without limits.

Reliable processing requires an explicit completion mechanism. Depending on the queue technology, a consumer acknowledges a message or deletes it after successful processing. A visibility timeout or delivery lock can temporarily prevent other workers from receiving the same message. If processing fails or the consumer crashes, the message can become available again. Set this interval appropriately and extend it when supported for long-running tasks.

Redelivery means consumers should generally be idempotent: processing the same message again should not repeat an unwanted business effect. For example, a payment worker can use a transaction identifier to detect an already completed charge. Standard queues commonly provide at-least-once delivery without strict ordering. FIFO options can provide ordering within defined scopes and deduplication under specified conditions, but they do not automatically guarantee exactly-once execution of external business operations.

Repeatedly failing messages should not retry forever. A dead-letter queue can isolate messages after a configured number of unsuccessful attempts so engineers can inspect, correct, and deliberately replay them. Message retention, access permissions, encryption, and sensitive-data handling also need explicit configuration. Queuing introduces additional request costs, operational monitoring, and eventual consistency compared with a simple synchronous call.

A work queue is not the same as a notification topic. With competing consumers, each queued message is intended to be handled by one worker, although redelivery remains possible. Publish-subscribe messaging distributes a published message to multiple subscriptions. On AWS, choose Amazon SQS when an application needs buffered background work; choose Amazon SNS when it needs fan-out or supported notification delivery, such as email or SMS. Combining a topic with separate queues gives each subscribing application its own processing backlog. Amazon CloudWatch alarms can publish to SNS to notify operators about conditions such as an aging queue backlog. A queue alone does not send a human-readable alert.