Cloud concept · Updated

Message Queuing - Message queuing is asynchronous communication in which producers place messages in a queue for consumers to retrieve and process independently.

In short

A message queue buffers work between applications so that producers and consumers do not need to run at the same speed or be available simultaneously. This helps absorb traffic spikes and isolate failures, but requires careful handling of retries, duplicate messages, processing delays, and retention limits.

Cloud Concepts Hub →

6 key aspects of Message Queuing

Message queuing provides asynchronous work buffering and failure isolation, with reliability determined by acknowledgment handling, idempotency, delivery semantics, and operational controls.

01
Asynchronous decoupling

Producers submit work without waiting for consumers to complete it. Applications must distinguish submission success from processing success and provide a separate way to track completion when required.

02
Buffering and load leveling

A queue absorbs temporary differences between arrival rate and processing capacity. Monitor the age of pending messages and scale consumers within downstream limits; buffering alone cannot fix a sustained capacity shortfall.

03
Acknowledgment and redelivery

Consumers acknowledge or delete messages after successful processing. Failed or interrupted processing can trigger redelivery, so consumers should use idempotency controls to avoid duplicate business effects.

04
Delivery and ordering semantics

Delivery guarantees and ordering depend on the queue type and configuration. Select FIFO behavior only where ordering is required, and do not confuse broker deduplication with exactly-once execution across databases or external APIs.

05
Failure isolation and recovery

Retries handle transient failures, while dead-letter queues isolate messages that repeatedly fail. Define retry limits, investigate failures, and replay messages only after addressing the cause.

06
Work distribution versus notifications

Competing consumers share work from a queue, whereas publish-subscribe messaging distributes messages to multiple subscriptions. Use notification delivery for alerts and combine topics with queues when multiple applications each need durable, independent processing.

Cloud services for Message Queuing

Amazon SQS buffers application work, while AWS Lambda or other workers retrieve and process messages. Amazon SNS adds fan-out by delivering a published message to separate SQS queues, allowing each application to maintain its own backlog and retry behavior. SNS can also deliver supported human-facing notifications. Amazon CloudWatch monitors queue health and can trigger SNS notifications when an alarm changes state. Amazon MQ is an alternative messaging foundation when existing applications require supported broker protocols or compatibility with ActiveMQ Classic or RabbitMQ; it is not a required component of an SQS architecture.

Amazon MQAmazon MQ

Provides managed message brokers for Apache ActiveMQ Classic and RabbitMQ. It is relevant when applications need broker protocols or compatibility with existing messaging software rather than the SQS API.

AWS LambdaAWS Lambda

Can process SQS messages through an event source mapping that polls the queue and invokes a function with message batches. Functions must handle retries and duplicates, and partial batch responses can prevent successfully processed messages from being retried alongside failed messages.

Amazon CloudWatchAmazon CloudWatch

Monitors queue metrics such as visible message count and the approximate age of the oldest message. CloudWatch alarms can publish notifications to SNS when backlog or processing delays cross configured thresholds.

Message Queuing exam questions

Real practice exam questions on Message Queuing, with the correct answer and its explanation. Answer them with every option in the exam simulator.

CLF-C02

A document-processing application must accept new processing requests even when its worker applications are offline for several hours. Requests must remain available until a worker can retrieve and process them. The company wants a fully managed solution that does not require the submission application to wait for a worker. Which solution best meets these requirements?

Show answerCORRECT ANSWER

Store requests in an Amazon SQS queue with an appropriate retention period, and have workers retrieve and delete messages after successful processing.

Amazon SQS durably buffers requests independently of worker availability. Workers can retrieve messages when they are ready and delete them after successful processing. The retention period must cover the expected outage and processing delay.

DEA-C01

A data pipeline places failure alerts in an Amazon SQS standard queue. Several worker instances consume the alerts and call an external incident-management API. The API supports an idempotency key: repeated requests with the same key create only one incident. Each alert contains a unique, stable alert ID.

A worker can crash before calling the API or after the API creates an incident but before the worker deletes the message. Workers recover and process pending alerts within the queue's retention period. The data engineer must prevent worker crashes and message redelivery from causing lost alerts or multiple incidents for the same alert.

Which processing approach meets these requirements?

Show answerCORRECT ANSWER

Call the incident-management API using the alert ID as the idempotency key. Delete the message only after the API confirms successful incident creation or returns the previously created incident.

Deleting only after confirmed success allows an unfinished alert to be retried after a worker crash. Using the stable alert ID as the idempotency key prevents an additional incident if the API succeeded but the worker crashed before deleting the message.

AIP-C01

A company is building an asynchronous document-summarization API for custom clients. Clients submit JSON containing a job ID and a prompt. Prompts can contain characters such as &, +, =, newlines, and non-ASCII text. Workers consume jobs from an Amazon SQS queue and invoke Amazon Bedrock independently of the submission request.

The company wants Amazon API Gateway to submit validated requests directly to the queue without invoking a Lambda function. The complete JSON request body must be preserved as the SQS message body. Clients must receive HTTP 202 only after SQS successfully accepts the message; queue submission failures must produce an error response. HTTP 202 must not imply that summarization has completed.

The API Gateway REST API already has authentication and JSON body validation configured. Which integration configuration meets these requirements?

Show answerCORRECT ANSWER

Configure an AWS service integration for SQS SendMessage with an execution role that permits sqs:SendMessage. Set the integration Content-Type to application/x-www-form-urlencoded and use Action=SendMessage&MessageBody=$util.urlEncode($input.body). Map successful SQS responses to HTTP 202 and unsuccessful responses to appropriate error statuses.

The SQS Query API expects form-encoded parameters. URL-encoding the entire JSON body prevents prompt characters from being interpreted as form delimiters or encoding syntax. SQS decodes the parameter and stores the original JSON as the message body. Returning 202 only for successful queue submissions acknowledges durable acceptance, not completion of model inference.

Practice Message Queuing in a full exam

Questions on Message Queuing in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Message Queuing

3 min read

Message queuing separates the application that creates work, called the producer, from the application that performs it, called the consumer. Instead of calling a worker directly and waiting for the task to finish, the producer sends a message to an intermediary queue. The consumer retrieves the message when it has capacity. Successful acceptance by the queue means the work has been submitted, not that the business operation has completed.

For example, an image-upload application can place a message containing an image identifier and storage location onto a queue. Background workers retrieve these messages, generate thumbnails, and record the results. Upload requests can finish without waiting for image processing. During a traffic spike, the queue holds pending work while additional workers start. Storing a reference rather than the image itself also avoids unnecessary message size and transfer overhead.

Queues provide temporal decoupling: producers and consumers do not have to be available at the same moment. Durable queues can retain accepted messages during a worker outage, subject to their configured retention and service guarantees. However, a queue does not create processing capacity. If messages arrive faster than workers can handle them for a sustained period, the backlog and processing delay grow. Engineers should monitor message age, backlog size, processing failures, and downstream capacity rather than scaling workers without limits.

Reliable processing requires an explicit completion mechanism. Depending on the queue technology, a consumer acknowledges a message or deletes it after successful processing. A visibility timeout or delivery lock can temporarily prevent other workers from receiving the same message. If processing fails or the consumer crashes, the message can become available again. Set this interval appropriately and extend it when supported for long-running tasks.

Redelivery means consumers should generally be idempotent: processing the same message again should not repeat an unwanted business effect. For example, a payment worker can use a transaction identifier to detect an already completed charge. Standard queues commonly provide at-least-once delivery without strict ordering. FIFO options can provide ordering within defined scopes and deduplication under specified conditions, but they do not automatically guarantee exactly-once execution of external business operations.

Repeatedly failing messages should not retry forever. A dead-letter queue can isolate messages after a configured number of unsuccessful attempts so engineers can inspect, correct, and deliberately replay them. Message retention, access permissions, encryption, and sensitive-data handling also need explicit configuration. Queuing introduces additional request costs, operational monitoring, and eventual consistency compared with a simple synchronous call.

A work queue is not the same as a notification topic. With competing consumers, each queued message is intended to be handled by one worker, although redelivery remains possible. Publish-subscribe messaging distributes a published message to multiple subscriptions. On AWS, choose Amazon SQS when an application needs buffered background work; choose Amazon SNS when it needs fan-out or supported notification delivery, such as email or SMS. Combining a topic with separate queues gives each subscribing application its own processing backlog. Amazon CloudWatch alarms can publish to SNS to notify operators about conditions such as an aging queue backlog. A queue alone does not send a human-readable alert.

Learn Message Queuing in lessons

12 short lessons on Message Queuing in 5 certification courses.

and 6 more lessons

Open the lessons