At a basic level, a queue is a data structure capable of storing messages in a specific order. Thus, queuing is responsible for managing the storage of data in these queues. On the other hand, messaging refers to the process of communication between distributed components of an application using messages.
Queuing and messaging are crucial concepts that enable asynchronous communication among different components of AWS services, applications, and even with external cloud providers
Asynchronous communication enables the exchange of messages without the need to wait for an immediate response. This can be highly beneficial in creating an environment with decoupled components. This type of communication is predominant in microservices and event-driven architectures available on AWS.
AWS offers several services capable of queuing and messaging, each differing in their methods of sending and receiving communications, as well as how they can be integrated into application architectures.
The main messaging and queing services offered by AWS are: SQS, SNS, EventBridge and Amazon MQ.
How does Amazon Simple Queue Service (SQS) work
In distributed systems, SQS acts as the middleware for messaging, involving producers (sending messages to the queue) and consumers (receiving messages). Messages are redundantly stored across SQS servers to ensure reliability and availability.
SQS operates on a pull communication model, where producers add messages to the queue without immediate knowledge of consumer processing, and consumers actively poll the queue to retrieve messages. This model facilitates component decoupling, allowing messages to remain in the queue until consumers are ready to process them
The message lifecycle involves a producer sending a message to the queue, where it's stored across servers. A consumer then processes these messages and can delete them to prevent reprocessing. SQS also automatically deletes messages that exceed the maximum retention period.
SQS is versatile, suitable for scenarios like decoupling user requests from intensive background tasks, allocating tasks across workers, or batch processing for databases. FIFO queues are essential for operations where order and no duplicates are crucial, like e-commerce systems or integrating with third-party services.
The default maximum size of messages in SQS is 256 KB.
SQS offers three types of queues:
Standard Queues
For high throughput and at-least-once delivery, with best-effort ordering.
FIFO Queues
For scenarios where order and uniqueness are critical, offering exactly-once processing and the option for high throughput mode.
Dead-letter queues
Serve as secondary queues to handle messages that the primary queue's consumers cannot process. They are defined by a "redrive policy" set by the primary queue's owner, which directs messages to the dead-letter queue after a specified number of unsuccessful processing attempts (maxReceiveCount).
These queues are useful for isolating problematic messages, allowing for analysis and potential reprocessing. It's important that the dead-letter queue matches the primary queue in type (FIFO or standard) and is located in the same AWS account and region. Additionally, a single dead-letter queue can be used by multiple primary queues.
How does Amazon Simple Notification Service (SNS) work
Amazon Simple Notification Service (SNS) operates on a publish-subscribe model, allowing applications to send messages to topics, which subscribers can then receive. The process involves:
Creating a Topic
Developers create an SNS topic, serving as a unique identifier for a specific type of notification.
Publishing Messages
Publishers send messages to the topic, which SNS then delivers instantly to all subscribers.
Subscribing to Topics
Subscribers, such as AWS services, HTTP endpoints, or applications, choose topics of interest. They specify endpoints like SQS queues or Lambda functions, where SNS delivers messages.
SNS is widely used for event notifications, mobile push notifications, microservices integration, and any scenario requiring asynchronous messaging.
SNS supports various endpoints for subscriptions, including HTTP/HTTPS webhooks, email addresses, mobile devices for push notifications, SMS phones, AWS services like Lambda and SQS queues, and CloudWatch Logs for logging.
Additionally, Amazon SNS FIFO topics can be used with Amazon SQS FIFO queues for strict message ordering and deduplication, suitable for applications requiring data consistency in near-real-time.
Standard SQS queues subscribed to SNS FIFO topics provide best-effort ordering and at-least-once delivery.
The default maximum size of messages in Amazon SNS is 256 KB, but it can be increased to 2 GB when the messages are stored in Amazon S3. This is achieved using the Amazon SNS Extended Client Library for Java, or through the AWS CLI.
Common Amazon SNS Scenarios:
Application Integration
The Fanout scenario replicates a message from an SNS topic to multiple endpoints like Kinesis streams, SQS queues, or Lambda functions for parallel processing. For instance, this can be used for order processing and data warehousing.
Application Alerts
SNS sends alerts triggered by predefined thresholds via SMS or email, useful for monitoring AWS resources like EC2 Auto Scaling or S3 uploads.
User Notifications
Send emails or SMS messages for e-commerce confirmations or group alerts.
Mobile Push Notifications
Deliver messages directly to mobile apps, such as update notifications with download links.
How does work Amazon EventBridge work
EventBridge acts as a central hub for collecting events from multiple sources such as AWS services, applications, or third-party software, with typical sources including S3, DynamoDB, and custom applications. These events are then directed to various targets, such as Lambda functions, SNS topics, or Step Functions, facilitating the creation of loosely coupled applications where components communicate through event production and response.
Key functionalities of EventBridge include filtering rules for event processing, transformations for altering event structure, and scheduling capabilities to align events with specific rules.
AWS EventBridge processes events using two methods: event buses and pipes.
Event Buses
These are routers that receive events from various sources and direct them to one or more targets. Ideal for routing numerous events to multiple destinations, they also offer optional event transformation before delivery. Your account includes a default event bus for AWS services, and you can create custom or partner event buses for specific events or integration with SaaS partners.
Event buses are often used as brokers between different workloads or to divide event traffic in applications, and can even aggregate events from multiple buses to a centralized bus.
EventBridge Pipes
Designed for point-to-point integrations, pipes handle events from a single source, delivering them to a single target with options for advanced transformations and enrichments. Pipes can be used independently or in conjunction with event buses, such as routing events from a DynamoDB stream to an event bus, which then distributes them according to specified rules. Setting up a pipe involves choosing a source, adding filters, defining enrichments, and selecting a target. Pipes are charged only for events matching the configured filter, maintaining order if events are batched.
For example, in an e-commerce system, a pipe could route order messages from an Amazon SQS queue, enrich the data with customer information via an EventBridge API Destination, and then send the enriched data to an AWS Step Functions state machine for processing.
References
What is Amazon Simple Queue Service? - Amazon Simple Queue Service
Message Queuing Service - Amazon Simple Queue Service - AWS
Amazon SQS Features | Message Queuing Service | AWS
What is Amazon SNS? - Amazon Simple Notification Service
Amazon Simple Notification Service (SNS) FAQs | Messaging Service | AWS