Amazon Managed Streaming for Apache Kafka
Cloud service · AWS · Updated

Amazon Managed Streaming for Apache Kafka

In short

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data without having to manage the Kafka infrastructure.

Cloud Services Hub →

7 key features of Amazon Managed Streaming for Apache Kafka

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that simplifies building and running applications that use Apache Kafka to process streaming data, offering robust scalability, high-throughput, and low-latency streaming capabilities with built-in security features.

01
Fully Managed Service

Amazon MSK is a fully managed service that takes care of the underlying infrastructure, including the provisioning, configuration, and maintenance of Apache Kafka clusters. This allows developers to focus on building applications rather than managing complex Kafka infrastructure.

02
Serverless Option

Amazon MSK now offers a serverless option, allowing users to automatically scale the throughput and storage with no capacity planning required. This eliminates the need to manage and operate Kafka clusters, reducing operational complexity and cost.

03
Highly Available and Secure

MSK ensures high availability by automatically replicating data across multiple AWS Availability Zones. It also integrates with AWS security services such as AWS Identity and Access Management (IAM), Amazon VPC, and AWS Key Management Service (KMS), offering encryption in transit and at rest to ensure data is secured.

04
Compatible with Open Source Tools

Being fully compatible with Apache Kafka, MSK allows you to easily migrate your existing Kafka applications to AWS without needing to change your code. It also supports Apache Kafka community and third-party tools, which facilitates seamless integration with your existing applications.

05
Monitoring and Logging

Amazon MSK integrates with Amazon CloudWatch for monitoring and Amazon CloudWatch Logs for logging, providing deep insights into the performance and health of your Kafka clusters. This enables you to quickly detect and respond to issues.

06
Easy to Start

With just a few clicks in the AWS Management Console, you can create and configure an Amazon MSK cluster. This simplifies the process of setting up Apache Kafka on AWS and significantly reduces the time to get started.

07
KRaft Metadata Management

Amazon MSK supports KRaft metadata management, which propagates cluster metadata among Kafka controllers instead of ZooKeeper nodes. KRaft controllers are included at no additional cost and require no additional setup or management.

Where Amazon Managed Streaming for Apache Kafka is used

Amazon Managed Streaming for Apache Kafka is used for building and running applications that process large streams of data in real time, such as log aggregation, stream processing, and event sourcing.

  1. 01
    Real-time Data Processing

    Amazon MSK is ideal for real-time data processing applications. It allows businesses to easily collect, process, and analyze streaming data in real time. This enables applications such as real-time analytics, dashboards, and alerting systems that need to process data the moment it arrives, providing timely insights and actions.

  2. 02
    Log Aggregation

    Organizations can use Amazon MSK to aggregate logs from various sources such as applications, systems, and services. This centralized approach to log aggregation simplifies log analysis and monitoring, making it easier to troubleshoot issues, monitor system health, and understand application behaviors across a distributed architecture.

  3. 03
    Event-Driven Architecture

    Amazon MSK supports building event-driven architectures, where producers generate events and consumers react to them, decoupling the producers and consumers of data. This architecture enhances scalability, flexibility, and responsiveness, enabling developers to build loosely coupled systems that can easily adapt to changing business requirements.

  4. 04
    Message Brokering

    MSK facilitates smooth message brokering between different parts of applications or between different applications. By acting as a scalable and highly available message queue, it can manage high-throughput, fault-tolerant messaging for inter-process communication, thereby ensuring reliable data delivery in complex distributed systems.

  5. 05
    Stream Analytics

    With Amazon MSK, organizations can perform stream analytics on data in motion. This is valuable for use cases that require real-time analysis of data streams to detect patterns, anomalies or to aggregate data over time. Stream analytics can be applied in scenarios such as fraud detection, monitoring IoT device states, or real-time marketing analytics.

How Amazon Managed Streaming for Apache Kafka is priced

Amazon Managed Streaming for Apache Kafka (MSK) pricing is based on the chosen cluster configuration, including the number and type of broker nodes, the storage provisioned, and data transfer costs, with no upfront investment required.

On-Demand Pricing

Pay for the capacity you use by the hour without any long-term commitments or upfront payments. Pricing varies based on the instance type and the region in which your MSK cluster is running.

Data Transfer Pricing

Charges apply for data transferred in and out of Amazon MSK across the internet or between AWS regions. Data transfer within the same AWS region (inter-zone) involving MSK is also subject to charges, but transferring data between nodes within the same MSK cluster does not incur costs.

Standard Broker Storage

Standard broker storage is charged for provisioned capacity in GB-months, calculated from the GB provisioned each hour.

Express Broker Storage and Ingestion

Express brokers charge for storage actually used in GB-months and for data written at a per-GB rate billed at per-byte resolution.

Provisioned Storage Throughput

Optional additional storage throughput is charged by provisioned MB/s per month, calculated from the throughput provisioned each hour per broker.

Tiered Storage

Using the low-cost storage tier adds charges for low-cost storage capacity and data retrieved from that tier.

Multi-VPC Private Connectivity

Optional multi-VPC private connectivity incurs an hourly charge per cluster and enabled authentication scheme, plus a per-GB data-processing charge. Standard AWS PrivateLink charges also apply to the managed VPC connections used by clients.

MSK Serverless

MSK Serverless charges per cluster-hour and partition-hour, per GB written and read, and for storage consumed.

MSK Connect

MSK Connect charges an hourly connector-usage rate billed at one-second resolution, based on worker count and worker size in MSK Connect Units (MCUs). Each MCU provides 1 vCPU and 4 GB of memory.

MSK Replicator

MSK Replicator charges an hourly rate for each Replicator and a per-GB rate for data processed, including metadata and data filtered to prevent replication loops when identical topic names are used.

Data Delivery

Native delivery from MSK to streaming tables in Amazon S3 Tables or general purpose Amazon S3 buckets is charged by the volume delivered, with pricing varying by destination type and no setup fees, minimum commitments, or upfront costs.

Services Amazon Managed Streaming for Apache Kafka works with

MSK Connect lets you use Kafka Connect to configure data delivery to Amazon S3 and Amazon S3 Tables.

MSK Replicator can migrate data from existing Kafka deployments to Express brokers on Amazon MSK.

Amazon AuroraAmazon Aurora

Amazon MSK Connect can run Debezium source connectors that capture changes from Aurora databases and publish them to Kafka topics. This supports streaming inserts, updates, and deletes to downstream consumers.

Amazon CloudWatchAmazon CloudWatch

Amazon MSK publishes cluster and broker metrics to CloudWatch for monitoring and alarms. MSK Provisioned clusters and MSK Connect connectors can also deliver logs to CloudWatch Logs.

Amazon Data FirehoseAmazon Data Firehose

Amazon MSK can serve as a source for Amazon Data Firehose delivery streams. Firehose reads records from Kafka topics and delivers them to supported destinations, with optional transformation, buffering, and format conversion.

Amazon DocumentDB (with MongoDB compatibility)Amazon DocumentDB (with MongoDB compatibility)

Amazon MSK Connect can run a MongoDB Kafka source connector configured to read Amazon DocumentDB change streams. Document changes are published to Kafka topics for downstream processing.

Amazon DynamoDBAmazon DynamoDB

Amazon MSK Connect can run a DynamoDB sink connector to write Kafka topic records into DynamoDB tables. This lets applications maintain a low-latency key-value or document view of incoming events.

Amazon EC2Amazon EC2

Amazon MSK Provisioned runs its brokers on managed EC2 compute capacity. Broker instance types determine the compute and memory available to the cluster, while MSK handles provisioning and maintenance.

Amazon Elastic Block Store (Amazon EBS)Amazon Elastic Block Store (Amazon EBS)

Amazon MSK Standard brokers use EBS volumes for Kafka log storage. You can increase broker storage capacity and, for supported configurations, provision storage throughput.

Amazon EMRAmazon EMR

Amazon MSK can supply streaming records to Spark applications running on Amazon EMR. Spark Structured Streaming can consume Kafka topics, transform the records, and publish results back to MSK or another destination.

Amazon EventBridgeAmazon EventBridge

Amazon MSK is a supported source for EventBridge Pipes. A pipe can consume Kafka records, filter or enrich them, and deliver them to a supported target without requiring a custom consumer application.

Amazon Managed GrafanaAmazon Managed Grafana

Amazon MSK metrics can be visualized in Amazon Managed Grafana using its CloudWatch data source. Dashboards can track broker CPU, storage utilization, throughput, and consumer lag.

What is Amazon Managed Streaming for Apache Kafka

2 min read

Amazon Managed Streaming for Apache Kafka, often abbreviated as Amazon MSK, is an innovative service that provides a fully managed, highly available, and secure environment for running Apache Kafka. Apache Kafka, a prominent open-source platform, is designed for building real-time streaming data pipelines and applications. 

Amazon MSK offers Provisioned and Serverless deployment options. With MSK Provisioned, you choose Standard or Express brokers and scale the cluster in units of brokers. MSK Serverless abstracts cluster scaling and management, so applications run without requiring you to provision, configure, or optimize clusters.

By handling data streams, it enables developers and organizations to publish, subscribe to, store, and process streams of records in real time. However, managing an Apache Kafka cluster in production can be complex and resource-intensive, requiring expertise in configuring, maintaining, and securing clusters, which is where Amazon MSK steps in to simplify the process. Amazon MSK essentially lifts the heavy lifting involved in managing Kafka clusters by automating tasks such as hardware provisioning, software patching, setup, and configuration. This automation serves to drastically reduce the operational load and complexity for developers and businesses, allowing them to focus more on the application development aspect rather than the underlying infrastructure management.

Furthermore, Amazon MSK is designed to be highly available and secure by default, providing multiple levels of security, including encryption in-transit and at-rest, VPC network isolation, IAM access controls, and private connectivity to help ensure that data is securely managed and processed. One of the significant benefits of Amazon MSK is its seamless integration with other AWS services, which allows developers to easily build and run applications that leverage rich data analytics, AI/ML, and other AWS services. For instance, integration with Amazon Kinesis makes it easier to ingest, buffer, and process data in real time, enhancing the capabilities of real-time analytics applications. 

Additionally, Amazon CloudWatch integration provides comprehensive monitoring and logging capabilities, enabling users to efficiently observe the health and performance of their Kafka clusters and applications. 

Cost management is another area where Amazon MSK shines. It offers a pay-as-you-go pricing model, which means customers pay only for the resources they use without upfront costs or commitments. This model provides the flexibility to scale resources up or down based on demand, optimizing cost efficiency. 

In summary, Amazon Managed Streaming for Apache Kafka offers a fully managed service that simplifies the deployment, management, and scaling of Apache Kafka clusters. It reduces the operational burden on users while ensuring high availability, security, and integration with other AWS services, thereby enabling organizations to focus on innovation and application development rather than infrastructure management.

Concepts related to Amazon Managed Streaming for Apache Kafka

Amazon Managed Streaming for Apache Kafka exam questions

Real practice exam questions on Amazon Managed Streaming for Apache Kafka, with the correct answer and its explanation. Answer them with every option in the exam simulator.

SAP-C03

A logistics company publishes immutable shipment events as JSON records to Amazon Managed Streaming for Apache Kafka (Amazon MSK). Each event contains a carrier ID. Operations analysts need a SQL dashboard that joins recent events with carrier-reference data and targets event freshness of less than one minute.

Each night, a source system delivers a complete replacement carrier-reference dataset to Amazon S3. Existing carriers can have changed attributes, and some carriers are removed. Every dashboard query must see either the complete previous reference dataset or the complete replacement dataset, never an empty, partially loaded, or mixed-version dataset. The joins must use the reference dataset current when the query starts, including for events ingested before the nightly replacement.

The company has sufficient Amazon Redshift capacity and network connectivity. It wants native managed ingestion without maintaining a separate stream-processing application. Streaming events must not pass through an intermediate S3 delivery pipeline.

Which design meets these requirements?

Show answerCORRECT ANSWER

Create an external schema for MSK and an automatically refreshed streaming materialized view that ingests only event data. Load each nightly reference snapshot into a staging table by using COPY and validate it. In one transaction, DELETE all rows from the live reference table and INSERT the staged rows, then commit. Expose an ordinary SQL view that joins the streaming materialized view with the live reference table.

Native Redshift streaming ingestion reads MSK events into an automatically refreshed materialized view without S3 staging or a custom consumer. An ordinary SQL view performs enrichment using the reference data visible to each query. Loading and validating a staging table first, then replacing the live table's contents with transactional DELETE and INSERT statements, makes the reference replacement atomic. Queries see a consistent committed reference dataset.

DEA-C01

A payment platform publishes events to an Amazon Managed Streaming for Apache Kafka (Amazon MSK) topic by using Kafka transactions. Each transaction contains several related payment events. Producers commit successful transactions and abort transactions when validation fails.

A data ingestion application uses a native Kafka consumer with isolation.level=read_uncommitted. The application sometimes receives events from aborted transactions. It must ingest events from committed transactions continuously without exposing events from aborted or still-open transactions to its processing logic. The team wants to avoid implementing transaction filtering in application code.

Which change meets these requirements?

Show answerCORRECT ANSWER

Set isolation.level=read_committed on the consumer, and retain the application's existing offset-management strategy.

A read_committed consumer excludes aborted transactional records and reads transactional records only after their transactions commit. Reading is bounded by the last stable offset, preventing still-open transactions from being exposed to the application's processing logic.

SAA-C03

A company uses an Amazon MSK Provisioned cluster to ingest application events into a Kafka topic with 8 partitions. Producers distribute events evenly across all available partitions. A transformation application runs 24 identical consumer instances in one consumer group, but only 8 instances actively process events.

During sustained peaks, producers publish 9,000 events per second. Each consumer instance processes records sequentially and can transform at most 500 events per second. Testing confirms that broker CPU, storage throughput, and network bandwidth have sufficient spare capacity. Events do not require ordering, and the transformation code cannot be changed.

Which change will allow the application to keep up with peak traffic without having multiple consumer groups independently transform the same events?

Show answerCORRECT ANSWER

Increase the topic's partition count to 24 and retain all 24 consumer instances in the existing consumer group.

Within a consumer group, each partition is assigned to only one consumer at a time. Increasing to 24 evenly loaded partitions allows all 24 consumers to work concurrently, providing a total processing capacity of 12,000 events per second, which exceeds the 9,000-event-per-second peak.

Practice Amazon Managed Streaming for Apache Kafka in a full exam

Questions on Amazon Managed Streaming for Apache Kafka in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

Learn Amazon Managed Streaming for Apache Kafka in lessons

7 short lessons on Amazon Managed Streaming for Apache Kafka in 4 certification courses.

and 1 more lesson

Open the lessons

AWS architecture icons are the property of Amazon Web Services. BlowStack is not affiliated with or endorsed by AWS.