Amazon Managed Service for Apache Flink
Cloud service · AWS · Updated

Amazon Managed Service for Apache Flink

In short

Amazon Managed Service for Apache Flink is a fully managed service that allows you to build and run applications that use Apache Flink - an open-source framework and engine for processing big data streams - without the need for infrastructure management.

Cloud Services Hub →

9 key features of Amazon Managed Service for Apache Flink

Amazon Managed Service for Apache Flink offers fully managed Apache Flink applications with features like automatic scaling, built-in monitoring and logging, AWS services integration, robust security, and cost-effective pricing.

01
Fully Managed Service

Amazon Managed Service for Apache Flink is a fully managed service that automatically manages the deployment, management, and scaling of Apache Flink applications, eliminating the need for infrastructure management.

02
Automatic Scaling

This service provides automatic scaling for Apache Flink applications, adjusting the resources as per the workload demands without manual intervention, ensuring efficient resource utilization and cost savings.

03
Built-in Monitoring and Logging

Amazon Managed Service for Apache Flink integrates with Amazon CloudWatch for metrics and logging, allowing users to easily monitor their Apache Flink applications and troubleshoot any issues quickly.

04
Seamless Integration with AWS Services

It offers seamless integration with other AWS services like Amazon S3, Amazon Kinesis, Amazon DynamoDB, and Amazon EMR, enabling complex and flexible data processing pipelines within the AWS ecosystem.

05
Secure

Provides robust security features that include encryption in transit and at rest, IAM for authentication and authorization, and VPC support to ensure that your data and applications are securely managed.

06
Cost-Effective

Charges are based on allocated resources rather than actual resource usage, at an hourly rate based on the maximum number of KPUs used to run the stream-processing application.

07
Application Languages

Applications can use Java, Scala, Python, or SQL to process and analyze streaming data.

08
In-Place Flink Version Upgrades

In-place version upgrades retain the application's ARN and its associated snapshots, logs, metrics, tags, and Flink configurations across Flink versions. This feature can be used with applications in RUNNING or READY state.

09
Runtime Properties

Runtime properties pass configuration parameters to an application without requiring its code to be modified and republished.

Where Amazon Managed Service for Apache Flink is used

Amazon Managed Service for Apache Flink is utilized for real-time data processing, building event-driven applications, implementing complex event processing (CEP), and conducting machine learning model inference directly on data streams, enabling quick insights and adaptive application responses.

  1. 01
    Real-time Data Processing

    Amazon Managed Service for Apache Flink enables businesses to analyze and process data in real-time. By leveraging this service, organizations can process streaming data from sources like IoT devices, social media feeds, and website activity logs. This capability allows for the immediate identification of insights, trends, and patterns, enabling quick decision-making and instantaneous response to market dynamics.

  2. 02
    Event-driven Applications

    Utilizing Apache Flink's powerful event processing capabilities, businesses can build responsive, event-driven applications. Amazon Managed Service for Apache Flink facilitates the development of applications that can react to events in real-time, such as instant notifications, dynamic pricing models, and automated fraud detection systems, thus improving user experience and operational efficiency.

  3. 03
    Complex Event Processing (CEP)

    With Amazon Managed Service for Apache Flink, companies can implement complex event processing (CEP) to identify patterns and relationships across multiple streams of event data. This is particularly useful in sectors like finance for real-time fraud detection, telecommunications for network monitoring, and e-commerce for recommendation systems, offering the ability to act on complex events in real-time.

  4. 04
    Machine Learning Model Inference

    This service facilitates the deployment and execution of machine learning models in real-time data streams. Organizations can use Apache Flink to preprocess data, deploy models, and infer predictions directly on the stream of data, enabling adaptive and intelligent applications that evolve based on incoming data, from predictive maintenance in manufacturing to real-time personalization in retail.

How Amazon Managed Service for Apache Flink is priced

Amazon Managed Service for Apache Flink pricing includes an On-Demand model based on vCPUs and memory usage, and Savings Plans for reduced rates with a commitment.

On-Demand Capacity

Charged based on the virtual CPUs (vCPUs) and the amount of memory (measured in Gibibytes or GiB) allocated to your Flink application. This model offers flexibility without any upfront commitment.

Running Application Storage

Running application storage is billed separately from compute, based on the storage provisioned for the application.

Durable Application Backups

Durable application backups, including application snapshots, are billed according to the amount of backup data stored.

Services Amazon Managed Service for Apache Flink works with

Amazon AuroraAmazon Aurora

Amazon Managed Service for Apache Flink applications can use JDBC connectors to write processed streaming data to Aurora MySQL-Compatible or Aurora PostgreSQL-Compatible databases. JDBC lookups can also enrich incoming events with data held in Aurora.

Amazon Aurora ServerlessAmazon Aurora Serverless

Amazon Managed Service for Apache Flink applications can connect to Aurora Serverless databases through compatible JDBC drivers and connectors. Applications can write streaming aggregates to database tables or query reference data to enrich events.

Amazon CloudWatchAmazon CloudWatch

Amazon Managed Service for Apache Flink publishes application metrics to CloudWatch and can send application logs to CloudWatch Logs. You can monitor checkpoint failures, processing latency, and resource utilization, and configure alarms for operational problems.

Amazon Data FirehoseAmazon Data Firehose

Amazon Managed Service for Apache Flink applications can use the Firehose sink connector to send processed records to Firehose delivery streams. Firehose then buffers and delivers those results to its configured destination.

Amazon DocumentDB (with MongoDB compatibility)Amazon DocumentDB (with MongoDB compatibility)

Amazon Managed Service for Apache Flink applications can use a MongoDB sink connector with DocumentDB-compatible connection settings to write processed records into DocumentDB collections. Connections must use settings supported by DocumentDB, including disabling retryable writes.

Amazon DynamoDBAmazon DynamoDB

Amazon Managed Service for Apache Flink applications can use the DynamoDB sink connector to write streaming results into DynamoDB tables. For example, an application can maintain a table containing the latest computed status for each device.

Amazon ElastiCacheAmazon ElastiCache

Amazon Managed Service for Apache Flink applications can use Redis connectors with compatible ElastiCache caches to publish processed streaming results. This can make frequently updated aggregates available to applications through low-latency cache reads.

Amazon EventBridgeAmazon EventBridge

Amazon Managed Service for Apache Flink API activity recorded by CloudTrail can be matched by EventBridge rules. Rules can route application-management events, such as application creation or deletion, to notification or automation targets.

Amazon Keyspaces (for Apache Cassandra)Amazon Keyspaces (for Apache Cassandra)

Amazon Managed Service for Apache Flink applications can use a Cassandra connector configured for Amazon Keyspaces to write processed records into Keyspaces tables. The connection uses the Keyspaces endpoint, TLS, and supported authentication settings.

Amazon Kinesis Data StreamsAmazon Kinesis Data Streams

Amazon Managed Service for Apache Flink provides connectors for reading from and writing to Kinesis data streams. Applications can consume events, perform stateful transformations or windowed aggregations, and publish results to another stream.

What is Amazon Managed Service for Apache Flink

3 min read

Amazon Managed Service for Apache Flink is a fully managed service that simplifies the process of building and running applications that process large streams of data in real time. Apache Flink is an open-source framework and distributed processing engine designed for high-performance, accurate, and stateful computations over unbounded and bounded data streams. With the service provided by Amazon Web Services (AWS), users can leverage the powerful features of Apache Flink without the operational overhead of managing the infrastructure, configuring the software, or tuning the performance. 

This service is especially beneficial for developers and data engineers who work with applications that require complex time-windowed computations, event-driven processing, or need to perform real-time analytics on streaming data. 

Typical use cases include real-time financial fraud detection, log and event stream processing, and real-time analytics on customer data. Amazon Managed Service for Apache Flink is designed with scalability and reliability at its core. It manages the provisioning of compute resources, automatically scales to meet the demands of the application, and ensures high availability with fault-tolerance mechanisms, such as checkpointing, that enable consistent state management and recovery in the event of a failure. 

Application resources are allocated in Kinesis Processing Units (KPUs). Each KPU provides one CPU core, 4 GB of memory split into 1 GB of native memory and 3 GB of heap memory, and 50 GB of disk space. The number of KPUs available to an application equals its Parallelism setting divided by its ParallelismPerKPU setting.

The default KPU quota is 64. A quota increase can be requested for the applicable application prefix.

Users of this service can focus on their application logic and data processing algorithms without worrying about the underlying infrastructure. They can easily deploy their Flink applications and integrate with other AWS services for storage, database, and analytics capabilities, providing a seamless data processing pipeline from ingestion to analysis. 

For applications using Flink 2.2, Java 17 is the default and recommended runtime, and Java 11 support has been removed. Python 3.12 is supported, while Python 3.8 support has been removed.

Java applications must be packaged with their external dependencies, including required Flink connectors, in a fat JAR for deployment to the service.

Amazon Managed Service for Apache Flink Studio provides an interactive notebook environment powered by Apache Zeppelin for developing and running stream-processing code using SQL and Python.

Moreover, the service offers robust monitoring and logging features through integration with Amazon CloudWatch, allowing developers to keep a close eye on their applications' performance and troubleshoot issues swiftly. 

When upgrading to Flink 2.2, update monitoring and alarms to use numRestarts instead of the removed fullRestarts metric. The uptime and downtime metrics are deprecated in favor of state-specific metrics, and bytesRequestedPerFetch has been removed for Kinesis Data Streams connector 6.0.0.

Security is also a key focus, with Amazon Managed Service for Apache Flink offering encryption in transit and at rest, identity and access management, and network isolation through Amazon VPC. 

In the fast-evolving landscape of data processing, Amazon Managed Service for Apache Flink stands out by offering a managed environment that combines the power of Apache Flink with the scalability, reliability, and security of the AWS cloud. It frees up developers to innovate and experiment with their streaming applications while significantly reducing the operational burden and costs associated with infrastructure management. This service highlights Amazon's commitment to providing robust tools and services that empower businesses to harness the full potential of real-time analytics and event-driven architecture.

Concepts related to Amazon Managed Service for Apache Flink

AWS architecture icons are the property of Amazon Web Services. BlowStack is not affiliated with or endorsed by AWS.