Amazon Managed Workflows for Apache Airflow (Amazon MWAA)
Cloud service · AWS · Updated

Amazon Managed Workflows for Apache Airflow (Amazon MWAA)

In short

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a cloud service that orchestrates complex data workflows, allowing users to manage, execute, and scale Apache Airflow environments on the AWS cloud without managing the underlying infrastructure.

Cloud Services Hub →

6 key features of Amazon Managed Workflows for Apache Airflow (Amazon MWAA)

Amazon MWAA offers scalable serverless workflow management, easy integration with AWS services, built-in security features, swift deployment options, and comprehensive monitoring and logging capabilities.

01
Serverless Workflow Management

Amazon MWAA automatically scales the workflow execution capacity to match your workload without requiring manual server management, ensuring efficient resource utilization.

02
Integrated with AWS Services

Seamlessly integrates with various AWS services such as AWS Lambda, Amazon S3, Amazon ECS, and Amazon RDS, enabling a more efficient and smooth operation of data pipelines.

03
Built-in Security

Provides built-in features such as encryption in transit and at rest, enabling you to secure your data workflows without additional configuration.

04
Easy Deployment

Offers simple and quick deployment options for your Apache Airflow environment, allowing you to get your data pipelines up and running rapidly.

05
Monitoring and Logging

Amazon MWAA integrates with Amazon CloudWatch and Amazon CloudTrail for extensive monitoring and logging, giving you full visibility into your Apache Airflow workflows.

06
Public or Private Airflow UI Access

Provisioned environments support private or public network access to the Apache Airflow web UI, with user access controlled by IAM policies in either configuration.

Where Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is used

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is principally used for orchestrating complex data pipelines, automating machine learning model training, executing ETL jobs, and managing batch processing tasks, streamlining these operations with efficient task management and scheduling.

  1. 01
    Data Pipeline Orchestration

    Amazon MWAA facilitates the scheduling and orchestration of complex data pipelines. Users can easily manage the execution of multiple data processing tasks in a specific sequence or in parallel, ensuring that dependencies are managed and that the workflow is executed efficiently.

  2. 02
    Machine Learning Model Training

    Amazon MWAA can be used to automate the training of machine learning models by orchestrating various steps involved in the process, such as data preprocessing, model training, model evaluation, and model deployment, ensuring that each step is executed once the prerequisites are met.

  3. 03
    ETL Jobs

    Amazon MWAA simplifies the execution of Extract, Transform, Load (ETL) jobs by managing the dependencies and execution order of tasks. It allows for efficient processing and transformation of large volumes of data from various sources before loading it into a data warehouse or data lake.

  4. 04
    Batch Processing

    This use case involves leveraging Amazon MWAA to manage and execute batch processing jobs, which can include tasks such as data analysis, computation, and the execution of scripts. Amazon MWAA ensures these jobs are run on a schedule or in response to specific triggers.

How Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is priced

Amazon MWAA pricing includes a fixed monthly charge per environment plus variable charges based on the vCPU and memory resources used by your workflows, with overall costs affected by the region.

Environment Pricing

Provisioned environments are charged hourly, billed at one-second resolution, with rates that vary by environment size.

Serverless Task Pricing

MWAA Serverless charges for AWS Managed Tasks and Customer Managed Tasks by task duration, at hourly rates billed at one-second resolution with a one-minute minimum.

Additional Worker Instance Pricing

Additional worker instances used for auto-scaling are charged hourly at one-second resolution, with rates that vary by environment size.

Additional Scheduler Instance Pricing

Scheduler instances beyond those included with the environment incur additional hourly instance charges.

Additional Web Server Instance Pricing

Additional web server instances are charged for the duration the environment is available, at hourly rates billed at one-second resolution that vary by environment size.

Metadata Database Storage

Metadata database storage is charged per GB-month based on storage consumed, without advance provisioning.

Services Amazon Managed Workflows for Apache Airflow (Amazon MWAA) works with

Amazon AppFlowAmazon AppFlow

Amazon MWAA workflows can use Apache Airflow's AppFlow operators to start flows and monitor their completion. This lets a workflow wait for SaaS data ingestion before starting downstream processing.

Amazon AthenaAmazon Athena

Amazon MWAA can submit Athena queries and wait for their results using dedicated Airflow operators and sensors. Workflows can use these queries to transform or validate data stored in Amazon S3.

Amazon AuroraAmazon Aurora

Amazon MWAA uses an AWS-managed Aurora PostgreSQL database for Apache Airflow metadata. AWS provisions and operates this database as part of the environment.

Amazon Bedrock

Amazon MWAA workflows can use dedicated Bedrock operators to invoke models and orchestrate supported model customization and batch inference jobs. Sensors can wait for asynchronous jobs before downstream tasks consume their results.

Amazon CloudWatchAmazon CloudWatch

Amazon MWAA publishes environment metrics and configurable Airflow logs to CloudWatch. You can monitor scheduler and worker health, inspect task failures, and create alarms for operational problems.

Amazon ComprehendAmazon Comprehend

Amazon MWAA can orchestrate Comprehend analysis jobs through dedicated Airflow operators. A workflow can submit documents for entity recognition and wait for the analysis before processing the output.

Amazon DataZone

Amazon MWAA can publish workflow lineage to Amazon DataZone using an Airflow OpenLineage integration. This connects Airflow job executions with their input and output datasets in the data catalog.

Amazon DynamoDBAmazon DynamoDB

Amazon MWAA workflows can use DynamoDB-specific Airflow hooks, sensors, and transfer operators. For example, a workflow can wait for table availability or export table records to Amazon S3 for downstream processing.

Amazon EC2Amazon EC2

Amazon MWAA can use dedicated EC2 operators and sensors to create, start, stop, terminate, and monitor instances. This allows workflows to coordinate instance-based processing with other pipeline tasks.

Amazon Elastic Container Registry (Amazon ECR)Amazon Elastic Container Registry (Amazon ECR)

Amazon MWAA workflows can use Airflow's ECR hook to obtain registry authorization credentials. These credentials can authenticate container workloads launched by a workflow against private ECR repositories.

What is Amazon Managed Workflows for Apache Airflow (Amazon MWAA)

3 min read

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a cloud service provided by Amazon Web Services (AWS) designed to simplify running Apache Airflow on the AWS cloud. Apache Airflow is an open-source platform used for orchestrating complex computational workflows and data processing pipelines. By leveraging Amazon MWAA, users can take advantage of Apache Airflow’s powerful capabilities without the complexity of setting up, configuring, and managing the environment themselves. This managed service automates tasks such as provisioning resources, scaling according to the workload, and monitoring the health of instances. Amazon MWAA builds and patches version images, but an environment continues using its specified image version until you explicitly update the environment to a later version. 

Amazon MWAA offers serverless and provisioned deployment options. Amazon MWAA Serverless automatically executes YAML or Python-based workflows on demand or on a schedule, without requiring Apache Airflow configuration.

Amazon MWAA Serverless has a maximum task execution timeout of 60 minutes.

The default Amazon MWAA Serverless concurrent-run quotas are 100 runs per account and 20 runs per workflow in each AWS Region.

One of the key benefits of using Amazon MWAA is its deep integration with other AWS services. This allows users to easily create workflows that utilize AWS services such as Amazon S3 for storage, Amazon Redshift for data warehousing, AWS Lambda for serverless compute, and many others without extensive configuration. The integration is designed to be seamless, enabling users to directly reference AWS resources within their Airflow DAGs (Directed Acyclic Graphs), which are the collection of tasks you want to run, organized to reflect their relationships and dependencies. 

Amazon MWAA follows best practices for high availability and security. For provisioned environments, Amazon MWAA creates an AWS-owned VPC hosting the webserver and metadata database; the scheduler and workers connect to private subnets in the VPC that you own and configure. The service is built to scale, so as the demand for processing increases, Amazon MWAA can automatically adjust resources to meet the workload demands without the need for manual intervention. Security in Amazon MWAA is a top priority, and the service offers features such as encryption in transit and at rest, integration with AWS Identity and Access Management (IAM) for fine-grained access control, and logging through Amazon CloudWatch. These capabilities ensure that user data is protected and that administrators can audit and monitor access and usage. 

Provisioned environments can be configured with up to 25 workers through the console; a quota increase can allow up to 50 workers.

Provisioned worker autoscaling operates between configured minimum and maximum worker counts. It scales out when queued and running tasks exceed existing worker capacity, and scales back to the minimum when the combined number of queued and running tasks remains zero for more than two minutes.

Getting started with Amazon MWAA is straightforward. Users can set up their Airflow environment through the AWS Management Console, via the AWS Command Line Interface (CLI), or using Infrastructure as Code tools such as AWS CloudFormation. This flexibility allows users to integrate Amazon MWAA into their existing CI/CD pipelines and infrastructure management practices easily. 

In conclusion, Amazon Managed Workflows for Apache Airflow simplifies the deployment, management, and scalability of Apache Airflow environments on AWS. It enables data engineers and developers to focus on designing workflows and writing DAGs rather than managing infrastructure. With its robust integration with AWS services, high availability, security features, and flexible deployment options, Amazon MWAA is a powerful tool for automating and orchestrating complex data processing tasks in the cloud.

AWS architecture icons are the property of Amazon Web Services. BlowStack is not affiliated with or endorsed by AWS.