Amazon AppFlowAmazon MWAA workflows can use Apache Airflow's AppFlow operators to start flows and monitor their completion. This lets a workflow wait for SaaS data ingestion before starting downstream processing.
Amazon AthenaAmazon MWAA can submit Athena queries and wait for their results using dedicated Airflow operators and sensors. Workflows can use these queries to transform or validate data stored in Amazon S3.
Amazon AuroraAmazon MWAA uses an AWS-managed Aurora PostgreSQL database for Apache Airflow metadata. AWS provisions and operates this database as part of the environment.
Amazon Bedrock
Amazon MWAA workflows can use dedicated Bedrock operators to invoke models and orchestrate supported model customization and batch inference jobs. Sensors can wait for asynchronous jobs before downstream tasks consume their results.
Amazon CloudWatchAmazon MWAA publishes environment metrics and configurable Airflow logs to CloudWatch. You can monitor scheduler and worker health, inspect task failures, and create alarms for operational problems.
Amazon ComprehendAmazon MWAA can orchestrate Comprehend analysis jobs through dedicated Airflow operators. A workflow can submit documents for entity recognition and wait for the analysis before processing the output.
Amazon DataZone
Amazon MWAA can publish workflow lineage to Amazon DataZone using an Airflow OpenLineage integration. This connects Airflow job executions with their input and output datasets in the data catalog.
Amazon DynamoDBAmazon MWAA workflows can use DynamoDB-specific Airflow hooks, sensors, and transfer operators. For example, a workflow can wait for table availability or export table records to Amazon S3 for downstream processing.
Amazon EC2Amazon MWAA can use dedicated EC2 operators and sensors to create, start, stop, terminate, and monitor instances. This allows workflows to coordinate instance-based processing with other pipeline tasks.
Amazon Elastic Container Registry (Amazon ECR)Amazon MWAA workflows can use Airflow's ECR hook to obtain registry authorization credentials. These credentials can authenticate container workloads launched by a workflow against private ECR repositories.
Amazon Elastic Container Service (Amazon ECS)Amazon MWAA can launch ECS tasks and monitor their completion using dedicated Airflow operators. A workflow can run a containerized transformation and continue only after the task succeeds.
Amazon Elastic Kubernetes Service (Amazon EKS)Amazon MWAA can orchestrate workloads on EKS using Airflow's Kubernetes and EKS operators. Workflows can launch pods for processing tasks and manage EKS cluster and node group lifecycles.
Amazon ElastiCacheAmazon MWAA workflows can use Airflow's ElastiCache hook to work with cache snapshots, including copying snapshot data to Amazon S3. This can incorporate cache exports into scheduled data workflows.
Amazon EMRAmazon MWAA can create EMR clusters, submit processing steps, and monitor their execution through dedicated operators and sensors. It also supports orchestration of EMR Serverless jobs and EMR on EKS job runs.
Amazon EventBridgeAmazon MWAA workflows can publish custom events to EventBridge using Airflow's EventBridge operator. A successful processing task can emit an event that starts downstream event-driven processing.
Amazon Kinesis Data StreamsAmazon MWAA workflows can use Airflow's Kinesis hook to write records to Kinesis Data Streams. This lets a scheduled workflow publish processed records into a streaming pipeline.
Amazon Managed Streaming for Apache KafkaAmazon MWAA can use Airflow's MSK integration to retrieve cluster connection information and orchestrate supported cluster operations. Airflow's Kafka operators and sensors can then produce or consume records through the cluster's Kafka endpoints.
Amazon OpenSearch ServiceAmazon MWAA can run Airflow's OpenSearch operators and hooks to execute searches and indexing operations against OpenSearch Service domains. A workflow can index newly processed documents or query an index to validate a data load.
Amazon QuickSightAmazon MWAA can start QuickSight dataset ingestions and monitor their completion using dedicated Airflow operators and sensors. This lets a workflow refresh a SPICE dataset after its underlying data has been updated.
Amazon RDSAmazon MWAA workflows can orchestrate RDS snapshot creation, copying, deletion, and snapshot exports using dedicated Airflow operators. Sensors can wait for snapshots and export tasks before subsequent processing begins.
Amazon RedshiftAmazon MWAA can execute Redshift SQL, manage clusters, and move data between Redshift and Amazon S3 through dedicated Airflow integrations. A workflow can load staged data, run transformations, and validate the resulting warehouse tables.
Amazon S3Amazon MWAA loads workflow definitions, plugins, Python requirements, and optional startup scripts from an S3 bucket. Workflows can also use S3 operators and sensors to detect arriving objects, transfer data, and manage stored files.
Amazon S3 GlacierAmazon MWAA workflows can use Airflow's Glacier integration to initiate archive retrieval jobs and obtain their output. This allows retrieval from Glacier vaults to be coordinated with subsequent data processing.
Amazon SageMakerAmazon MWAA can orchestrate SageMaker training, processing, tuning, batch transform, model deployment, and pipeline executions through dedicated Airflow operators. Sensors can wait for machine learning jobs before evaluation or deployment tasks proceed.
Amazon Simple Email Service (Amazon SES)Amazon MWAA can use Airflow's SES email backend or hook to send workflow notifications. For example, a DAG can email an operational summary or notify maintainers when a task fails.
Amazon Simple Notification Service (Amazon SNS)Amazon MWAA workflows can publish messages to SNS topics using a dedicated Airflow operator. This can notify subscribers when a pipeline finishes or encounters a failure.
Amazon Simple Queue Service (Amazon SQS)Amazon MWAA uses SQS as the queue for distributing work to its Airflow workers. Workflows can also use SQS operators and sensors to send messages or wait for incoming messages.
Amazon TextractAmazon MWAA can start Textract document-processing jobs and coordinate their results using dedicated Airflow operators. A workflow can extract text or structured information from documents before loading it into an analytics system.
Amazon VPCAmazon MWAA connects its schedulers and workers to private subnets in your VPC. Subnet routing and security groups control access to databases, service endpoints, and other resources used by workflows.
AWS BatchAmazon MWAA can submit AWS Batch jobs and monitor their execution through dedicated Airflow operators and sensors. This allows compute-intensive batch processing to run as a managed step in a larger workflow.
AWS Clean Rooms
Amazon MWAA can use dedicated Airflow operators to start Clean Rooms protected queries and wait for their completion. This lets collaborative analysis become a scheduled step in a data pipeline.
AWS CloudFormationAmazon MWAA environments can be provisioned and updated through CloudFormation using the AWS::MWAA::Environment resource type. MWAA workflows can also use Airflow's CloudFormation operators to create or delete stacks and wait for stack operations.
AWS CloudTrailAmazon MWAA management API calls are recorded by CloudTrail. These records support auditing actions such as creating, updating, and deleting environments.
AWS CodeBuildAmazon MWAA workflows can start CodeBuild builds and monitor their status through Airflow's CodeBuild integration. A pipeline can build or test an artifact before using it in subsequent processing.
AWS CodePipelineAmazon MWAA can use Airflow's CodePipeline hook and sensors to coordinate with pipeline executions and wait for pipeline or stage completion. This allows a workflow to depend on a successful software delivery stage.
AWS ConfigAWS Config can record configuration changes to Amazon MWAA environments. The recorded resource history can be used to assess environment configuration and investigate changes.
AWS Database Migration Service (AWS DMS)Amazon MWAA can create, start, stop, and monitor DMS replication tasks using dedicated Airflow operators and sensors. A workflow can coordinate a database migration or replication stage with downstream validation.
AWS DataSyncAmazon MWAA can launch DataSync task executions and wait for transfers to finish using a dedicated Airflow operator. Downstream processing can begin only after the required files have reached their destination.
AWS FargateAmazon MWAA provisions and operates Fargate compute for its Airflow schedulers, workers, and webservers. Workflows can also launch their own Fargate tasks through Airflow's ECS integration.
AWS GlueAmazon MWAA can start Glue jobs and crawlers, monitor their completion, and coordinate Glue Data Catalog partitions. Workflows can also orchestrate Glue Data Quality runs before allowing data to move to later stages.
AWS Glue DataBrew
Amazon MWAA workflows can start DataBrew jobs using a dedicated Airflow operator and wait for their completion. This lets recipe-based data preparation run as part of a scheduled pipeline.
AWS Identity and Access Management (IAM)Amazon MWAA uses an IAM execution role to access workflow files, logs, queues, and services called by Airflow tasks. IAM policies also control who can manage environments and obtain access to the Airflow interface.
AWS Key Management Service (AWS KMS)Amazon MWAA supports customer-managed KMS keys for encrypting environment data, including its metadata database and associated managed resources. The environment's execution role and key policy must permit the required cryptographic operations.
AWS LambdaAmazon MWAA can invoke Lambda functions and wait for function readiness using dedicated Airflow operators and sensors. A workflow can delegate a short processing or validation task to a function.
AWS PrivateLinkAmazon MWAA supports interface VPC endpoints powered by PrivateLink for private access to supported MWAA API and environment endpoints. This allows access without routing traffic over the public internet.
AWS Secrets ManagerAmazon MWAA can use Secrets Manager as an Apache Airflow secrets backend for connections and variables. Workflow tasks can retrieve database credentials without embedding them in DAG files.
AWS Security Token Service (AWS STS)Amazon MWAA workflows can use Airflow AWS connections configured to assume IAM roles through STS. This supplies temporary credentials for tasks that access resources in another account or require a separate role.
AWS Step FunctionsAmazon MWAA can start Step Functions state machine executions and monitor their completion using dedicated Airflow operators and sensors. This lets an Airflow DAG coordinate a state machine as one stage of a larger workflow.
AWS Systems ManagerAmazon MWAA can use Systems Manager Parameter Store as an Airflow secrets backend. Connections and variables can be retrieved from parameters instead of being stored directly in Airflow's metadata database.