Amazon Data Firehose
Cloud service · AWS · Updated

Amazon Data Firehose

In short

Amazon Data Firehose is a fully managed service designed to load streaming data in real time into data lakes, data stores, and analytics services such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk.

Cloud Services Hub →

9 key features of Amazon Data Firehose

Amazon Kinesis Data Firehose offers an easy-to-use, automatically scalable, near real-time data streaming service with capabilities for data transformation and extensive support for various analytics destinations, along with simplified monitoring and automation.

01
Easy to Use

Amazon Data Firehose provides a straightforward way to reliably load streaming data into data lakes, data stores, and analytics services. It eliminates the need to write applications or manage resources, facilitating quick data transfer.

02
Automatic Scaling

It automatically scales to match the throughput of your data and requires no ongoing administration. This ensures that your data streaming does not suffer from bottlenecks or require manual intervention for scaling.

03
Near Real-time

Amazon Kinesis Data Firehose enables near real-time analytics with minimal latency, typically less than 60 seconds, allowing businesses to quickly analyze and respond to incoming streaming data.

04
Data Transformation

It offers the ability to transform data before loading it into the destination, making it possible to convert raw streaming data into formats required by downstream analytics tools without additional processing layers.

05
Supported Destinations

Amazon Data Firehose supports a wide range of destinations including Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk, making it versatile for various use cases and analytics needs.

06
Monitoring and Automation

With Amazon Kinesis Data Firehose, monitoring is simplified through integration with Amazon CloudWatch, and data streaming processes can be automated using AWS Lambda for event-driven processing tasks.

07
Iceberg, Serverless, and HTTP Destinations

Firehose also supports delivery to Apache Iceberg tables, Amazon OpenSearch Serverless, and custom HTTP endpoints, including endpoints owned by supported third-party providers.

08
Kinesis Data Streams Source

A Firehose stream can automatically read records from an existing Kinesis data stream and deliver them to its destination.

09
Amazon MSK Source

Firehose supports Amazon MSK as a source, with a default read-throughput quota of 10 MB per second per partition and a maximum record size of 10 MB when Lambda transformation is disabled. With Lambda enabled, the maximum record size is 6 MB; oversized records are delivered to an S3 error bucket.

Where Amazon Data Firehose is used

Amazon Kinesis Data Firehose is used for real-time data analytics, log and event data capture, streamlining IoT data, and enabling machine learning model inference by capturing, transforming, and loading streaming data into AWS data stores.

  1. 01
    Real-time Data Analytics

    Amazon Data Firehose allows organizations to capture, transform, and load streaming data into AWS data stores, such as Amazon S3, Amazon Redshift, and Amazon OpenSearch Service, in real-time. This enables immediate analytics and insights using tools like Amazon Athena and Amazon QuickSight for data-driven decision-making.

  2. 02
    Log and Event Data Capture

    Organizations can use Amazon Data Firehose to efficiently collect, aggregate, and process log and event data from various sources, such as application logs, website clickstreams, and IoT devices. This facilitates centralized logging, monitoring, and analysis for operational excellence and compliance auditing.

  3. 03
    Streamlining IoT Data

    With Amazon Data Firehose, businesses can manage the vast influx of data from IoT devices and sensors by ingesting, transforming, and delivering this data to the appropriate analytics tools and storage solutions on AWS. This supports real-time and batch analytics, enabling actionable insights into IoT operations and customer behaviors.

  4. 04
    Machine Learning Model Inference

    Kinesis Data Firehose can be used to preprocess and deliver streaming data to Amazon SageMaker for real-time machine learning model inference. This integration allows businesses to add AI and ML capabilities to their applications, providing features like predictive analytics, anomaly detection, and personalized content delivery.

How Amazon Data Firehose is priced

Amazon Kinesis Data Firehose pricing is based on the volume of data ingested, the use of data conversion features, and data transfer costs outside the region or to the internet.

Data Ingestion Pricing

With Amazon Data Firehose, you pay for the amount of data you transmit through the service. The price is calculated based on the volume of data ingested into the service, measured in gigabytes.

Data Conversion Pricing

If you enable built-in data conversion features, which convert JSON into Apache Parquet or Apache ORC, additional charges are applied. These charges are based on the volume of data processed by the conversion feature.

Data Transfer Pricing

Data transferred out of Amazon Data Firehose to another AWS service (within the same region) is free. However, if the data is transferred to AWS services in different regions or to the internet, standard AWS data transfer fees apply.

Dynamic Partitioning Pricing

When dynamic partitioning is enabled for Amazon S3 delivery, additional charges apply per gigabyte processed and per object delivered to S3; using jq for parsing also incurs processing-hour charges.

CloudWatch Logs Decompression Pricing

Enabling decompression of CloudWatch Logs incurs an additional charge based on the volume of decompressed data.

Services Amazon Data Firehose works with

Amazon S3Amazon S3

Amazon Data Firehose can deliver streaming data to Amazon S3 buckets, providing durable storage with high availability for big data analytics.

Amazon RedshiftAmazon Redshift

Amazon Data Firehose can load streaming data into Amazon Redshift tables by first delivering it to Amazon S3 and then issuing a Redshift COPY command, enabling near real-time analytics and querying of structured data.

Amazon API GatewayAmazon API Gateway

Amazon Data Firehose can receive API Gateway REST API access logs for delivery to analytics and storage destinations. API Gateway can also use an AWS service integration to submit request data to a Firehose stream.

Amazon AuroraAmazon Aurora

Amazon Data Firehose can capture changes from supported Aurora MySQL and PostgreSQL databases and deliver them to Apache Iceberg tables. This supports continuously updated analytical copies of operational data.

Amazon Aurora ServerlessAmazon Aurora Serverless

Amazon Data Firehose can capture database changes from supported Aurora Serverless v2 configurations and deliver them to Apache Iceberg tables. The database must have the required replication settings and permissions enabled.

Amazon CloudFrontAmazon CloudFront

Amazon Data Firehose can receive CloudFront standard access logs through CloudFront standard logging v2. Firehose buffers and delivers the logs to its configured destination.

Amazon CloudWatchAmazon CloudWatch

Amazon Data Firehose receives CloudWatch Logs subscription data and CloudWatch metric streams for delivery to storage and monitoring destinations. Firehose also publishes its own delivery metrics and can write delivery-error logs to CloudWatch.

Amazon ConnectAmazon Connect

Amazon Data Firehose can receive contact records through Amazon Connect's data-streaming configuration. Firehose delivers these records to a destination for contact-center reporting and analysis.

Amazon EC2Amazon EC2

Amazon Data Firehose can receive files and logs collected by the Amazon Kinesis Agent running on EC2 instances. The agent monitors configured files and batches new records into a Firehose stream.

Amazon ECS AnywhereAmazon ECS Anywhere

Amazon Data Firehose can receive container logs from ECS Anywhere tasks through FireLens log routing. This lets applications on external ECS instances use the same Firehose delivery pipeline as containers running in AWS.

Amazon Data Firehose exam questions

Real practice exam questions on Amazon Data Firehose, with the correct answer and its explanation. Answer them with every option in the exam simulator.

CLF-C02

A company’s applications continuously generate operational events. The company needs to deliver these events to Amazon S3 for later analysis, with a delivery delay of a few minutes being acceptable. It wants to buffer and compress the events automatically without writing a consumer application or creating an ETL job. Which solution best meets these requirements?

Show answerCORRECT ANSWER

Send the events to Amazon Data Firehose with an Amazon S3 destination and enable compression.

Amazon Data Firehose provides managed delivery of streaming data to S3. It can buffer and compress records before delivery without requiring a custom consumer application or an ETL job.

MLA-C02 (beta)

A retailer collects JSON customer interaction events to build ML training datasets. Each event contains an email address that must be removed before the event is written to a curated Amazon S3 bucket. Compliance also requires every original, unmodified event to be retained in a separate, access-restricted S3 archive bucket. Delivery delays of several minutes are acceptable. Which solution meets these requirements with the least custom processing infrastructure?

Show answerCORRECT ANSWER

Send events to Amazon Data Firehose. Use a Lambda transformation to remove email addresses before delivery to the curated bucket, and enable source-record backup of all incoming records to the separate archive bucket.

Firehose can invoke Lambda to transform records before delivering them to S3. Source-record backup preserves the original, untransformed records in a separate S3 bucket, satisfying both the curated-data and archival requirements without a continuously running processing application.

SAP-C03

A company has 60 workload accounts in an AWS Organizations organizational unit (OU). Applications write logs to Amazon CloudWatch Logs in two AWS Regions. A separate log-archive account owns a central Amazon S3 bucket. The company must archive every newly ingested log event from existing and future workload log groups, with events normally reaching S3 within a few minutes. Logs must remain searchable in their source accounts for 30 days and archived in S3 for 7 years. Archived logs older than 90 days are rarely accessed, and a retrieval time of up to 48 hours is acceptable. The solution must automatically cover new workload accounts and must not require custom log-processing code.

Which solution meets these requirements with the LEAST ongoing operational effort?

Show answerCORRECT ANSWER

In the log-archive account, create a CloudWatch Logs logical destination in each source Region, backed by a regional Amazon Data Firehose stream that delivers to the central S3 bucket. Configure destination access policies to authorize the workload accounts and service roles that allow delivery. Use an organization-integrated CloudFormation StackSet with automatic OU deployment to configure regional account-level subscription policies covering all workload log groups, the necessary sender permissions, and 30-day source retention. Configure S3 Lifecycle to transition objects to S3 Glacier Deep Archive after 90 days and expire them after 7 years.

Regional logical destinations provide the supported cross-account subscription path, and Firehose provides managed, buffered delivery to S3 without custom processing code. Account-level subscription policies cover existing and future log groups. StackSets automatic deployment extends the configuration to new accounts in the OU. Source retention is independent of the S3 archive, and Glacier Deep Archive supports the stated retrieval window while reducing long-term storage costs.

Practice Amazon Data Firehose in a full exam

Questions on Amazon Data Firehose in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Amazon Data Firehose

3 min read

Amazon Data Firehose is a fully managed service that facilitates the effortless streaming of data in real-time to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, and Splunk, allowing for near real-time analytics with existing business intelligence tools and dashboards. 

It's part of the broader Amazon Kinesis suite, which provides solutions to collect, process, and analyze real-time, streaming data, enabling developers and businesses to gain timely insights and react promptly to new information. The process of utilizing Amazon Data Firehose begins with the creation of a Firehose delivery stream. Once created, you can send your streaming data, such as application logs, website clickstreams, IoT telemetry data, and more to the Firehose service, which then automatically delivers the data to the specified destination. 

Firehose buffers incoming data before delivery using a buffer-size threshold and a buffer interval measured in seconds.

PutRecordBatch accepts up to 500 records per request, with each record limited to 1,000 KB before Base64 encoding and the entire request limited to 4 MB. These limits cannot be changed.

Bursts of incoming bytes or records lasting only a few seconds might not be fully visible in one-minute CloudWatch metrics.

One of the key features of Firehose is its ability to transform and prepare data before delivering it. This includes tasks like converting data formats to match the target storage solution, compressing the data to reduce storage costs, and even encrypting the data for enhanced security before it lands in the destination service. Data transformation is facilitated through integration with AWS Lambda, allowing customization of the data as it flows through Firehose, making it extremely adaptable to various data processing needs. Through this integration, data analytics and transformation can be deployed without managing any additional infrastructure, further simplifying the architecture required for real-time analytics. Amazon Data Firehose manages all the resources required to operate and automatically scales to match the throughput of your data, making it incredibly scalable. 

When dynamic partitioning is enabled, a Firehose stream has a default quota of 500 active partitions. Active partitions are those currently held in the delivery buffer; a partition stops being active once its data is delivered.

It can handle any amount of data, from small to colossal streams, without upfront cost or the need to provision resources manually. This level of scalability and management frees up developers and data scientists to focus on analyzing data rather than managing backend infrastructure. Billing for Amazon Kinesis Data Firehose is straightforward and is based on the amount of data ingested into the service. This pay-as-you-go model ensures that costs are directly aligned with usage, offering an efficient way to manage expenses related to streaming data analytics.

In summary, Amazon Data Firehose provides a robust, scalable, and manageable solution for streaming large volumes of data in real-time to various destinations. Its ability to transform data on the fly, coupled with automatic scaling and simple yet powerful integration points, makes it an indispensable tool for businesses looking to leverage their data for timely insights and decision-making.

Learn Amazon Data Firehose in lessons

15 short lessons on Amazon Data Firehose in 7 certification courses.

and 9 more lessons

Open the lessons

Concepts related to Amazon Data Firehose

AWS architecture icons are the property of Amazon Web Services. BlowStack is not affiliated with or endorsed by AWS.