Cloud concept · Updated

Event Streaming - Event streaming is the continuous publication, retention, and consumption of event records so independent applications can react to changes and replay event history.

In short

Producers append events to a stream, while consumers read and process those events independently, often within seconds or less. This decouples event generation from downstream processing and supports incident detection, operational analytics, and automated responses without requiring producers to know every consumer.

Cloud Concepts Hub →

6 key aspects of Event Streaming

Event streaming combines continuous ingestion, retained history, partitioned processing, and independent consumers, with explicit responsibility for duplicates, lag, and safe side effects.

01
Continuous, decoupled event flow

Producers publish facts without calling every downstream application. Consumers process new records independently, reducing direct dependencies between event sources and response systems.

02
Retention and replay

Records remain available for a configured retention period rather than disappearing after one consumer reads them. Consumers can resume from checkpoints or replay history, provided the required records have not expired.

03
Partitioned scalability and ordering

Partition keys distribute traffic and group related events. Design around partition-level ordering and throughput limits rather than assuming global order or unlimited parallelism.

04
Independent consumer progress

Different applications maintain their own reading positions and can process the same records at different speeds. Within a consumer group, workers typically divide partitions rather than each receiving every record.

05
Duplicate-safe processing

Retries and checkpoint recovery can cause repeated processing. Use idempotency and carefully scoped transactional guarantees, especially when events trigger external side effects.

06
Lag-aware operations and safe response

Monitor whether consumers can keep up before retained records expire. Bound retries, isolate malformed records, and prevent replayed events from unintentionally repeating incident-remediation actions.

Cloud services for Event Streaming

Use Kinesis Data Streams or Amazon MSK as the retained event backbone; they are alternatives rather than mandatory components of the same pipeline. Lambda can process event batches, while Managed Service for Apache Flink supports stateful correlation and windowed analysis. Processing results can flow through EventBridge for rule-based response routing or through SNS for notification fan-out. Keep raw event retention separate from notification delivery, and carry event identifiers through every stage. Apply idempotency, bounded retries, lag monitoring, and remediation safeguards across the complete path rather than assuming the streaming service provides them automatically.

Amazon Kinesis Data StreamsAmazon Kinesis Data Streams

Provides managed, retained data streams with shards, partition keys, and independently positioned consumers. It supports operational event pipelines that need replay and ordering within shards, subject to producer and consumer behavior.

Amazon Managed Streaming for Apache KafkaAmazon Managed Streaming for Apache Kafka

Provides managed Apache Kafka infrastructure for partitioned topics, configurable retention, and independent consumer groups. It suits architectures that require Kafka protocols and ecosystem integrations while retaining responsibility for topic design and consumer correctness.

Amazon Managed Service for Apache FlinkAmazon Managed Service for Apache Flink

Runs stateful stream-processing applications that perform operations such as windowed aggregation, event-time analysis, and pattern detection. Checkpoints support recovery, but end-to-end delivery guarantees also depend on source and sink capabilities.

AWS LambdaAWS Lambda

Processes batches from supported streaming sources through event source mappings, including Kinesis Data Streams and Amazon MSK. Configure retries and failure handling for the source, and make functions idempotent because records can be processed more than once.

Amazon EventBridgeAmazon EventBridge

Routes detected operational events to response targets using event-bus rules, while EventBridge Pipes can connect supported streaming sources to targets. An event bus is not equivalent to a partitioned stream with consumer offsets, even though EventBridge also offers archive and replay capabilities.

Event Streaming exam questions

Real practice exam questions on Event Streaming, with the correct answer and its explanation. Answer them with every option in the exam simulator.

DVA-C02

A logistics application continuously publishes package-location events to an Amazon Kinesis Data Streams stream with a retention period of 48 hours. An AWS Lambda function consumes the stream through an event source mapping and updates a live tracking system.

A developer must add a second Lambda function that transforms location events for a new analytics application. The second function must process all events still retained in the stream and then continue processing new events in near real time. Its processing progress and retries must be independent of the tracking function. The existing tracking function must continue operating without changes or interruption.

Which solution meets these requirements?

Show answerCORRECT ANSWER

Create a separate event source mapping from the same stream to the analytics function, with StartingPosition set to TRIM_HORIZON.

TRIM_HORIZON starts at the oldest available records, allowing the analytics function to process the retained history and then continue with new events. Each event source mapping maintains its own processing position, so the analytics function's progress and retries do not change the tracking function's checkpoint. Reading records does not remove them from the stream.

DOP-C02

A company collects operational events from thousands of concurrent deployment jobs. Each job has one log collector that generates events in the required processing order, such as STARTED, VALIDATED, and COMPLETED. Different jobs generate events independently.

A DevOps engineer must build a managed ingestion pipeline that makes events available to a custom processor within seconds. The processor must receive each job's events in the collector's submission order without sorting or buffering events to reconstruct that order. Events from different jobs can be processed in parallel, and no global ordering is required. Collectors can wait for a successful write response before submitting the next event for the same job. The pipeline will have sufficient capacity, and duplicate events are acceptable.

Which implementation meets these requirements?

Show answerCORRECT ANSWER

Send events to Amazon Kinesis Data Streams using the deployment job ID as the partition key. For each job, issue PutRecord calls serially, complete retries before advancing, and pass the previous successful write's sequence number as SequenceNumberForOrdering. Process each shard in record order.

A stable job ID routes a job's events to the same shard. Serial PutRecord calls with SequenceNumberForOrdering preserve the submission order for that partition key. Completing retries before advancing prevents later events from overtaking earlier events. Ordered shard processing preserves that order downstream while allowing parallel processing across shards.

SAA-C03

An online gaming company publishes player activity events to an Amazon Kinesis Data Streams stream that retains records for 7 days. A Kinesis Client Library (KCL) application continuously updates live leaderboards.

The company has developed a replacement scoring algorithm. A second KCL application must use the new algorithm to process events from the previous 3 days into a separate leaderboard table and then continue processing newly arriving events. The existing application must continue updating the production leaderboards without interruption or replaying historical events. Stream read capacity and application processing capacity are sufficient for both workloads.

Which configuration meets these requirements?

Show answerCORRECT ANSWER

Give the second application a distinct KCL application name and configure its initial position as AT_TIMESTAMP with a timestamp from 3 days ago. Maintain separate checkpoints for the two applications.

The distinct application name establishes independent leases and checkpoints. Starting at the required timestamp lets the second application read retained events, build the replacement leaderboard, and continue consuming new events without changing the existing application's progress.

Practice Event Streaming in a full exam

Questions on Event Streaming in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Event Streaming

3 min read

Event streaming treats changes as a continuous flow of records rather than isolated requests or periodic batch files. An event describes something that happened, such as a deployment completing, a payment failing, or a server reporting an unhealthy condition. Producers publish these records to a streaming platform, which typically retains them for a configured period. Consumers track their reading positions and process the records independently, allowing multiple applications to use the same event history.

In cloud architectures, streams commonly connect application services, telemetry pipelines, change-data-capture systems, and incident-response automation. For example, application instances can publish deployment and error events using a service identifier as the partition key. One consumer calculates error rates, another archives events for investigation, and another correlates failures with recent deployments. A detected incident can then trigger a notification or a controlled remediation workflow. These consumers can evolve without changing the producing application.

Partitioning provides scalability but establishes important ordering boundaries. A stream is usually divided into partitions or shards, with related records routed using a key. Ordering guarantees normally apply within a partition, not across the entire stream, and producer concurrency or retries can affect the observed business sequence. Include event identifiers, occurrence timestamps, schema versions, and entity sequence numbers where needed. A poor partition key can concentrate traffic on one partition even when other partitions have spare capacity.

Retention and consumer checkpoints distinguish typical streaming systems from ordinary work queues. Reading a record does not normally remove it from the stream, so independent consumers can replay retained records to recover from a failure or rebuild derived state. In a work queue, successful processing typically leads to message deletion or acknowledgment, and competing workers usually divide the work. Publish-subscribe fan-out distributes events to multiple subscribers, but fan-out alone does not guarantee a retained, independently replayable event log. Choose the pattern based on processing requirements rather than treating streaming, queuing, and event routing as interchangeable.

Assume duplicates can occur unless the complete processing path provides stronger guarantees. A consumer might update a database and fail before saving its checkpoint, causing the same event to be processed again. Use idempotent writes, deduplication keys, or transactional coordination where supported. Exactly-once processing inside a streaming engine does not automatically make external actions, such as sending a notification or restarting a server, exactly once. Replay procedures must also prevent historical events from repeating dangerous remediation actions.

Operational reliability depends on controlling lag and failures. Monitor ingestion failures, throttling, consumer lag, processing errors, and the age of unprocessed records. Retention must exceed the expected outage and recovery period; otherwise, a slow consumer can permanently lose access to expired events. Handle poison records with bounded retries and a quarantine or failure destination, recognizing that skipping a record can affect downstream correctness. Scale consumers within partition and service limits, and account for downstream database or API capacity.

For incident-response scenarios, select streaming when continuous ingestion, ordered processing within a key, independent consumers, or replay are explicit requirements. Use an event bus for rule-based routing and a queue for buffered work distribution when those requirements are more important. Protect streams with least-privilege access and encryption, manage schema compatibility, and place authorization and safety checks around automated corrective actions.

Learn Event Streaming in lessons

8 short lessons on Event Streaming in 4 certification courses.

and 2 more lessons

Open the lessons