Cloud concept · Updated

Change Data Capture - Change Data Capture is a technique for detecting inserts, updates, and deletes in a source data store and making those changes available to downstream systems.

In short

Change Data Capture (CDC) lets systems process incremental changes instead of repeatedly copying an entire dataset. It supports low-downtime migrations, continuously updated analytics, and current search or vector indexes, while requiring careful handling of ordering, retries, deletions, and recovery.

Cloud Concepts Hub →

6 key aspects of Change Data Capture

Reliable CDC combines incremental capture with coordinated initialization, replay-safe processing, explicit freshness targets, and complete downstream lifecycle handling.

01
Incremental change propagation

Process inserts, updates, and deletes instead of repeatedly scanning and transferring all source data. Savings depend on the change rate, capture overhead, and downstream processing costs.

02
Coordinated bootstrap and checkpoints

Align an initial snapshot with a durable change-stream position so writes during the load are not lost. Persist checkpoints and retain enough source history to recover from expected outages.

03
Capture-method trade-offs

Choose among transaction-log reading, triggers, and watermark-based polling according to source capabilities and operational constraints. Verify delete detection and whether intermediate changes must be preserved.

04
Replay-safe processing

Use stable keys, idempotent writes, and source-version comparisons to tolerate retries and duplicates. Do not assume global ordering or atomic application of source transactions across downstream systems.

05
Freshness matched to workload

Select continuous streaming, micro-batching, or scheduled consumption based on the allowed data staleness. Monitor end-to-end freshness rather than only whether the capture process is running.

06
Derived-data lifecycle management

Propagate deletions, permission changes, and obsolete-record cleanup to search and vector indexes. Combine incremental updates with reconciliation and planned rebuilds when transformations or embedding models change.

Cloud services for Change Data Capture

AWS DMS can capture changes from supported databases and deliver them to supported targets, including Kinesis Data Streams for custom processing. DynamoDB Streams provides a separate capture path for DynamoDB item changes. Lambda consumers can transform these events and update derived data in OpenSearch Service, invoking an embedding model when changed text requires new vectors. These are alternative capture paths, not a mandatory chain. Across either path, coordinate the initial load with ongoing events, preserve stable identifiers, handle retries and deletions, and monitor lag against retention limits and application freshness requirements.

Amazon Kinesis Data StreamsAmazon Kinesis Data Streams

Provides a durable streaming layer for change events emitted by supported producers, including AWS DMS for supported configurations. Partition keys influence ordering and workload distribution; the stream does not itself detect changes in an arbitrary database.

AWS LambdaAWS Lambda

Processes change records from DynamoDB Streams or Kinesis Data Streams using event source mappings. Functions can transform records, apply idempotent updates, and coordinate downstream embedding work, with explicit retry and failure handling.

Amazon OpenSearch ServiceAmazon OpenSearch Service

Can serve as a downstream search and vector-search destination for CDC-driven indexing pipelines. Consumers must translate source changes into document upserts, deletions, metadata updates, and embedding regeneration where needed.

Change Data Capture exam questions

Real practice exam questions on Change Data Capture, with the correct answer and its explanation. Answer them with every option in the exam simulator.

DEA-C01

A company uses an AWS DMS task to replicate an on-premises MySQL database to Amazon RDS for MySQL. The initial load is complete, and ongoing replication is current. During planned maintenance, the replication task will be stopped for 24 hours while applications continue writing to the source. After the task restarts, catching up will take no more than 6 hours. The source currently retains binary logs for 8 hours.

The company must preserve all changes made during maintenance and resume replication without reloading the tables. Which approach meets these requirements?

Show answerCORRECT ANSWER

Before maintenance, configure the source to retain binary logs for at least 48 hours. After maintenance, resume the existing task from its saved checkpoint.

Resuming from the saved checkpoint requires the corresponding source binary logs to remain available. A 48-hour retention period covers the 24-hour interruption and the stated catch-up period, allowing DMS to process the intervening changes without a new full load.

SAP-C03

A financial services company is migrating an on-premises PostgreSQL database to Amazon RDS for PostgreSQL by using an AWS Database Migration Service (AWS DMS) full-load-and-CDC task. During a 3-week coexistence period, the on-premises database will remain the only writer. A proposed application change would route regulatory risk queries to the target database. These queries must never use data that is more than 2 seconds behind committed source transactions.

A pilot shows CDC latency below 200 milliseconds. However, the pilot did not include month-end processing, when transaction volume and average row size increase significantly. The company permits risk queries to remain on the source until final cutover. A brief write pause is also permitted at cutover, but no committed transactions can be lost.

Which migration plan meets the requirements?

Show answerCORRECT ANSWER

Keep risk queries on the source throughout coexistence. Test and tune CDC under representative peak workloads. At cutover, pause source writes, confirm that all changes through the final committed source position have been applied, complete validation, and then move application traffic to the target.

AWS DMS CDC does not provide real-time replication or a latency SLA; replication can lag by several minutes or longer. Keeping correctness-sensitive queries on the authoritative source avoids depending on an unsupported 2-second guarantee. Pausing writes and verifying application of all final committed changes provides a controlled cutover without losing committed transactions.

SAA-C03

A company uses an AWS DMS ongoing replication task to ingest changes from an Amazon RDS for MySQL database into Amazon S3. The initial load is complete. During a planned maintenance window, the replication task will be stopped for up to 12 hours while applications continue inserting, updating, and deleting database records. The source currently retains binary logs for 3 hours. Performance testing shows that processing the accumulated changes after maintenance will take no more than 2 hours.

The company must resume ingestion without losing changes or reloading the source tables. Which action should a solutions architect take before maintenance?

Show answerCORRECT ANSWER

Increase source binary log retention to 24 hours, verify sufficient source storage, and resume the task from its saved checkpoint after maintenance.

The source must retain unread changes throughout the outage and subsequent catch-up period. Retaining binary logs for 24 hours provides margin beyond the 12-hour outage and 2-hour catch-up period. DMS can resume from its checkpoint and read those changes without reloading the tables.

Practice Change Data Capture in a full exam

Questions on Change Data Capture in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Change Data Capture

3 min read

Change Data Capture separates an initial dataset from the changes that occur afterward. A pipeline commonly loads a consistent snapshot, records an associated source position, and then processes subsequent inserts, updates, and deletes. Change records typically include a record identifier, operation type, and ordering or checkpoint information; before-and-after values depend on the source configuration. Coordinating the snapshot with the change stream prevents gaps or incorrect results while applications continue writing.

Common capture methods include reading database transaction logs, using database triggers, and querying for records modified after a stored watermark. Log-based capture can reduce repeated table scans, but requires engine-specific configuration, privileges, and sufficient log retention. Triggers add work to source transactions. Timestamp polling is simpler in some environments, but needs reliable modification tracking, careful boundary handling, and a separate mechanism for hard deletes. Incremental polling does not necessarily preserve every intermediate change.

In cloud architectures, a capture component can publish changes to a durable stream, object storage, or a target database. Consumers then transform records and update analytical tables, caches, search indexes, or other derived stores. Choose streaming when freshness requirements justify continuously running infrastructure; use micro-batches or scheduled incremental processing when minutes or hours of delay are acceptable. CDC describes which changes are captured, not a guarantee of real-time processing or a particular ingestion frequency.

For a database migration, a typical approach is to convert incompatible schemas, perform a full load, and replicate ongoing changes until the target catches up. For example, a heterogeneous migration may require translating data types, stored procedures, and indexes separately from moving table data. Before cutover, validate the target, stop or redirect source writes, drain remaining changes, and confirm synchronization. CDC reduces the write interruption but does not automatically eliminate downtime or provide bidirectional conflict resolution. Compare ongoing replication costs with a simpler offline migration when downtime is acceptable.

For foundation-model augmentation, CDC can keep a vector store aligned with authoritative content. A consumer maps each changed document to stable document and chunk identifiers, regenerates embeddings for affected text, and upserts the resulting vectors. It must remove obsolete chunks when documents shrink and delete vectors when source documents are deleted. Metadata-only changes may not require new embeddings, but permission changes still require prompt propagation so retrieval does not expose unauthorized content. Model or chunking changes can require broader reindexing even when source data has not changed.

Design consumers for retries and duplicate delivery unless the entire pipeline provides stronger guarantees. Idempotent upserts, deletion handling, source-version checks, and checkpointing help prevent stale events from overwriting newer state. Ordering is often limited to a record or partition rather than the whole dataset, and a multi-row transaction may not be applied atomically downstream. Monitor end-to-end lag, failed events, throughput, and retained-log coverage. If an outage exceeds the available change history, rebuilding a snapshot may be necessary.

CDC is not a backup, a schema-conversion tool, or a synonym for file transfer. AWS Transfer Family can move files containing snapshots or exported change batches, but it does not itself discover database row changes. Protect capture credentials and change payloads because logs and before-images may contain sensitive values. Periodic reconciliation remains useful for detecting missed deletes, transformation defects, or drift that incremental processing alone does not reveal.

Learn Change Data Capture in lessons

8 short lessons on Change Data Capture in 4 certification courses.

and 2 more lessons

Open the lessons