# Incremental Transfer

Incremental transfer moves only new or changed data after an established baseline instead of repeatedly transferring the entire dataset.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/incremental-transfer

> Incremental transfer reduces recurring data movement by identifying differences between a source and a previously transferred state. It can shorten synchronization windows, reduce bandwidth consumption, and keep cloud destinations current, but it requires reliable change detection, progress tracking, and recovery handling.

Practice exams covering Incremental Transfer: [SAA-C03 AWS Certified Solutions Architect - Associate](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate), [SAP-C03 AWS Certified Solutions Architect - Professional](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional).

## 6 key aspects of Incremental Transfer

Effective incremental transfer combines accurate change detection, coordinated baselines, durable progress, and safe application of changes to reduce recurring data movement without sacrificing correctness.

1. **Baseline followed by changes**: Establish an initial destination state, then transfer differences relative to a known boundary. Coordinate the baseline with ongoing writes so changes occurring during the initial copy are not lost.
2. **Explicit change detection and granularity**: Identify changes using metadata, checksums, transaction logs, or application-maintained markers. Verify whether the transfer unit is an entire object, a file, a block, or a record because that determines actual network savings.
3. **Durable progress tracking**: Persist checkpoints or watermarks so subsequent runs know where to continue. Couple progress advancement to successful destination processing to avoid silently skipping data.
4. **Safe retries and replay**: Assume a failed transfer may repeat previously delivered changes. Use idempotent writes, deduplication, and appropriate ordering rules to keep replay from corrupting the destination.
5. **Defined deletion and consistency semantics**: Decide whether deletions should propagate and how concurrent or out-of-order updates are handled. A change-only copy is not automatically an exact mirror or a transactionally consistent snapshot.
6. **Performance based on change rate**: Size transfer and destination capacity to keep up with incoming changes while meeting freshness targets. Include source scanning, API requests, validation, and reconciliation in cost and performance estimates.

## Cloud services for Incremental Transfer

AWS DataSync applies incremental transfer to supported storage datasets, while AWS DMS handles database baselines and ongoing changes. Amazon S3 can receive transferred files or database change records, depending on the selected service and endpoint configuration. AWS Glue can then process supported inputs incrementally using job bookmarks. These mechanisms track different kinds of progress, so they do not create a shared end-to-end checkpoint automatically. Coordinate retries, deletion handling, target writes, and reconciliation across the pipeline, and monitor both transfer lag and transformation lag to measure when data actually becomes usable.

- **[AWS DataSync](https://blowstack.com/tools/cloud-services-hub/aws-datasync)**: AWS DataSync supports recurring transfers of new or changed data between supported storage locations rather than repeatedly copying the entire dataset. Configure transfer, overwrite, deletion, and verification behavior to match the required destination semantics; change granularity and behavior depend on the storage endpoints.
- **[AWS Glue](https://blowstack.com/tools/cloud-services-hub/aws-glue)**: AWS Glue job bookmarks track processing state for supported sources so recurring ETL jobs can avoid reprocessing previously handled input. Bookmark behavior depends on the source, and bookmarks do not automatically provide delete capture, target cleanup, or end-to-end exactly-once processing.
- **[Amazon S3](https://blowstack.com/tools/cloud-services-hub/amazon-s3)**: Amazon S3 commonly serves as a destination or source for incremental ingestion, with workflows selecting new or changed objects. S3 Replication can replicate eligible new objects and updates after configuration, while existing objects require separate handling such as S3 Batch Replication; object replication does not imply byte-level delta transfer.

## Incremental Transfer exam questions

Real practice exam questions on Incremental Transfer, with the correct answer and its explanation.

### Question 1 (SAA-C03)

**Question:**

A company exports a compressed scientific dataset from an on-premises NFS server to Amazon S3 each night. The export application supports two formats: a single 600 GB file that is replaced each night, or immutable 1 GB segment files with a small manifest identifying the current segments. In the segmented format, each nightly export creates at most 15 new segments and updates the manifest. Previously exported segments are not modified, and cloud applications support either format.

The company has a 100 Mbps internet connection that cannot be upgraded. Each nightly transfer must finish within 1 hour. The initial baseline transfer can occur outside this window. The company wants a managed transfer solution with minimal custom code.

Which solution meets these requirements?

**Correct answer:** Export the segmented format. Configure a scheduled AWS DataSync task to transfer only new or changed files and to overwrite the destination manifest when it changes.

**Explanation:** After the baseline transfer, DataSync can skip unchanged segments and transfer only the new segments and updated manifest. At most approximately 15 GB must be transferred nightly, which fits within the 1-hour window on a 100 Mbps connection without custom change-tracking code.

### Question 2 (SAP-C03)

**Question:**

A geospatial company maintains a 12 TB terrain dataset in Amazon S3 in us-east-1 and a recovery copy in eu-west-1. The dataset consists of multiple large objects. Every night, the application replaces these objects, although only about 0.5% of the dataset's fixed-size, aligned 64 MiB regions contain different bytes. Object sizes and the positions of unchanged data remain stable.

The recovery application must reconstruct an exact nightly dataset entirely from data stored in eu-west-1. It must not expose a nightly dataset until all data referenced by that dataset is available locally. Both buckets have versioning enabled. The company can modify the dataset writer and recovery reader, and a one-time reseeding transfer is acceptable. Previous nightly datasets do not need to be retained after a newer dataset is validated.

Which solution will minimize recurring inter-Region data transfer while meeting these requirements?

**Correct answer:** Store each 64 MiB region as a separate immutable object and maintain a manifest that references the objects for each nightly dataset. Reuse existing objects for unchanged regions and write new objects only for changed regions. Seed the destination, configure S3 Cross-Region Replication for new chunk objects and manifests, and publish a destination manifest only after verifying that every referenced object is available there.

**Explanation:** This layout aligns the replication unit with the application's change granularity. After the initial baseline, unchanged regions reuse objects already present in the destination, so only changed chunks and small manifests require transfer. Destination-side verification prevents asynchronous replication ordering from exposing an incomplete dataset. Unreferenced chunks can subsequently be removed after the newer dataset is validated.

## Practice Incremental Transfer in a full exam

Questions on Incremental Transfer in each practice exam, with every answer option and an explanation for each:

- [SAP-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional): 3 questions
- [SAA-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate): 1 question

## What is Incremental Transfer

Incremental transfer is a data movement strategy in which an initial baseline is followed by transfers containing only new or changed data. Depending on the implementation, a change might be a new object, an updated file, a modified block, or a database row operation. The central design question is how to identify and deliver the required differences without repeatedly moving the entire source dataset.

In cloud architectures, this strategy commonly supports recurring ingestion, migration, replication, and synchronization. For example, a company might initially copy a 20 TB file repository into cloud storage and then transfer approximately 100 GB of daily changes. This can substantially reduce network traffic compared with copying 20 TB every day. Actual transfer volume depends on change granularity: modifying a small portion of a large file might require transferring the whole file if the mechanism does not support block-level differences.

Change detection can use file metadata, object modification times, checksums, database transaction logs, or application-maintained sequence numbers. Each method has limitations. Metadata comparisons can miss changes if timestamps are preserved or unreliable. Computing checksums can require reading substantial amounts of source data even when little is transferred. Database log capture can represent inserts, updates, and deletes, but depends on supported engines, appropriate permissions, and sufficient log retention.

A baseline and its change boundary must be coordinated. During a live database migration, changes that occur while the initial load runs must also reach the destination. A suitable migration mechanism can capture those changes and apply them after or alongside the baseline. Otherwise, the destination may contain gaps or inconsistent records even though every transfer task reports success. Cutover planning should account for replication lag and any period during which source writes must be paused.

Reliable incremental transfer also requires durable progress tracking. A checkpoint, bookmark, or watermark records which source changes have been processed. Advance that state only when the required destination work has completed successfully. Because failures can cause changes to be replayed, use idempotent destination writes, stable record identifiers, or deduplication where appropriate. Timestamp-based ingestion may need an overlapping read window and deduplication to accommodate late-arriving records and timestamp precision limits.

Deletions need explicit treatment. A process that selects records with a recent modification timestamp does not automatically discover rows that have disappeared. Preserving an accurate mirror may require deletion events, tombstones, or periodic reconciliation. Conversely, automatically propagating source deletions may be undesirable for an archive. Define whether the destination is a mirror, an append-only history, or a transformed analytical dataset before choosing synchronization behavior.

Incremental transfer is not synonymous with compression, multipart upload, or resumable transfer. Compression reduces the representation size of data; multipart upload divides an object into upload parts; resumability avoids repeating completed work after an interruption. These mechanisms can complement incremental transfer but do not independently identify changes between dataset versions. Incremental transfer also does not necessarily mean continuous replication: changes can be transferred hourly, nightly, or as a stream.

For architecture questions, look for a large existing dataset, relatively small recurring changes, limited bandwidth, or a short migration cutover window. Evaluate the initial load separately from ongoing updates, and choose a mechanism appropriate to files, objects, or database records. Measure end-to-end freshness, source scanning overhead, destination write capacity, and reconciliation results rather than assuming fewer transferred bytes always means a faster or cheaper solution.

## Learn Incremental Transfer in lessons

3 short lessons on Incremental Transfer in 2 certification courses.

- SAA-C03: Incremental Transfer: change-based synchronization (Premium), https://app.blowstack.com/lessons/incremental-transfer-change-based-synchronization-saa-c03
- SAP-C03: Change Data Capture versus Incremental Transfer (Premium), https://app.blowstack.com/lessons/change-data-capture-versus-incremental-transfer-sap-c03
- SAP-C03: Incremental Transfer: reliable delta synchronization (Premium), https://app.blowstack.com/lessons/incremental-transfer-reliable-delta-synchronization-sap-c03
