# Checkpointing

Checkpointing is the practice of durably saving processing progress and any required execution state so work can resume from a known recovery point.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/checkpointing

> A checkpoint records enough information to recover a task without restarting all of its work, such as source positions, intermediate calculations, or an agent's workflow state. It reduces recovery time and repeated computation, but reliable recovery also depends on retained inputs, compatible state, and safe handling of repeated side effects.

Practice exams covering Checkpointing: [DEA-C01 AWS Certified Data Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate), [DVA-C02 AWS Certified Developer - Associate](https://blowstack.com/practice-exams/aws-certified-developer-associate), [MLA-C02 (beta) AWS Certified Machine Learning Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-machine-learning-engineer-associate), [SAA-C03 AWS Certified Solutions Architect - Associate](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate), [SAP-C03 AWS Certified Solutions Architect - Professional](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional).

## 6 key aspects of Checkpointing

Effective checkpointing combines durable state, consistent progress tracking, replayable inputs, safe side effects, and tested recovery procedures.

1. **Durable recovery boundary**: Save progress and necessary state in storage that survives the relevant worker or process failure. Recovery should use a complete, valid checkpoint rather than partially written state.
2. **Progress and state consistency**: Keep source positions aligned with the processing state they describe. Restoring newer offsets with older aggregation state can silently produce incorrect results.
3. **Replay prerequisites**: Retain source records long enough to cover expected outages and recovery delays. A saved offset cannot recover data that the source has already deleted.
4. **Safe output and side effects**: Expect work after the last successful checkpoint to run again unless stronger coordination prevents it. Use transactional writes, idempotency keys, or deduplication to avoid duplicate output and repeated agent actions.
5. **Checkpoint interval trade-offs**: Choose checkpoint frequency according to acceptable reprocessing time and persistence overhead. Monitor checkpoint duration, failures, age, and state size rather than assuming a configured interval guarantees a fresh recovery point.
6. **State lifecycle and compatibility**: Version persisted state and validate whether a changed job or agent can restore it. Separate checkpoint locations by workload, protect sensitive contents, and test recovery before deleting or migrating state.

## Cloud services for Checkpointing

Kinesis Data Streams can supply retained events while Glue, Spark on EMR, or Managed Service for Apache Flink processes them and maintains framework-specific recovery state. S3 can hold supported checkpoint files and longer-lived raw input archives, allowing recovery beyond a stream's retention window. DynamoDB can hold Kinesis Client Library checkpoints or application-defined agent state with conditional concurrency control. These components do not automatically form one atomic transaction: engineers must coordinate progress with durable output, configure retention, and make repeated writes or tool actions safe.

- **[AWS Glue](https://blowstack.com/tools/cloud-services-hub/aws-glue)**: AWS Glue job bookmarks track processing progress for supported sources and can be reset or rewound for controlled reprocessing. Glue streaming jobs use checkpoints instead of job bookmarks; resetting progress does not remove previously written target data.
- **[Amazon EMR](https://blowstack.com/tools/cloud-services-hub/amazon-emr)**: Spark Structured Streaming running on Amazon EMR can use durable checkpoint storage to recover query progress and state. Recovery depends on compatible query changes, available input data, and the selected sink's delivery guarantees.
- **[Amazon Managed Service for Apache Flink](https://blowstack.com/tools/cloud-services-hub/amazon-managed-service-for-apache-flink)**: The service supports Flink checkpointing for recovery of stateful stream processing and application snapshots for supported operational workflows. End-to-end exactly-once behavior also depends on the source and sink connectors and their configuration.
- **[Amazon Kinesis Data Streams](https://blowstack.com/tools/cloud-services-hub/amazon-kinesis-data-streams)**: Applications using the Kinesis Client Library checkpoint processed sequence numbers, with coordination and checkpoint metadata stored in DynamoDB. Records remain replayable only while they are within the stream's configured retention period.
- **[Amazon DynamoDB](https://blowstack.com/tools/cloud-services-hub/amazon-dynamodb)**: DynamoDB can store application-managed checkpoints and structured agent workflow state using stable execution identifiers. Conditional writes or transactions can coordinate concurrent updates, but applications must still handle external tool actions idempotently.
- **[Amazon S3](https://blowstack.com/tools/cloud-services-hub/amazon-s3)**: Amazon S3 provides durable storage for checkpoint files where supported by the processing framework and for raw input archives used during replay. Keep checkpoint paths separate from input and output data, and ensure lifecycle rules do not prematurely delete active recovery state.

## Checkpointing exam questions

Real practice exam questions on Checkpointing, with the correct answer and its explanation.

### Question 1 (MLA-C02 (beta))

**Question:**

A company uses an Amazon Kinesis Client Library (KCL) application on Amazon EC2 instances to validate events from Amazon Kinesis Data Streams and write training examples to a persistent data store. Each event has a globally unique event_id, and validation produces the same result whenever an event is replayed. The destination supports atomic upserts keyed by event_id, but it cannot participate in a transaction with KCL progress updates.

An instance can fail at any point, including immediately after a destination write succeeds. Replacement instances must resume near the last successfully processed records without omitting events or leaving multiple training rows for the same event. The stream retains all records needed for recovery.

Which implementation meets these requirements?

**Correct answer:** Write training rows by using atomic upserts keyed by event_id. After all records through a sequence number in a shard have been durably written, checkpoint that sequence number through KCL.

**Explanation:** KCL durably tracks progress for each shard so replacement workers can resume from recorded positions. Advancing progress only after successful writes prevents omitted output. A failure before the checkpoint can cause replay, but upserting by the stable event_id makes those repeated writes safe.

### Question 2 (SAP-C03)

**Question:**

A financial services company uses an AWS Step Functions Standard workflow to coordinate settlement processing. One Task state invokes a worker that sequentially processes an immutable Amazon S3 input file in chunks. For each chunk, the worker calculates a settlement amount and adds a ledger entry to an Amazon DynamoDB table. The worker also maintains a running total for a final reconciliation step. Each chunk's updates fit within DynamoDB transaction limits.

A worker can fail at any point, including after a successful DynamoDB write but before receiving its response. Step Functions retries the Task state after a failure. The company requires recovery without recalculating all previously completed chunks. No chunk's settlement amount can be omitted or applied more than once.

Which design meets these requirements?

**Correct answer:** After each chunk, use a DynamoDB transaction to write its ledger entry and update a durable progress record containing the next input position and running total. Use stable chunk identifiers and a conditional progress-version check. On each invocation, read the progress record and resume from that position.

**Explanation:** The transaction makes the ledger output and recovery state an atomic unit. The saved input position and running total allow recovery without recalculating completed chunks. Stable chunk identifiers and conditional version checks prevent a repeated or stale attempt from applying a chunk twice. If a transaction succeeds but its response is lost, reloading the progress record reveals the committed progress.

### Question 3 (DEA-C01)

**Question:**

A logistics company runs a custom Kafka consumer for an Amazon Managed Streaming for Apache Kafka (Amazon MSK) topic. The consumer dispatches records to asynchronous ingestion workers, so records from the same partition can finish processing out of order. A record is complete only after its destination write is durable. During testing, a worker failure followed by a consumer restart causes some records to be skipped.

The company needs the consumer to recover without skipping unfinished records or rereading the entire topic. Reprocessing some completed records is acceptable, and the source records remain available throughout recovery.

Which offset-management strategy meets these requirements?

**Correct answer:** Disable automatic offset commits. For each partition, commit the offset immediately after the highest contiguous sequence of records whose destination writes are durable.

**Explanation:** Kafka commits identify the next offset to read. Advancing each partition's committed offset only past a contiguous sequence of completed records ensures that recovery does not skip unfinished work. A failure before a commit can cause completed records to be processed again, which the scenario permits.

### Question 4 (DVA-C02)

**Question:**

An application uses the Amazon Kinesis Client Library (KCL) to process records from Amazon Kinesis Data Streams. A worker processes each batch concurrently, and all output writes are idempotent. The most recent saved progress position is immediately before the current batch.

The current batch contains records labeled R1 through R10 in shard order. Output writes succeed for every record except R5. The worker must restart because of the processing error, and the replacement worker will resume using the same KCL application's saved progress. There is no separate retry queue or durable record of failed work.

Before restarting, which action should the worker take to ensure that R5 is retried while minimizing replay of successfully processed records?

**Correct answer:** Save progress through R4 by passing R4's sequence number to the KCL checkpointer.

**Explanation:** R4 is the end of the contiguous successfully processed portion of the batch. Saving its sequence number allows the replacement worker to resume after R4 and retry R5. R6 through R10 can be replayed safely because their output writes are idempotent.

### Question 5 (SAA-C03)

**Question:**

An energy company ingests meter readings through Amazon Kinesis Data Streams. An Amazon Managed Service for Apache Flink application maintains each meter's cumulative energy usage since midnight in keyed application state. The stream retains readings for 48 hours.

During a worker failure test, the application loses its running totals and must replay all readings since midnight. The company needs automatic recovery of both the totals and the corresponding input positions. Recovery must normally require replaying no more than approximately 2 minutes of readings. Testing confirms that saving the application's state takes less than 10 seconds, and replacement workers become available within 5 minutes.

Which change will meet these requirements?

**Correct answer:** Enable Flink checkpointing at a 60-second interval and configure automatic failure recovery to restore the latest completed checkpoint, including keyed state and source positions.

**Explanation:** A completed Flink checkpoint consistently captures application state and source positions. Restoring it recovers the accumulated totals and replays only the readings processed after that checkpoint. With the stated checkpoint duration, a 60-second interval meets the approximate 2-minute replay requirement. The stream's retention covers the worker replacement period.

## Practice Checkpointing in a full exam

Questions on Checkpointing in each practice exam, with every answer option and an explanation for each:

- [DEA-C01](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate): 6 questions
- [SAP-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional): 4 questions
- [MLA-C02 (beta)](https://blowstack.com/practice-exams/aws-certified-machine-learning-engineer-associate): 3 questions
- [DVA-C02](https://blowstack.com/practice-exams/aws-certified-developer-associate): 1 question
- [SAA-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-associate): 1 question

## What is Checkpointing

Checkpointing creates a recoverable boundary in a running process. Depending on the workload, a checkpoint can contain an input offset, completed-file metadata, aggregation state, pending work, or a structured representation of an agent's execution. Recovery loads the latest valid checkpoint and continues from that point rather than beginning again. The checkpoint must survive the failure it is intended to address; a file stored only on a disposable worker is insufficient protection against losing that worker.

In ingestion pipelines, checkpointing commonly tracks which source records or objects have been processed successfully. For example, a stream consumer can save a sequence number for each partition after processing its records. After a failure, it resumes from those positions, potentially processing some records again. Partitioned sources generally require progress tracking per partition rather than one global offset.

Checkpointing supports replayability but does not create it by itself. A source must retain the records needed for recovery, or the pipeline must preserve an independent copy, such as raw events in object storage. If a checkpoint refers to records that have expired from a stream, recovery from that position is no longer possible using that stream alone. Replaying an older interval may require selecting an earlier source position or resetting progress metadata and directing output to an isolated destination.

The relationship between output writes and checkpoint advancement is critical. If a consumer writes output and fails before recording progress, recovery can write that output again. If it records progress before output is durable, a failure can cause missing output. Checkpoints alone therefore do not guarantee end-to-end exactly-once processing. Use transactional coordination where supported, or make output operations idempotent through stable event identifiers, deduplication, or upserts. Stateful processing also requires a consistent relationship between source positions and saved operator state.

Cloud frameworks implement different forms of this idea. AWS Glue job bookmarks track supported source-processing progress, whereas AWS Glue streaming jobs use checkpoints for streaming recovery. A bookmark is not a complete snapshot of arbitrary application memory. On Amazon EMR, Spark Structured Streaming can persist query progress and state to a durable checkpoint location. Ordinary Spark caching is different: cached data improves performance but does not, by itself, provide durable execution recovery.

When troubleshooting a pipeline, inspect the last successful checkpoint, checkpoint errors, source retention, storage permissions, and compatibility between saved state and the deployed job. A processing backlog can grow because checkpoint writes are slow or failing. Removing a checkpoint directory can force reprocessing or prevent restoration of stateful computations; it is not a routine repair step. Preserve evidence and establish a controlled replay plan before resetting progress.

For an AI agent, a checkpoint can store conversation state, the current workflow node, completed tool actions, pending approvals, and references to external artifacts. Save it under a stable execution or conversation identifier so another worker can resume the same workflow. Persisting a transcript alone may not capture control flow or outstanding actions. If an agent submits a payment and fails before saving its next checkpoint, recovery must reconcile the action or reuse an idempotency key rather than blindly submit it again.

Checkpoint frequency balances overhead against recovery effort. Frequent checkpoints reduce repeated work but consume storage, network bandwidth, and processing time. Large state may require incremental snapshots or asynchronous persistence, depending on the framework. Protect checkpoint data with access controls and encryption, isolate independent jobs or agent sessions, manage retention, and test restoration after application changes. A checkpoint is an operational recovery mechanism, not a substitute for backups, source retention, or disaster-recovery planning.

## Learn Checkpointing in lessons

5 short lessons on Checkpointing in 4 certification courses.

- MLA-C02 (beta): Checkpointing: resuming agent workflows (Premium), https://app.blowstack.com/lessons/checkpointing-resuming-agent-workflows-mla-c02
- SAP-C03: Checkpointing: recoverable workflow boundaries (Premium), https://app.blowstack.com/lessons/checkpointing-recoverable-workflow-boundaries-sap-c03
- DEA-C01: Checkpointing: recovering data pipelines (Premium), https://app.blowstack.com/lessons/checkpointing-recovering-data-pipelines-dea-c01
- SAA-C03: Checkpointing: recoverable progress in streaming pipelines (Premium), https://app.blowstack.com/lessons/checkpointing-recoverable-progress-in-streaming-pipelines-saa-c03
- MLA-C02 (beta): Checkpointing and conversation memory (Premium), https://app.blowstack.com/lessons/checkpointing-and-conversation-memory-mla-c02
