Cloud concept · Updated

Checkpointing - Checkpointing is the practice of durably saving processing progress and any required execution state so work can resume from a known recovery point.

In short

A checkpoint records enough information to recover a task without restarting all of its work, such as source positions, intermediate calculations, or an agent's workflow state. It reduces recovery time and repeated computation, but reliable recovery also depends on retained inputs, compatible state, and safe handling of repeated side effects.

Cloud Concepts Hub →

6 key aspects of Checkpointing

Effective checkpointing combines durable state, consistent progress tracking, replayable inputs, safe side effects, and tested recovery procedures.

01
Durable recovery boundary

Save progress and necessary state in storage that survives the relevant worker or process failure. Recovery should use a complete, valid checkpoint rather than partially written state.

02
Progress and state consistency

Keep source positions aligned with the processing state they describe. Restoring newer offsets with older aggregation state can silently produce incorrect results.

03
Replay prerequisites

Retain source records long enough to cover expected outages and recovery delays. A saved offset cannot recover data that the source has already deleted.

04
Safe output and side effects

Expect work after the last successful checkpoint to run again unless stronger coordination prevents it. Use transactional writes, idempotency keys, or deduplication to avoid duplicate output and repeated agent actions.

05
Checkpoint interval trade-offs

Choose checkpoint frequency according to acceptable reprocessing time and persistence overhead. Monitor checkpoint duration, failures, age, and state size rather than assuming a configured interval guarantees a fresh recovery point.

06
State lifecycle and compatibility

Version persisted state and validate whether a changed job or agent can restore it. Separate checkpoint locations by workload, protect sensitive contents, and test recovery before deleting or migrating state.

Cloud services for Checkpointing

Kinesis Data Streams can supply retained events while Glue, Spark on EMR, or Managed Service for Apache Flink processes them and maintains framework-specific recovery state. S3 can hold supported checkpoint files and longer-lived raw input archives, allowing recovery beyond a stream's retention window. DynamoDB can hold Kinesis Client Library checkpoints or application-defined agent state with conditional concurrency control. These components do not automatically form one atomic transaction: engineers must coordinate progress with durable output, configure retention, and make repeated writes or tool actions safe.

AWS GlueAWS Glue

AWS Glue job bookmarks track processing progress for supported sources and can be reset or rewound for controlled reprocessing. Glue streaming jobs use checkpoints instead of job bookmarks; resetting progress does not remove previously written target data.

Amazon EMRAmazon EMR

Spark Structured Streaming running on Amazon EMR can use durable checkpoint storage to recover query progress and state. Recovery depends on compatible query changes, available input data, and the selected sink's delivery guarantees.

Amazon Managed Service for Apache FlinkAmazon Managed Service for Apache Flink

The service supports Flink checkpointing for recovery of stateful stream processing and application snapshots for supported operational workflows. End-to-end exactly-once behavior also depends on the source and sink connectors and their configuration.

Amazon Kinesis Data StreamsAmazon Kinesis Data Streams

Applications using the Kinesis Client Library checkpoint processed sequence numbers, with coordination and checkpoint metadata stored in DynamoDB. Records remain replayable only while they are within the stream's configured retention period.

Amazon DynamoDBAmazon DynamoDB

DynamoDB can store application-managed checkpoints and structured agent workflow state using stable execution identifiers. Conditional writes or transactions can coordinate concurrent updates, but applications must still handle external tool actions idempotently.

Amazon S3Amazon S3

Amazon S3 provides durable storage for checkpoint files where supported by the processing framework and for raw input archives used during replay. Keep checkpoint paths separate from input and output data, and ensure lifecycle rules do not prematurely delete active recovery state.

Checkpointing exam questions

Real practice exam questions on Checkpointing, with the correct answer and its explanation. Answer them with every option in the exam simulator.

MLA-C02 (beta)

A company uses an Amazon Kinesis Client Library (KCL) application on Amazon EC2 instances to validate events from Amazon Kinesis Data Streams and write training examples to a persistent data store. Each event has a globally unique event_id, and validation produces the same result whenever an event is replayed. The destination supports atomic upserts keyed by event_id, but it cannot participate in a transaction with KCL progress updates.

An instance can fail at any point, including immediately after a destination write succeeds. Replacement instances must resume near the last successfully processed records without omitting events or leaving multiple training rows for the same event. The stream retains all records needed for recovery.

Which implementation meets these requirements?

Show answerCORRECT ANSWER

Write training rows by using atomic upserts keyed by event_id. After all records through a sequence number in a shard have been durably written, checkpoint that sequence number through KCL.

KCL durably tracks progress for each shard so replacement workers can resume from recorded positions. Advancing progress only after successful writes prevents omitted output. A failure before the checkpoint can cause replay, but upserting by the stable event_id makes those repeated writes safe.

SAP-C03

A financial services company uses an AWS Step Functions Standard workflow to coordinate settlement processing. One Task state invokes a worker that sequentially processes an immutable Amazon S3 input file in chunks. For each chunk, the worker calculates a settlement amount and adds a ledger entry to an Amazon DynamoDB table. The worker also maintains a running total for a final reconciliation step. Each chunk's updates fit within DynamoDB transaction limits.

A worker can fail at any point, including after a successful DynamoDB write but before receiving its response. Step Functions retries the Task state after a failure. The company requires recovery without recalculating all previously completed chunks. No chunk's settlement amount can be omitted or applied more than once.

Which design meets these requirements?

Show answerCORRECT ANSWER

After each chunk, use a DynamoDB transaction to write its ledger entry and update a durable progress record containing the next input position and running total. Use stable chunk identifiers and a conditional progress-version check. On each invocation, read the progress record and resume from that position.

The transaction makes the ledger output and recovery state an atomic unit. The saved input position and running total allow recovery without recalculating completed chunks. Stable chunk identifiers and conditional version checks prevent a repeated or stale attempt from applying a chunk twice. If a transaction succeeds but its response is lost, reloading the progress record reveals the committed progress.

DEA-C01

A logistics company runs a custom Kafka consumer for an Amazon Managed Streaming for Apache Kafka (Amazon MSK) topic. The consumer dispatches records to asynchronous ingestion workers, so records from the same partition can finish processing out of order. A record is complete only after its destination write is durable. During testing, a worker failure followed by a consumer restart causes some records to be skipped.

The company needs the consumer to recover without skipping unfinished records or rereading the entire topic. Reprocessing some completed records is acceptable, and the source records remain available throughout recovery.

Which offset-management strategy meets these requirements?

Show answerCORRECT ANSWER

Disable automatic offset commits. For each partition, commit the offset immediately after the highest contiguous sequence of records whose destination writes are durable.

Kafka commits identify the next offset to read. Advancing each partition's committed offset only past a contiguous sequence of completed records ensures that recovery does not skip unfinished work. A failure before a commit can cause completed records to be processed again, which the scenario permits.

Practice Checkpointing in a full exam

Questions on Checkpointing in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Checkpointing

3 min read

Checkpointing creates a recoverable boundary in a running process. Depending on the workload, a checkpoint can contain an input offset, completed-file metadata, aggregation state, pending work, or a structured representation of an agent's execution. Recovery loads the latest valid checkpoint and continues from that point rather than beginning again. The checkpoint must survive the failure it is intended to address; a file stored only on a disposable worker is insufficient protection against losing that worker.

In ingestion pipelines, checkpointing commonly tracks which source records or objects have been processed successfully. For example, a stream consumer can save a sequence number for each partition after processing its records. After a failure, it resumes from those positions, potentially processing some records again. Partitioned sources generally require progress tracking per partition rather than one global offset.

Checkpointing supports replayability but does not create it by itself. A source must retain the records needed for recovery, or the pipeline must preserve an independent copy, such as raw events in object storage. If a checkpoint refers to records that have expired from a stream, recovery from that position is no longer possible using that stream alone. Replaying an older interval may require selecting an earlier source position or resetting progress metadata and directing output to an isolated destination.

The relationship between output writes and checkpoint advancement is critical. If a consumer writes output and fails before recording progress, recovery can write that output again. If it records progress before output is durable, a failure can cause missing output. Checkpoints alone therefore do not guarantee end-to-end exactly-once processing. Use transactional coordination where supported, or make output operations idempotent through stable event identifiers, deduplication, or upserts. Stateful processing also requires a consistent relationship between source positions and saved operator state.

Cloud frameworks implement different forms of this idea. AWS Glue job bookmarks track supported source-processing progress, whereas AWS Glue streaming jobs use checkpoints for streaming recovery. A bookmark is not a complete snapshot of arbitrary application memory. On Amazon EMR, Spark Structured Streaming can persist query progress and state to a durable checkpoint location. Ordinary Spark caching is different: cached data improves performance but does not, by itself, provide durable execution recovery.

When troubleshooting a pipeline, inspect the last successful checkpoint, checkpoint errors, source retention, storage permissions, and compatibility between saved state and the deployed job. A processing backlog can grow because checkpoint writes are slow or failing. Removing a checkpoint directory can force reprocessing or prevent restoration of stateful computations; it is not a routine repair step. Preserve evidence and establish a controlled replay plan before resetting progress.

For an AI agent, a checkpoint can store conversation state, the current workflow node, completed tool actions, pending approvals, and references to external artifacts. Save it under a stable execution or conversation identifier so another worker can resume the same workflow. Persisting a transcript alone may not capture control flow or outstanding actions. If an agent submits a payment and fails before saving its next checkpoint, recovery must reconcile the action or reuse an idempotency key rather than blindly submit it again.

Checkpoint frequency balances overhead against recovery effort. Frequent checkpoints reduce repeated work but consume storage, network bandwidth, and processing time. Large state may require incremental snapshots or asynchronous persistence, depending on the framework. Protect checkpoint data with access controls and encryption, isolate independent jobs or agent sessions, manage retention, and test restoration after application changes. A checkpoint is an operational recovery mechanism, not a substitute for backups, source retention, or disaster-recovery planning.

Learn Checkpointing in lessons

5 short lessons on Checkpointing in 4 certification courses.

Open the lessons