Optimistic Concurrency Control

Optimistic Concurrency Control separates doing work from obtaining permission to publish its results. A writer reads a version of the data, prepares its changes, and attempts to commit only if the relevant assumptions still hold. Validation and publication must be coordinated atomically; checking a version and then performing an unconditional write leaves a race condition. If validation fails, the writer must refresh its state, retry safely, or report a conflict rather than overwrite another writer's changes.

A simple example is an application updating a record with version 12. It submits the new values with a condition that the stored version must still equal 12, and increments the version as part of the same atomic operation. If another request has already advanced the record to version 13, the update fails. The application can then reread the record and decide whether its intended change remains valid. A last-writer-wins overwrite would instead risk silently losing the earlier update.

For open table formats such as Apache Iceberg, the same principle applies to table metadata and snapshots rather than just individual records. Data files normally reside in object storage, while metadata identifies the files belonging to each table snapshot. A writer prepares new files and metadata, validates its operation against intervening commits, and uses the catalog's atomic commit mechanism to publish a new current metadata reference. Readers use a consistent snapshot instead of discovering table contents by simply listing every object in a storage prefix.

Suppose two ingestion jobs append different files to the same Iceberg table. Both may start from the same snapshot. After one commits, the other can often refresh the table state and retry its metadata commit while retaining its prepared files. Two jobs rewriting overlapping files, or a delete running concurrently with a conflicting update, may require stronger validation and can fail rather than merge automatically. Exact conflict rules depend on the operation, engine, and configured isolation level.

In a cloud data lake, object storage durability and strong consistency do not by themselves provide an atomic transaction across all the objects belonging to a table. The table format, catalog, and compatible writer implementations provide the commit protocol. Every writer must follow that protocol; directly replacing files or modifying metadata outside it can invalidate correctness. When selecting a data store, assess transactional table behavior and catalog compatibility, not just storage cost and file-format support.

The main advantage is concurrency without holding exclusive locks throughout potentially lengthy data processing. The cost is wasted work and retries when writers compete heavily. Use bounded retries with backoff for retryable commit conflicts, but do not blindly retry failed semantic validation against stale data. Failed commits can leave unreferenced files, so use supported orphan-file cleanup with a retention period that protects active writes. If a commit response is lost, establish whether the commit succeeded before replaying the operation.

Optimistic concurrency is not the same as multiversion concurrency control, snapshot isolation, or serializable isolation. Versioned snapshots can support the mechanism, while isolation settings determine which anomalies validation must prevent. It also does not make every operation automatically idempotent or guarantee that conflicts never occur. For exam scenarios involving Iceberg, recognize concurrent writers, atomic catalog commits, snapshot-based reads, and safe conflict handling as the central design considerations.