# Schema Evolution

Schema evolution is the controlled modification of data structures over time while managing compatibility between existing data, producers, and consumers.

Cloud concept. Updated 2026-10-29. Page: https://blowstack.com/tools/cloud-concepts-hub/schema-evolution

> Data schemas change as applications add fields, revise types, restructure records, or integrate new sources. Schema evolution defines how those changes are introduced, validated, and interpreted so that ingestion pipelines, transformations, tables, and downstream applications continue to work correctly.

Practice exams covering Schema Evolution: [DEA-C01 AWS Certified Data Engineer - Associate](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate), [DVA-C02 AWS Certified Developer - Associate](https://blowstack.com/practice-exams/aws-certified-developer-associate), [SAP-C03 AWS Certified Solutions Architect - Professional](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional).

## 6 key aspects of Schema Evolution

Reliable schema evolution combines compatibility rules, explicit mappings, coordinated deployments, metadata discipline, and validation of both historical and incoming data.

1. **Directional compatibility**: Evaluate whether new readers can consume old data and whether old readers can consume new data. Apply the actual rules of the serialization format and processing engine rather than assuming every additive change is safe.
2. **Explicit source-to-target mapping**: Normalize field names, types, units, timestamp conventions, and null handling when combining sources. Preserve business meaning as well as structural validity.
3. **Controlled change rollout**: Version contracts and coordinate producer, reader, and destination updates. Use staged transitions for breaking changes, and test against historical records and consumers that have not yet upgraded.
4. **Separation of metadata and stored data**: A catalog update changes the published description of data, not necessarily the data itself. Determine whether each change needs only metadata updates, reader changes, a backfill, or a physical rewrite.
5. **Table-format-aware evolution**: Use supported table-format operations for additions, renames, deletions, and type promotions. Stable field identifiers in formats such as Iceberg help preserve column identity, but engine support and downstream dependencies still require validation.
6. **Drift detection and failure isolation**: Compare observed schemas with approved contracts and alert on unexpected differences. Quarantine invalid records and monitor conversion failures so that permissive processing does not silently corrupt the dataset.

## Cloud services for Schema Evolution

A streaming application can publish to Amazon MSK using serializers integrated with AWS Glue Schema Registry, providing versioned event contracts and compatibility checks. AWS Glue or Amazon EMR can then normalize incoming records and write curated tables, including Iceberg tables where supported. The AWS Glue Data Catalog supplies table metadata, while crawlers can discover changes subject to configured update policies. Amazon Athena consumes the curated tables for SQL analysis. Keep these responsibilities separate: a registry governs event contracts, a catalog describes datasets, processing jobs transform values, and a table format manages table metadata. Changes must be tested across the entire path rather than treated as complete when one schema definition is updated.

- **[AWS Glue](https://blowstack.com/tools/cloud-services-hub/aws-glue)**: AWS Glue crawlers discover schemas and populate the AWS Glue Data Catalog, while ETL jobs can explicitly map and convert changing source structures. AWS Glue Schema Registry versions supported event schemas and applies configured compatibility checks; its contract-validation role is distinct from catalog discovery.
- **[Amazon Athena](https://blowstack.com/tools/cloud-services-hub/amazon-athena)**: Athena queries datasets using catalog and table metadata, making mismatches between declared schemas and underlying files operationally important. Its supported Iceberg DDL operations provide controlled table schema changes, but do not automatically update downstream SQL or business logic.
- **[Amazon EMR](https://blowstack.com/tools/cloud-services-hub/amazon-emr)**: Amazon EMR runs processing engines such as Apache Spark that can reconcile source schemas, apply explicit conversions, and work with open table formats such as Iceberg. Supported evolution behavior depends on the selected EMR release, engine, and table-format configuration.
- **[Amazon Managed Streaming for Apache Kafka](https://blowstack.com/tools/cloud-services-hub/amazon-managed-streaming-for-apache-kafka)**: Amazon MSK supports streaming pipelines whose producers and consumers may run different schema versions simultaneously. Applications can integrate AWS Glue Schema Registry through compatible serializers and deserializers; Kafka brokers themselves do not enforce application-level schema compatibility.

## Schema Evolution exam questions

Real practice exam questions on Schema Evolution, with the correct answer and its explanation.

### Question 1 (DEA-C01)

**Question:**

A payment company stores settlement amounts in an Amazon Redshift table. The settlement_amount column is DECIMAL(12,2). A new payment source will send amounts with up to 12 digits before the decimal point and up to 4 digits after it. Financial reports must preserve these amounts exactly, without rounding or approximate numeric representations. Historical amounts must retain their existing values.

The company has approved a maintenance window during which ingestion and reporting can be paused and redirected to a replacement table if necessary. Which solution should a data engineer implement?

**Correct answer:** Create a replacement table with settlement_amount defined as DECIMAL(18,4). During the maintenance window, copy historical rows with an explicit cast to the new type, validate the data, and redirect ingestion and reporting to the replacement table.

**Explanation:** DECIMAL(18,4) supports 14 integer digits and 4 fractional digits, which accommodates the new source while preserving exact decimal values. Existing DECIMAL(12,2) values convert exactly. A replacement table also avoids Redshift's restriction on altering a DECIMAL column's precision and scale in place.

### Question 2 (SAP-C03)

**Question:**

An industrial equipment company is designing an Amazon S3 lakehouse that will be queried through Amazon Athena engine version 3 and registered in the AWS Glue Data Catalog. The dataset will grow to several petabytes.

The initial schema includes a nullable integer column named sensor_offset. A planned application release will rename this column to calibration_offset without changing its meaning. A later release will introduce a new nullable integer column named sensor_offset that represents a different measurement. After both releases, queries must return the original historical values under calibration_offset and NULL under the new sensor_offset for historical rows.

The company requires these schema changes to avoid rewriting historical data files. Analysts must query a single table directly, without version-specific views or query-time transformation logic. The ingestion team can coordinate writer updates with each schema change.

Which design meets these requirements?

**Correct answer:** Use an Apache Iceberg table. Perform a supported metadata-level rename from sensor_offset to calibration_offset, preserving its field ID. Later, add sensor_offset as a new nullable column with a new field ID, and coordinate the writers with each schema version.

**Explanation:** Iceberg tracks columns by stable field IDs rather than relying only on names or positions. Renaming preserves the original field ID, so historical values remain available as calibration_offset without rewriting files. Adding sensor_offset creates a distinct field ID. Historical files do not contain that new field, so those rows return NULL for it.

### Question 3 (DVA-C02)

**Question:**

An application stores invoice documents as JSON objects in Amazon S3. Existing documents contain {"schemaVersion":1,"total":1250}, where total is an integer number of US cents. A new release will write documents such as {"schemaVersion":2,"total":"12.50","currency":"USD"}, where total is a decimal string in the major units of the specified currency. Version 2 also supports currencies other than USD.

A reporting component must read both versions and produce an internal record containing an exact decimal amount in major units and a currency code. Historical objects must not be rewritten, and valid invoice amounts can overlap numerically between the two versions.

Which implementation meets these requirements?

**Correct answer:** Read schemaVersion and apply a version-specific mapping. For version 1, convert integer cents to an exact decimal major-unit amount and set the currency to USD. For version 2, parse the decimal string exactly and preserve its currency. Reject unsupported versions explicitly.

**Explanation:** Version-specific deserialization preserves the meaning of each representation while normalizing both into the same internal model. It supports historical and new documents without rewriting stored objects and avoids guessing units from values.

## Practice Schema Evolution in a full exam

Questions on Schema Evolution in each practice exam, with every answer option and an explanation for each:

- [DEA-C01](https://blowstack.com/practice-exams/aws-certified-data-engineer-associate): 18 questions
- [SAP-C03](https://blowstack.com/practice-exams/aws-certified-solutions-architect-professional): 2 questions
- [DVA-C02](https://blowstack.com/practice-exams/aws-certified-developer-associate): 1 question

## What is Schema Evolution

Schema evolution treats a schema as a versioned contract rather than a permanent description of data. The contract includes field names, types, nullability, nested structures, and sometimes defaults or identifiers. A change must be evaluated against both stored records and running applications: accepting a new record does not necessarily mean that an older transformation or dashboard can interpret it correctly.

Compatibility has a direction. Backward compatibility means that a new reader schema can read data written with an older schema. Forward compatibility means that an older reader schema can read data written with a newer schema. Full compatibility requires both. The exact rules depend on the serialization format, table format, and reader implementation. Adding an optional field is often compatible, but required fields, default values, unknown-field handling, and type conversions must be checked. Transitive compatibility checks extend beyond the immediately preceding version to earlier registered versions.

When integrating multiple sources, first establish a canonical schema and explicit mappings. For example, one customer source might send customer\_id as an integer while another sends alphanumeric identifiers as strings. Converting both to a sufficiently capable string representation can preserve values, but blindly casting the second source to an integer can fail or discard information. Similarly, timestamps require agreement about time zones and precision, and monetary values require consistent currency and decimal scale. Structural compatibility alone does not guarantee semantic compatibility.

A practical rollout might add a nullable loyalty\_tier field to an orders dataset. Update the destination and readers to tolerate the field before producers begin populating it. Historical records can remain without the value if readers correctly interpret its absence; backfill them only when business requirements demand it. A rename requires more care because a name-based system may interpret it as removing one field and adding another. A staged transition can publish both names temporarily, update consumers, and retire the old name after dependencies have migrated.

Open table formats such as Apache Iceberg make schema changes explicit table metadata operations. Iceberg tracks columns using stable field identifiers, allowing supported renames and reordering without confusing one column with another. Supported additions, deletions, and type promotions can generally occur without rewriting existing data files. However, arbitrary type changes are not safe promotions, and metadata-only changes do not convert historical values or update application code. Supported operations also depend on the engine and connector. Partition evolution is a separate capability and should not be confused with changing a table's column schema.

In a cloud data lake, distinguish the physical file schema, the catalog schema, and the schema expected by each processing job. Updating a catalog does not rewrite Parquet files or repair incompatible partitions. AWS Glue crawlers can discover schemas and populate the AWS Glue Data Catalog, but discovery is not a compatibility guarantee. Configure crawler grouping and schema-change behavior deliberately; for controlled datasets, consider logging detected changes instead of automatically replacing production definitions. Validate inferred types against representative files, especially when fields are sparse or vary between source versions.

When transformations fail, compare the incoming records, file metadata, catalog definitions, and job schema. Common causes include strings appearing in numeric fields, nested objects replacing scalar values, missing required fields, incompatible partition schemas, and casts that overflow. Preserve failing records in a quarantine location with diagnostic information rather than silently converting every error to null. Test old and new records together, monitor rejected-record and unexpected-null counts, and distinguish genuine schema changes from malformed data.

Automatic schema merging reduces manual work but can increase planning time and metadata reads across many files. Repeated inference, unnecessary casts, and mixed physical types can also increase processing cost or prevent efficient reads. Prefer explicit schemas for stable production interfaces and use discovery to detect drift. Schema-on-read does not eliminate evolution work: it moves compatibility decisions to readers. A reliable approach combines versioned contracts, compatibility tests, controlled catalog updates, staged deployments, and backfills only when required.

## Learn Schema Evolution in lessons

5 short lessons on Schema Evolution in 2 certification courses.

- DEA-C01: Slowly Changing Dimensions versus Schema Evolution (Premium), https://app.blowstack.com/lessons/slowly-changing-dimensions-versus-schema-evolution-dea-c01
- SAP-C03: Schema Evolution: compatible lakehouse table changes (Premium), https://app.blowstack.com/lessons/schema-evolution-compatible-lakehouse-table-changes-sap-c03
- DEA-C01: Schema Evolution: compatible data contracts (Premium), https://app.blowstack.com/lessons/schema-evolution-compatible-data-contracts-dea-c01
- DEA-C01: Schema Evolution: diagnosing schema-related transformation problems (Premium), https://app.blowstack.com/lessons/schema-evolution-diagnosing-schema-related-transformation-problems-dea-c01
- DEA-C01: Schema Evolution: managing data lake schema metadata (Premium), https://app.blowstack.com/lessons/schema-evolution-managing-data-lake-schema-metadata-dea-c01
