API Versioning

API versioning manages changes to the contract between an API provider and its consumers. That contract includes endpoints, request parameters, response fields, data types, authentication requirements, error semantics, and operational behavior. For data engineers, a contract change can affect more than an HTTP request: it can break ingestion jobs, change analytical results, invalidate checkpoints, or introduce incompatible records into downstream tables.

A breaking change makes a previously supported consumer fail or behave incorrectly. Examples include removing a response field, changing an identifier from a string to a number, requiring a new request parameter, or replacing offset pagination with cursor pagination. Adding an optional response field is often backward compatible, but not universally: consumers that reject unknown fields can still fail. Compatibility must therefore be evaluated against actual consumer expectations, not just a general rule about additive changes.

Providers commonly expose versions through URL paths such as /v1/orders and /v2/orders, query parameters, custom request headers, or versioned media types. Path versions are easy to discover, document, and route, but create separate public addresses. Header-based versions keep resource URLs stable but require careful documentation and cache configuration. When a response depends on a version header, caching infrastructure must distinguish that header's values to avoid returning the wrong representation. No single selection mechanism is correct for every API.

In cloud systems, an API gateway can route versioned requests to separate handlers or services. Both versions might share a database, provided the storage design supports their contracts. An implementation deployment, function version, or environment named production is not automatically an API contract version. Internal code can change repeatedly while the public contract remains v1, and a gateway stage name alone does not guarantee compatibility.

Consider a scheduled ingestion job consuming /v1/orders, where total is a decimal amount. A new /v2/orders contract replaces total with amount_minor and currency. The consumer must update parsing, currency handling, validation, and downstream mappings rather than merely changing the URL. Test representative responses, pagination, empty results, errors, and incremental extraction before switching production traffic. Compare record counts and business totals during a controlled parallel run, while preventing duplicate writes.

Consumers should explicitly select a supported version when the provider offers that capability, rather than relying on an unspecified default. Keep the version configurable and record it with ingestion metadata. Persist pagination cursors and watermarks with enough context to identify their originating contract; a cursor from v1 may not work with v2. Authentication, throttling, retries with backoff, and idempotent writes remain necessary because versioning does not solve availability or delivery problems.

Maintaining multiple versions increases testing, security patching, documentation, and operational costs. Providers should publish migration guidance and retirement dates, monitor usage by version, and contact remaining consumers before removing an old contract. Contract tests and representative consumer tests help detect accidental breaking changes. Semantic version numbers can communicate intent, but numbering alone does not enforce compatibility, and an API version is distinct from an SDK package version or a stored dataset's schema version.