Pagination

Pagination is a request-and-response pattern for retrieving a collection in manageable portions called pages. A request specifies a desired page size and, after the first request, a position from which to continue. The response contains records and information indicating how to request another page. Cloud APIs use pagination to bound response sizes, memory consumption, processing time, and network transfer per request.

Common approaches include page numbers, offsets, and continuation tokens or cursors. An offset-based API might accept limit=100 and offset=200 to request records after the first 200 results. A cursor-based API instead returns a position associated with the previous page. Offset pagination supports convenient navigation but can become expensive for deep database queries. Cursor-based pagination often scales better for sequential traversal, although arbitrary jumps to a particular page may not be supported.

When ingesting data from an API, request the first page, process its records, and follow the documented continuation mechanism until the API indicates completion. Do not assume that a short or empty page means the collection is exhausted. Some APIs apply filters after reading a bounded amount of data, so they can return no matching records while still providing continuation information. Keep filters, sorting, and other parameters consistent across requests as required by the API, and treat continuation tokens as opaque values rather than decoding or modifying them.

For example, an ingestion job can retrieve pages from a supplier API and write each page to object storage before requesting the next one. Persist progress only after the corresponding records are durably processed. A failure between writing records and saving progress can cause the same page to be processed again, so use idempotent writes or deduplication keys. Check whether tokens expire or are valid only within a particular session; a saved token is not necessarily a durable incremental-ingestion checkpoint.

When creating a data API, define a default page size, enforce a maximum, and document the continuation contract. Use deterministic ordering with a unique tie-breaker, such as an event timestamp followed by a record identifier. Validate tokens and bind them to relevant query parameters or authorization context where appropriate. A token must never replace authorization checks on each request. API gateways can expose and validate pagination parameters, but they do not automatically paginate arbitrary backend results.

Pagination does not guarantee a consistent snapshot. Concurrent inserts, updates, and deletes can cause records to move between pages, particularly with offsets. Keyset cursors reduce positional shifts but do not automatically provide snapshot isolation. Exports requiring a fixed view may need database snapshot support, versioned data, or an API-defined snapshot token. Pagination also differs from incremental ingestion: dividing results into pages does not determine which records changed since a previous run.

When calling AWS APIs from code, consult the operation's pagination contract or use an SDK paginator when available. For example, Boto3 provides client.get_paginator('list_objects_v2') for Amazon S3 object listings. A single low-level SDK call generally returns only one service response, whereas a paginator iterates through multiple responses. SDK page-size controls and total-item limits have different purposes, and support varies by operation. Larger pages reduce request overhead but increase per-response memory and retry work; smaller pages increase request count and possible throttling. Apply bounded retries with backoff, process pages incrementally, and monitor request counts, errors, and completed records.