KV Caching | Cloud Concepts

What is KV Caching

During LLM inference, KV caching exchanges additional memory consumption for less repeated computation and faster token generation. Cloud deployments must budget this growing per-request state alongside model weights, because context length and concurrent requests can exhaust accelerator memory even when the model itself fits.

KV caching is an inference optimization used by autoregressive transformer language models. In each attention layer, tokens produce query, key, and value tensors. When generating the next token, the model needs the keys and values associated with earlier tokens. Retaining those tensors allows subsequent decoding steps to process the new token without rerunning the entire preceding sequence. In this context, KV means attention keys and values, not entries in a general-purpose key-value database.

Inference typically has two phases. Prefill processes the input prompt and populates the cache, often using parallel computation across prompt tokens. Decode then generates tokens sequentially, reading previously cached keys and values and appending new ones. Caching reduces repeated work, but it does not eliminate attention over previous tokens. For full-context attention, longer sequences still require more cache storage and more attention-related memory traffic during decoding.

Memory planning must include model weights, runtime workspaces, temporary activations, and the KV cache. For a conventional full-attention decoder with uniform layers, approximate cache storage is 2 × layer count × cached token count × KV head count × head dimension × bytes per element. The factor of two represents keys and values. Total storage must account for all active sequences; actual allocation can differ because of padding, block allocation, architecture, and parallelism. Grouped-query and multi-query attention reduce the number of KV heads relative to query heads, reducing cache requirements.

For example, a model with 32 layers, 8 KV heads, a head dimension of 128, and two-byte cache elements requires approximately 128 KiB per cached token. An 8,192-token sequence therefore requires about 1 GiB for its cache alone. Thirty-two such independent sequences require about 32 GiB before model weights and other allocations. This explains why an endpoint can load successfully yet run out of GPU memory under concurrent long-context traffic.

In cloud deployments, engineers configure an inference container, such as one running vLLM or TensorRT-LLM, to manage cache allocation, context limits, and request scheduling. Block-based cache management, such as PagedAttention implementations, reduces fragmentation and avoids requiring one large contiguous allocation per sequence. Continuous batching lets the scheduler admit new requests as others finish, but admission must respect available token capacity rather than only a fixed request count. Maximum input length, output-token limits, and concurrent sequence limits should be tested together.

Specialized model loading and KV caching solve different deployment problems. Sharded checkpoints, supported weight quantization, and tensor parallelism help load or distribute model weights. They do not automatically provide sufficient memory for runtime caches. Weight quantization also does not imply cache quantization. Lower-precision caches can save memory where supported, but require compatible hardware and kernels and evaluation of output quality. CPU offloading can increase usable capacity, but cache transfers can hurt inter-token latency. Parallel deployments must account for whether their implementation shards or replicates cache tensors.

Ordinary KV caching reuses state within a generation. Cross-request prefix caching is an additional optimization that can reuse compatible cached state for an identical token prefix. It is not response caching, semantic similarity matching, or persistent conversation memory. A chat client that sends the entire conversation on each request does not automatically receive cross-request cache reuse. Reuse depends on the serving engine, model configuration, prefix identity, routing, retention policy, and isolation controls.

Operationally, benchmark realistic prompt lengths, output lengths, and concurrency. Monitor time to first token, inter-token latency, aggregate tokens per second, cache utilization, request queues, and out-of-memory failures. Scaling on GPU utilization alone can miss cache exhaustion or memory-bandwidth bottlenecks. Treat caches as transient state: replica replacement can discard them, and scale-in should drain active generations. Restrict access to cached tensors and isolate cross-request reuse appropriately because caches contain representations derived from user inputs.

Key aspects of KV Caching

KV caching accelerates autoregressive decoding, but effective deployment requires token-based memory planning, cache-aware scheduling, and explicit management of state and reuse.

Reuse of attention state

Store previously computed attention keys and values so each decoding step avoids processing the entire preceding sequence again. Queries for the current token are still computed, and attention still reads relevant cached state.

Token-based memory budgeting

Estimate cache demand from model architecture, cache precision, sequence length, and concurrency. Reserve accelerator memory for both the loaded model and the maximum admitted token workload.

Prefill and decode have different demands

Prefill builds the cache from the prompt, while decode incrementally reads and extends it. Measure time to first token separately from inter-token latency because optimizations can affect these phases differently.

Cache-aware serving and batching

Use inference engines with efficient cache allocation and scheduling. Block-based allocation reduces fragmentation, while token-aware admission control prevents excessive concurrent sequences from exhausting memory.

Memory and performance trade-offs

Cache quantization, offloading, and distributed execution can change memory capacity and latency. Validate each option against the actual model, hardware, inference engine, and workload rather than assuming that saving memory always improves throughput.

Transient state and controlled reuse

A cache normally belongs to an active generation and may disappear when its replica stops. Cross-request prefix reuse requires explicit engine support and compatible inputs, with appropriate tenant isolation and retention controls.

Cloud services for KV Caching

SageMaker AI provides managed model hosting, while EC2 provides direct control over accelerator instances and inference processes. EKS adds Kubernetes orchestration for containerized inference running on suitable worker nodes. These are alternative deployment approaches rather than components that must all be combined. In each approach, the inference engine implements KV caching, and accelerator memory must accommodate weights, caches, and runtime overhead. Use workload measurements to set token limits, admission policies, and scaling thresholds. Replica replacement or scale-in can discard cached state, so deployment and routing policies should drain active generations and avoid assuming cache persistence.

Amazon SageMaker

SageMaker AI endpoints can host compatible LLM inference containers that implement KV caching. Configure engine-specific context limits, cache allocation, and concurrency settings, then select instance capacity using both model-weight and runtime-cache requirements.

Open Amazon SageMaker →

Amazon EC2

GPU-backed EC2 instances provide infrastructure for self-managed inference engines that maintain KV caches in accelerator memory. Engineers control model loading, parallelism, cache precision, and scheduling, while remaining responsible for capacity management and recovery.

Open Amazon EC2 →