Continuous batching schedules inference work at generation-iteration boundaries rather than treating a group of requests as an indivisible batch. During autoregressive decoding, each active sequence typically produces one token per iteration. When a sequence finishes or is canceled, the serving engine removes it and can admit another waiting request without waiting for all other sequences to finish. Requests must still satisfy the engine's compatibility and memory constraints; arbitrary requests for different models cannot simply share one execution batch.
Consider a support assistant receiving three requests that generate 20, 100, and 500 tokens. In a fixed batch without replenishment, the shorter requests finish while the longest continues, leaving fewer active sequences and potentially wasting available capacity. With continuous batching, queued requests can join subsequent iterations as capacity becomes available. This improves aggregate token throughput under sufficient demand, but it does not make an individual request's tokens generate independently or eliminate contention.
LLM inference includes prefill, which processes prompt tokens, and decode, which generates subsequent tokens using cached attention keys and values. Long prompts can consume substantial compute and delay ongoing decoding. Some serving engines support chunked prefill to divide prompt processing into smaller pieces and interleave it with other work. Chunked prefill and paged key-value cache management complement continuous batching, but neither is another name for it nor a universal requirement.
In cloud deployments, continuous batching normally resides inside the model-serving runtime, such as vLLM or TensorRT-LLM, rather than in a load balancer. Engineers deploy a compatible runtime and model in GPU-enabled containers on managed inference endpoints, virtual machines, or Kubernetes. Configuration commonly includes limits on active sequences, scheduled tokens, model context length, and cache allocation. Parameter names and whether a token budget applies per iteration or across admitted work depend on the runtime.
Memory planning must include model weights, runtime overhead, temporary buffers, and the key-value cache for active sequences. Loading weights successfully does not prove that the deployment can support its target concurrency. Longer prompts and outputs generally increase cache demand; excessive admission can cause allocation failures, cache eviction, or recomputation. Quantization and model parallelism can change capacity, but they introduce separate compatibility and performance considerations.
Optimize against service objectives rather than batch size alone. Measure queue time, time to first token, inter-token latency, completed requests per second, input and output tokens per second, cache occupancy, and accelerator utilization. Larger active batches can improve throughput while increasing per-request latency. Benchmark realistic prompt and output distributions, including bursts and cancellations, and use concurrency limits, bounded queues, and backpressure to prevent overload.
Scale replicas using demand and saturation signals together: request count alone misses differences in token workload, while GPU utilization alone can hide memory constraints. Account for model download, weight loading, and warm-up time before new capacity becomes useful. Distributing low traffic across too many replicas can reduce batching opportunities. For managed provisioned throughput, benchmark the exposed capacity and invocation limits rather than assuming control over the provider's internal scheduler.
Continuous batching is not offline batch inference, ordinary request aggregation before execution, or autoscaling. It can serve interactive streaming responses as well as noninteractive generation. Its benefit is keeping existing execution capacity productive; it does not guarantee lower latency, remove token quotas, or compensate for insufficient memory.
Amazon EC2