Capacity Planning

Capacity planning starts with a workload’s service-level objectives, such as response latency, throughput, availability, and recovery time. Engineers forecast normal demand, seasonal peaks, unexpected bursts, and future growth, then determine the resources needed to meet those objectives. The goal is not maximum utilization at all times: resources running close to saturation can develop long queues and sharply increasing latency. Effective plans provide deliberate headroom while avoiding unnecessary idle capacity.

Translate business demand into technical units before selecting resources. An application serving 1,000 requests per second might be constrained by database connections, storage IOPS, network bandwidth, or CPU rather than the number of application instances. Benchmark representative requests and datasets, measure sustained performance, and identify the bottleneck across the complete request path. Average utilization alone is insufficient; examine peak demand, tail latency, queue depth, and uneven load distribution.

For foundation model workloads, requests per second are not enough to predict capacity. Track input and output token distributions, concurrent requests, context lengths, time to first token, and output generation rate. For example, 10 requests per second averaging 2,000 input tokens and 500 output tokens imply approximately 20,000 input tokens and 5,000 output tokens per second. These are separate demand estimates, not interchangeable units of compute: prompt processing and token generation behave differently, and service quota accounting depends on the provider and model. Long contexts and concurrent sequences also increase memory requirements, including memory used for attention caches in self-hosted inference.

Benchmark the selected model, hardware, and serving configuration under realistic prompt and completion patterns. Dynamic batching can improve accelerator utilization and aggregate throughput, but waiting to assemble batches can increase latency. Large batches can also exhaust memory. Tune batch limits and waiting time against latency objectives, and use asynchronous batch processing when workloads do not require interactive responses. For provisioned model throughput, compare sustained demand with measured usable capacity and contractual costs; consistently underused provisioned capacity can cost more than an on-demand approach.

Autoscaling implements part of a capacity plan rather than replacing it. Configure minimum capacity, maximum capacity, scaling metrics, and cooldown or stabilization behavior around workload characteristics. For model serving, queued work, active sequences, or estimated outstanding tokens may reveal pressure better than CPU utilization alone. Account for instance launch time, model loading, accelerator availability, and metric delays. Scheduled scaling can prepare for predictable peaks, while bounded queues, admission control, and backpressure protect the application during bursts that scaling cannot absorb immediately.

Network planning must consider future address consumption, connection counts, bandwidth, packet rates, and intermediary limits. A subnet that fits the initial deployment may run out of IP addresses during scale-out or rolling replacement. Allocate address space for growth and failover, and verify that load balancers, NAT gateways, private connectivity, and instance network limits can support projected traffic. Adding compute does not resolve an exhausted address pool or a constrained network path.

Storage planning separates capacity in bytes from performance in IOPS, throughput, and latency. Estimate data growth, retention, replication, temporary files, and recovery copies. A volume with enough free space can still throttle because its provisioned performance or the attached instance’s storage bandwidth is insufficient. Managed storage may expand automatically, but access patterns, request limits, partition distribution, and downstream consumers still require planning.

Finally, validate capacity during failures, not just normal operation. If two zones normally share traffic equally, the surviving resources must accommodate the required workload when one zone fails. A standby Region needs appropriate quotas, dependencies, data, and a tested scale-up path before failover. Quota increases do not necessarily propagate across Regions, and an approved quota is permission to consume resources, not a guarantee of available physical capacity. Request increases early, use capacity reservations where supported and necessary, and load-test recovery scenarios. Revisit the plan whenever traffic, models, architecture, or service limits change.