Model Parallelism

Model parallelism distributes the computation and parameters of one model across multiple devices, usually GPUs or other accelerators. The devices work together on the same model execution rather than independently hosting complete copies. This is particularly important for foundation model deployment, where model weights, temporary tensors, and attention caches can exceed the memory available on one accelerator.

Two common approaches are tensor parallelism and pipeline parallelism. Tensor parallelism splits operations within layers across devices, such as distributing a large matrix multiplication and combining its partial results. Pipeline parallelism assigns different groups of layers to different devices and passes intermediate activations between them. Pipeline stages can process different microbatches concurrently, but uneven stage workloads and pipeline bubbles can leave devices idle. A deployment can combine both approaches when the model architecture and serving framework support them.

For cloud hosting, first estimate the memory needed by the entire serving workload. A model with 70 billion parameters stored at two bytes per parameter requires approximately 140 GB just for weights, before runtime overhead, activations, and the key-value cache used during autoregressive generation. Two GPUs with 80 GB each might accommodate those weights, but the remaining memory may be insufficient for the desired context length and concurrent requests. Aggregate GPU memory is not a single automatically shared pool: the runtime must explicitly partition the model, and each device must have enough memory for its assigned workload.

Next, select an accelerator topology and a compatible inference engine. A multi-GPU instance with high-bandwidth device interconnects is often a simpler starting point than splitting a latency-sensitive model across hosts. Cross-host deployment additionally requires supported distributed serving, coordinated processes, and adequate network bandwidth. Configure the parallelism degree, such as the number of devices participating in tensor parallel execution, according to the model architecture, accelerator count, and framework constraints. Merely choosing a machine with several GPUs does not automatically distribute the model.

Model parallelism trades memory capacity and potentially faster computation for communication and synchronization costs. Tensor-parallel execution commonly exchanges intermediate results throughout the forward pass, making interconnect performance important. Increasing the number of devices can therefore increase cost without improving response latency. Benchmark time to first token, time per output token, throughput, accelerator utilization, and memory consumption using realistic prompt lengths and concurrency. Longer contexts and larger batches can substantially increase cache memory requirements even when model weights remain unchanged.

Distinguish partitioning one model from scaling complete model replicas. Adding endpoint instances normally increases the number of independent replicas; it does not automatically combine their accelerator memory into one larger model host. If one replica needs four GPUs, scaling to three replicas generally requires three complete four-GPU allocations. Health checks, startup coordination, and recovery must also account for the fact that a failed participating process can make its whole model replica unavailable.

For deployment decisions, use model parallelism when model capacity or measured performance justifies it. Quantization may reduce memory requirements enough to use fewer devices, while request batching and replica scaling address different throughput needs. These techniques can be combined, but none eliminates the need to validate model quality, memory headroom, latency objectives, and cost per completed request.