Mixed Precision Training

Mixed precision training assigns different floating-point formats to different parts of a training workload. Matrix multiplications and convolutions often run in FP16 or BF16, while sensitive calculations, accumulation, and optimizer state may use FP32. The goal is to exploit faster accelerator arithmetic and smaller tensors without introducing unacceptable numerical errors. It is not simply converting every tensor in a model to a lower-precision format.

FP16 and BF16 have different numerical properties. FP16 has a narrower representable range than FP32, so small gradients can underflow to zero and large values can overflow. BF16 retains the exponent range of FP32 but has fewer significant bits, making it less susceptible to range problems while still introducing rounding error. Many training implementations retain FP32 parameters or master copies and FP32 optimizer state so that small updates are not lost. Exact behavior depends on the framework and precision policy.

In cloud environments, engineers select accelerator instances that support efficient arithmetic for the chosen format, then enable a framework-level mixed precision policy. For example, PyTorch provides autocasting through torch.amp.autocast to choose suitable operation types. FP16 training commonly also uses torch.amp.GradScaler. TensorFlow provides mixed precision policies with corresponding optimizer support. Selecting an accelerator instance alone does not guarantee that the training code uses mixed precision.

Loss scaling helps protect small FP16 gradients by multiplying the loss before backpropagation and unscaling gradients before the optimizer update. Dynamic loss scaling adjusts the scale when nonfinite gradients occur and can skip unsafe updates. When gradient clipping is used, gradients should be unscaled before clipping. BF16 training generally does not require loss scaling because of its wider exponent range, although numerical stability still requires monitoring.

For example, an engineer fine-tuning a transformer can establish an FP32 baseline, enable BF16 autocasting on a compatible GPU, and compare step time, peak memory, validation quality, and total training cost. Reduced activation memory may allow a larger microbatch or reduce gradient accumulation steps. However, changing the effective batch size can alter convergence and require learning-rate adjustments. Memory consumption will not necessarily halve because parameters, gradients, optimizer state, and temporary buffers may remain in higher precision.

Mixed precision is most effective when supported accelerator operations dominate execution. Input bottlenecks, small models, unsupported operators, and frequent type conversions can limit or eliminate speedups. Engineers should compare time and cost to reach a target validation metric, rather than judging success only by examples processed per second. Persistent loss spikes, nonfinite values, or degraded validation results warrant investigating unstable operations and retaining higher precision where necessary.

For training-time reduction questions, distinguish mixed precision from early stopping and distributed training. Early stopping avoids unnecessary training after validation improvement stalls; distributed training spreads work across devices; mixed precision changes the arithmetic used during training. These techniques can be combined, but mixed precision does not automatically reduce distributed gradient communication, and it is not equivalent to inference quantization or a guarantee of unchanged accuracy.