Gradient Clipping

Gradient clipping is a training technique for controlling excessively large gradients. Backpropagation computes gradients that describe how model parameters affect the loss. When those gradients become very large, an optimizer can make disruptive updates, causing loss spikes, oscillation, numerical overflow, or divergence. Clipping constrains the gradients before the optimizer applies its update, reducing the impact of unusually large gradient magnitudes.

The two common approaches are clipping by norm and clipping by value. Global norm clipping treats the gradients across a specified collection of parameters as one vector. If its norm exceeds a threshold, all included gradients are scaled by the same factor. For example, if the global L2 norm is 20 and the threshold is 1, multiplying every gradient by 0.05 produces a norm of 1 while preserving the vector's direction. Value clipping instead clamps individual gradient components to an interval such as -1 to 1. This can change the overall gradient direction. Clipping each parameter tensor separately is also different from clipping the global norm across the whole model.

In cloud training, clipping normally belongs in the framework training code or supported trainer configuration, not in the infrastructure configuration. A managed training job can run a PyTorch script that performs backpropagation, calls torch.nn.utils.clip_grad_norm_ on the model parameters, and then calls optimizer.step(). TensorFlow optimizers provide clipping options such as clipnorm, global_clipnorm, and clipvalue, whose semantics should be checked before use. A GPU instance does not automatically enable clipping simply because it hosts the training workload.

For convergence debugging, inspect training loss, validation loss, learning rate, gradient norms before clipping, and the fraction of steps that trigger clipping. A recurrent model that develops large gradient norms followed by sudden loss spikes is a reasonable candidate for clipping. However, those symptoms are not conclusive: malformed examples, incorrect loss reduction, poor initialization, and an excessive learning rate can produce similar behavior. Compare controlled runs and inspect the batches associated with instability rather than treating clipping as a substitute for diagnosis.

The clipping threshold is a hyperparameter, not a universal constant. An overly restrictive threshold can slow learning or suppress useful gradient information. A threshold that is too large may provide little protection. Interpret thresholds in the context of loss scaling, batch size, gradient accumulation, and whether gradients are summed or averaged. With gradient accumulation, clipping is commonly applied after the intended accumulation is complete and before the optimizer update; clipping every microbatch changes the resulting update.

Mixed-precision and distributed training require particular care. With loss scaling, unscale gradients before applying the clipping threshold, using the framework's supported procedure. Clipping cannot reliably repair gradients that already contain NaN or infinity. In distributed training, use clipping that respects the gradient synchronization and parameter-sharding strategy; a local shard norm is not necessarily the full model's global norm.

Clipping addresses excessively large gradients, not vanishing gradients, overfitting, or insufficient model capacity. It complements learning-rate tuning but is not equivalent to lowering the learning rate: norm clipping changes gradients only when the threshold is exceeded. It also bounds gradients rather than guaranteeing a bound on the final parameter update for every optimizer, because momentum, adaptive scaling, and weight decay can affect that update. For exam scenarios involving exploding gradients and unstable convergence, clipping is a targeted mitigation, while continued monitoring verifies whether training actually improves.