Batch Size | Cloud Concepts

What is Batch Size

Batch size controls how training data is grouped to calculate gradients and update model parameters. It affects accelerator memory, throughput, gradient variability, and the number of updates per epoch, making it an important hyperparameter for both conventional machine learning and foundation model fine-tuning.

In machine learning training, batch size determines how many examples are grouped for a computation. With standard mini-batch gradient descent, the model processes a batch, calculates a loss, computes gradients, and updates its parameters. A batch size of one corresponds to processing individual examples, while full-batch gradient descent uses the entire training dataset for an update. Most deep learning workloads use intermediate sizes to balance computational efficiency and learning behavior.

Batch size, epochs, and steps describe different parts of the training process. An epoch is one pass through the training dataset. A step often means an optimizer update, although some frameworks also use the term for an individual batch-processing iteration. With 10,000 examples, a batch size of 100, and no gradient accumulation, one epoch contains 100 updates. Increasing the batch size to 200 reduces that to 50 updates. Consequently, keeping the number of epochs unchanged does not keep the number of parameter updates unchanged. When the dataset is not evenly divisible by the batch size, the final batch may be smaller or discarded, depending on the data loader configuration.

In the cloud, engineers configure batch size in training scripts, framework data loaders, or supported algorithm hyperparameters. The selected instance must accommodate model parameters, optimizer state, activations, and other runtime allocations. Larger batches generally require more activation memory and can improve accelerator utilization, but throughput eventually reaches a limit. If training fails with an accelerator out-of-memory error, reducing the per-device batch size is a common first adjustment. Larger instances, mixed precision, or activation checkpointing may also help, but each changes cost or execution characteristics.

Smaller batches generally produce noisier gradient estimates, while larger batches generally reduce sampling variability. Neither guarantees better accuracy. Large batches can require changes to learning rate, warmup, or training duration to reach comparable validation performance. Learning-rate scaling rules are starting points rather than universal guarantees. Compare configurations using validation metrics and elapsed time or cost to reach a target quality level, rather than assuming that the highest examples-per-second rate is the best result.

Distributed training introduces a distinction between local and effective batch size. In ordinary synchronous data-parallel training, effective batch size equals the per-device batch size multiplied by the number of data-parallel workers and the number of gradient accumulation steps. For example, four data-parallel GPUs processing eight examples each and accumulating gradients over four iterations produce an effective batch size of 128 examples per optimizer update. GPUs used only for model parallelism do not independently multiply this value. Gradient accumulation allows a larger effective batch without holding all examples' activations simultaneously, but it does not necessarily reproduce every behavior of a physically larger batch, especially with batch-dependent operations.

For foundation model fine-tuning, sequence length also strongly affects memory requirements. Two batches containing the same number of examples can require different memory when their token counts differ. Padding, sequence packing, and token-based batching therefore matter alongside example counts. Training batch size should also not be confused with inference batching: inference groups requests for execution efficiency but does not compute training gradients or update model weights. Record the batch-size definition, accumulation settings, worker count, and sequence-length configuration so that experiments remain reproducible.

Key aspects of Batch Size

Batch size connects training data grouping to memory consumption, computational efficiency, optimizer update frequency, and model convergence.

Relationship to epochs and updates

Increasing batch size reduces the number of optimizer updates per epoch when dataset size and accumulation settings remain fixed. Check whether training schedules and learning-rate schedules are expressed in epochs, batch iterations, or optimizer steps.

Memory and throughput balance

Larger batches can improve accelerator utilization but increase activation memory requirements. Benchmark several sizes and measure memory usage, examples or tokens per second, and training cost rather than automatically choosing the largest batch that fits.

Gradient variability and convergence

Batch size changes the variability of gradient estimates and can alter convergence behavior. Tune it alongside learning rate and evaluate validation quality instead of treating larger or smaller batches as universally superior.

Local versus effective batch size

Distinguish the examples processed by each data-parallel device from the examples contributing to one optimizer update. Record data-parallel worker count and gradient accumulation steps to make the effective batch size explicit.

Gradient accumulation

Accumulating gradients across multiple smaller batches enables a larger effective batch under memory constraints. Ensure loss normalization and optimizer scheduling match the intended update behavior; accumulation does not eliminate the computation required to process those examples.

Variable-length input handling

For language models, example count alone does not describe batch memory requirements. Track token counts, padding, and sequence lengths when comparing configurations or diagnosing out-of-memory failures.

Cloud services for Batch Size

Amazon SageMaker AI and Amazon EC2 provide alternative execution environments for training workloads: SageMaker AI manages training jobs, while EC2 gives engineers direct control over infrastructure and framework configuration. In either environment, batch-size experiments should track validation quality, memory consumption, throughput, and total training cost. Amazon Bedrock provides a managed customization path for supported foundation models, with model-specific hyperparameter controls. These services do not need to be combined in a single workflow; the common practice is to establish a reproducible baseline and adjust batch size together with learning rate, training duration, and any available accumulation settings.

Amazon SageMaker

Managed training jobs run algorithms or training scripts that expose batch size through algorithm-specific hyperparameters or framework configuration. Automatic model tuning can compare supported batch-size choices alongside other hyperparameters using a selected objective metric.

Open Amazon SageMaker →

Amazon EC2

Self-managed training on CPU or accelerator instances lets engineers configure data loaders, distributed workers, and gradient accumulation directly. Instance memory and accelerator capacity constrain the practical per-device batch size.

Open Amazon EC2 →

Amazon Bedrock

For supported foundation model customization workflows, batch size can be an exposed training hyperparameter. Available settings and allowed values depend on the model and customization method, so do not assume the same controls apply to every model.

Open Amazon Bedrock →