Quantization reduces the numerical precision used to represent a machine learning model's weights, activations, or other tensors. Instead of storing every weight as a higher-precision floating-point number, a deployment might use an 8-bit integer or a packed 4-bit representation. Integer quantization commonly uses scaling factors and sometimes zero points to map between floating-point values and quantized values. Rounding and clipping introduce approximation error, so a quantized model must be evaluated rather than assumed equivalent to the original.
For foundation model hosting, the immediate benefit is often lower weight memory. A model with seven billion parameters requires approximately 14 GB for its weights alone at 16 bits per parameter, compared with approximately 3.5 GB at 4 bits. These are theoretical payload sizes, not total deployment memory requirements. Quantization metadata, unquantized layers, runtime workspaces, activations, and the attention key-value cache also consume memory. Consequently, a model whose quantized weights fit on an accelerator can still run out of memory under long prompts or high concurrency.
Post-training quantization converts an already trained model without repeating full model training. Some approaches use representative calibration inputs to estimate value ranges or identify sensitive weights; other approaches do not require calibration data. Quantization-aware training simulates quantization effects during training or fine-tuning so that the model can adapt. It generally requires more engineering effort and compute than post-training conversion, making post-training methods an attractive starting point when deploying an existing foundation model.
Quantization scope matters. Weight-only quantization reduces weight storage but may retain higher-precision activations and perform computation through specialized kernels that unpack or dequantize weights. Quantizing both weights and activations can enable additional savings or faster arithmetic when the hardware and runtime support it. Quantizing the key-value cache is a separate decision that can help memory usage for long contexts, but weight quantization does not automatically reduce that cache.
In a cloud deployment workflow, first establish a quality and performance baseline, then select a quantization format supported by the intended serving engine and instance hardware. Produce a versioned artifact with the required tokenizer, quantization configuration, and runtime dependencies. Deploy it to a test endpoint and measure task quality, time to first token, output tokens per second, tail latency, memory consumption, and cost at representative prompt lengths and concurrency. A format supported by one GPU runtime may not work efficiently on a different GPU or accelerator.
For example, a quantized model might fit on one accelerator instead of requiring tensor parallelism across two devices. This can simplify hosting and reduce cost, but additional memory headroom may still be necessary for batching and longer contexts. Conversely, a smaller artifact can produce slower inference if unpacking overhead dominates or optimized kernels are unavailable. Reduced storage size is therefore not proof of lower latency.
Common misconceptions include treating quantization as lossless compression, assuming every layer must use the same precision, and confusing it with pruning or distillation. Quantization changes numerical representation; it does not inherently remove parameters or train a smaller replacement model. For deployment decisions, select the lowest precision that satisfies measured quality and service-level requirements, retain the original artifact, and use a controlled rollout with a rollback path.
Amazon EC2