Temperature sampling is an inference-time technique that adjusts how a generative model selects its next token. A language model produces scores, called logits, for possible next tokens. In a standard implementation, these scores are divided by a positive temperature value before a softmax function converts them into probabilities. The decoder then samples from the resulting distribution. This process repeats as the response is generated.
A lower temperature makes the distribution more concentrated around the highest-scoring tokens. Responses generally become less varied and more consistent for the same prompt. A higher temperature makes the distribution flatter, increasing the chance of selecting less probable tokens. This can produce more varied wording and unexpected ideas, but it can also increase irrelevant or incoherent output. A temperature of one preserves the original softmax distribution, although a service’s default temperature is not necessarily one.
Cloud applications typically pass temperature as a generation parameter when invoking a hosted foundation model. Parameter names, supported ranges, defaults, and interactions differ across models and APIs. Some models do not expose an adjustable temperature. Engineers should validate settings against the selected model rather than assume that one configuration works identically across providers or model versions.
For example, an application extracting invoice fields might use a low temperature to reduce variation, together with schema-constrained generation where supported and application-side validation. A campaign-writing application might use a higher temperature to generate several distinct slogans, then filter and review the results. Neither setting replaces clear instructions, suitable source material, or evaluation against representative inputs.
Temperature is separate from input and output length. Input length determines how much prompt material and retrieved context the model receives, within its context-window constraints. A maximum output-token setting limits how many tokens the model may generate; it does not guarantee an exact response length. If that limit is too small, a response can stop before completing a sentence or structured object. Raising temperature does not directly increase this limit, expand the context window, or provide access to additional facts.
Temperature also differs from top-p and top-k sampling. Those parameters restrict the candidate tokens considered for selection, while temperature reshapes their relative probabilities. Their combined behavior depends on the inference implementation. Follow model-specific guidance and change one parameter at a time when measuring effects.
A common misconception is that zero temperature guarantees identical or correct answers. Where an API accepts zero, it commonly selects greedy decoding or equivalent lowest-randomness behavior rather than literally dividing by zero. Infrastructure behavior, numerical differences, and model updates can still affect reproducibility. Low-temperature responses can confidently repeat factual errors, and high-temperature responses are not automatically more useful or creative. Evaluate factuality, task success, output validity, diversity, latency, and token consumption separately, and record the model version, prompt, and inference settings for each experiment.
Amazon SageMaker