Cloud concept · Updated

Nucleus Sampling - Nucleus sampling is a text-generation technique that samples each next token from the smallest set of likely tokens whose cumulative probability reaches a configured threshold, called top-p.

In short

Nucleus sampling dynamically limits the next-token choices available to a generative model, balancing response diversity against the exclusion of unlikely continuations. Engineers configure it during inference to influence output variability, usually alongside controls such as temperature and maximum output tokens.

Cloud Concepts Hub →

6 key aspects of Nucleus Sampling

Nucleus sampling controls next-token diversity through an adaptive cumulative-probability cutoff, independently of factuality guarantees and token-length limits.

01
Cumulative-probability threshold

Top-p retains the smallest highest-probability token set whose combined probability reaches the threshold. It is not a per-token probability cutoff or an accuracy score.

02
Adaptive candidate selection

The number of eligible tokens changes at every generation step according to the probability distribution. Unlike top-k, nucleus sampling does not specify a fixed candidate count.

03
Weighted sampling within the nucleus

Retained token probabilities are renormalized before sampling. Tokens with higher probabilities remain more likely to be selected; inclusion does not give all candidates equal chances.

04
Interaction with temperature

Temperature changes the shape of the probability distribution, while top-p limits which candidates remain eligible. Tune one parameter at a time and check model-specific restrictions on combining them.

05
Inference-time configuration

Applications can adjust top-p through supported generation APIs without retraining a model. Record the model version and inference settings to make evaluations and troubleshooting more repeatable.

06
Separate from correctness and length controls

A narrower nucleus can reduce variation but cannot establish factual accuracy or enforce an output schema. Use independent controls for grounding, validation, context size, and maximum output tokens.

Cloud services for Nucleus Sampling

Managed foundation-model APIs such as Amazon Bedrock and Vertex AI let applications configure nucleus sampling without operating the underlying inference infrastructure. SageMaker AI provides an option for hosting models with a chosen serving framework, where that framework defines the sampling interface. These services are alternative deployment choices rather than required stages of one pipeline. Across deployments, applications should validate model-specific parameter support, record generation settings, and run comparable evaluations before changing production defaults. Identical top-p values do not guarantee equivalent behavior across different models or serving implementations.

Amazon Bedrock

Amazon Bedrock exposes top-p for supported foundation models through inference APIs, including the Converse API's topP setting where supported. Check the chosen model's supported parameters and restrictions rather than assuming that all models accept the same sampling configuration.

Amazon SageMakerAmazon SageMaker

SageMaker AI can host generative models whose inference containers implement nucleus sampling. The serving framework and request schema determine whether top_p is available and how it interacts with temperature, top-k, and sampling enablement.

Nucleus Sampling exam questions

Real practice exam questions on Nucleus Sampling, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIF-C01

A publisher uses an Amazon Bedrock foundation model to draft newsletter articles. The model supports topP, and inference charges are based on input and output token counts. To reduce costs, an engineer proposes changing topP from 0.90 to 0.45 while keeping prompts and the maximum output token setting unchanged. The engineer estimates that this change will cut output-token charges in half. Which assessment should guide the publisher's budget?

Show answerCORRECT ANSWER

The estimate is unsupported because topP changes eligible next-token choices, not a token budget. Measure actual token usage with representative requests.

Nucleus sampling restricts next-token candidates according to cumulative probability. Lowering topP generally narrows those choices, but it does not guarantee shorter responses or a proportional reduction in token charges. Budget estimates should use measured input and output usage.

AIP-C01

A company hosts an open-weight foundation model on an Amazon SageMaker AI real-time endpoint to draft product-release summaries. Summaries contain both standardized transitions and free-form descriptions of customer benefits. Editors report occasional unusual wording in the transitions, but they want to preserve the variety of useful benefit descriptions.

The serving container supports temperature, top_k, and top_p. Currently, temperature is 0.7, top_p is 1.0, and top_k filtering is disabled. Decoder diagnostics collected after temperature scaling show two recurring patterns:

• At standardized transitions, the most likely next token has probability 0.94.
• In free-form passages, the five most likely tokens together have probability 0.28, and the 40 most likely tokens together have probability 0.91.

The developer wants sampling to retain only the dominant token in the first pattern, while retaining more than five plausible candidates in the second pattern and excluding the low-probability tail. Temperature must remain unchanged to isolate the effect of candidate filtering.

Which inference configuration meets these requirements?

Show answerCORRECT ANSWER

Keep temperature at 0.7, leave top_k filtering disabled, and set top_p to 0.90.

With top_p set to 0.90, sampling retains the smallest leading set of tokens whose cumulative probability reaches at least 0.90. The token with probability 0.94 is sufficient by itself at standardized transitions. In free-form passages, more than five and no more than 40 candidates are needed. This adapts candidate count to the distribution while excluding the low-probability tail.

Practice Nucleus Sampling in a full exam

Questions on Nucleus Sampling in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Nucleus Sampling

2 min read

When a generative language model produces text, it calculates a probability distribution over possible next tokens. A token can represent a word, part of a word, or punctuation. Nucleus sampling sorts these candidates by probability and retains the smallest leading set whose cumulative probability reaches or exceeds a threshold called top-p. It then renormalizes the probabilities within that set and samples one token. This process repeats as the response is generated.

Suppose the next-token probabilities are 0.50, 0.25, 0.15, and 0.10. With top-p set to 0.80, the model retains the first three candidates because their cumulative probability is 0.90; the first two reach only 0.75. Sampling remains weighted by their relative probabilities rather than becoming a uniform choice. The threshold therefore does not mean that every retained token has an 80 percent probability, nor that the response has an 80 percent chance of being correct.

The candidate set adapts to the model's uncertainty at each generation step. When one token dominates, the nucleus may contain only that token. When probability is spread across many plausible continuations, it may contain many tokens. This distinguishes top-p from top-k sampling, which limits selection to a fixed number of highest-probability tokens. Some inference implementations support both controls, but their interaction depends on the model and serving framework.

Cloud applications typically pass top-p as an inference request parameter to a managed foundation-model API or a model-serving endpoint. Parameter names may include topP or top_p. Support, accepted ranges, defaults, and interactions differ between models, so applications should validate settings against the selected model's API. Changing top-p affects generation behavior without retraining the model, modifying its weights, or expanding its context window.

Lowering top-p generally narrows the available choices and reduces variation. A support application producing standardized explanations might evaluate a lower threshold, while an application drafting alternative product descriptions might evaluate a higher threshold. Neither setting guarantees factual accuracy, safety, valid JSON, or reproducibility. Grounding, schema enforcement, output validation, and access controls address different requirements.

Temperature is related but distinct: it reshapes the probability distribution, while top-p truncates the candidate set using cumulative probability. In common implementations, temperature is applied before nucleus selection, so changing temperature can also change which tokens enter the nucleus. To understand their effects, tune one control at a time while holding the prompt, model version, and other settings constant. A top-p value of 1 generally removes nucleus-based filtering, but does not make generation deterministic.

Input and output length are separate design considerations. Input length affects context usage, cost, and processing requirements; maximum output tokens caps generation length and can truncate an answer. Top-p does not directly set either length or guarantee lower latency or cost. Evaluate candidate settings on representative tasks, measuring factuality, diversity, format compliance, token usage, and latency rather than assuming that a lower threshold is always better.

Learn Nucleus Sampling in lessons

5 short lessons on Nucleus Sampling in 2 certification courses.

Open the lessons