Cloud concept · Updated

Tokenization - Tokenization is the process of converting text into smaller units called tokens and mapping them to numerical identifiers that a language model can process.

In short

Tokens can represent whole words, word fragments, punctuation, whitespace, or other character sequences, depending on the model’s tokenizer. Tokenization determines how much content fits into a model request and directly affects token-metered inference costs, processing requirements, and response latency.

Cloud Concepts Hub →

6 key aspects of Tokenization

Model-specific tokenization governs text representation, context capacity, and the cost and performance of token-metered inference.

01
Model-specific segmentation

Token boundaries and numerical identifiers depend on the tokenizer paired with the model. Count tokens again when changing models rather than carrying over assumptions from the previous model.

02
Tokens are not words

A word can occupy one token or several, and punctuation or whitespace can contribute tokens. Technical identifiers, code, and different languages can have substantially different token counts for similar amounts of visible text.

03
Context-window budgeting

Budget for instructions, history, retrieved content, tool definitions, and the generated response. Enforce the model’s context and output limits before submitting requests.

04
Input and output metering

Token-metered inference commonly applies separate rates to input and output, with model-specific rules for caching, reasoning, or other modalities. Estimate workload cost using expected request volume and measured token usage, not prompt length alone.

05
Latency and resource impact

More input tokens generally require more prompt processing, and more generated tokens generally extend completion time. Measure latency alongside answer quality when shortening prompts or limiting responses.

06
Relationship to chunking and embeddings

Chunking creates passages, tokenization converts text into model-processing units, and embedding models produce vectors. In retrieval pipelines, size passages with the relevant model’s token limits in mind and count the final assembled generation prompt separately.

Cloud services for Tokenization

Managed model APIs such as Amazon Bedrock and Vertex AI can handle tokenization internally while exposing supported token-counting operations and usage measurements. Amazon SageMaker AI offers more control over model serving, including tokenizer packaging and generation configuration. Across these services, an application should assemble its full prompt, estimate tokens using the appropriate model interface, reserve output capacity, and record actual usage after inference. Compare quality, latency, and total cost using representative requests: equal token counts do not imply equal model capability or equal charges. These services are alternative deployment choices and do not need to be combined to implement token-aware applications.

Amazon Bedrock

Amazon Bedrock provides access to foundation models with model-specific token limits and inference pricing. Supported models expose token usage, and the CountTokens API can estimate input token counts for supported model requests before inference.

Amazon SageMakerAmazon SageMaker

Amazon SageMaker AI can host language models with their matching tokenizers in inference containers. Engineers control token limits and generation settings, while endpoint compute charges make token throughput and utilization important even when billing is not directly per token.

Tokenization exam questions

Real practice exam questions on Tokenization, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIB-C01 (beta)

An engineering company uses an Amazon Bedrock foundation model to analyze technical reports. Some reports exceed the model's input-token limit. A team proposes fine-tuning the model on the company's terminology, arguing that the model will then process specialized terms as fewer tokens. The proposed customization preserves the base model's tokenizer and input limit, and the application will submit the same report text. Which assessment should the business sponsor use when evaluating this proposal?

Show answerCORRECT ANSWER

Do not approve the proposal as a solution to the input-capacity problem; require a separate plan to manage report inputs using token counts for the selected model.

With the tokenizer, input limit, and submitted text unchanged, fine-tuning will not reduce the reports' input-token counts. The team needs a separate capacity strategy, such as dividing report processing into appropriate stages, validated with the selected model's token-counting method.

MLA-C02 (beta)

A retailer uses an Amazon SageMaker Data Wrangler flow to prepare short case notes for a linear classifier that predicts whether a refund was approved. The flow lowercases the notes, tokenizes them into words, and produces unigram TF-IDF features. Notes such as “refund approved; exchange denied” and “refund denied; exchange approved” receive identical feature vectors even though their labels differ. The classifier receives only the generated text features.

The engineer must preserve the association between each request type and its immediately following decision without changing the classifier type. Which change to the Data Wrangler flow best meets this requirement?

Show answerCORRECT ANSWER

Add a custom PySpark transformation that generates adjacent word bigrams within each clause while retaining the unigrams. Generate TF-IDF features from the combined tokens, fitting the vocabulary and IDF values on training data and reusing them for validation and inference.

Ordered bigrams such as “refund approved” and “refund denied” preserve the request-decision association that unigram features lose. Data Wrangler supports custom PySpark transformations. Reusing the training-fitted vocabulary and IDF values ensures a consistent feature representation.

AIP-C01

A company operates a multilingual code-analysis application on Amazon Bedrock. The application normally invokes one foundation model but switches to a second model when the first model is throttled. Both models meet the application's quality requirements and support the CountTokens API.

To control input-token costs, an admission gate must accept requests containing at most 12,000 input tokens and reject larger requests before generation. The gate currently counts the complete request with the primary model's tokenizer and caches the result using a hash of the request content. During failover, some requests exceed the token ceiling even though the gate admits them. The problem occurs most frequently with non-English comments and source-code identifiers.

Which change will make the admission gate accurate for both models while retaining the benefit of cached counts?

Show answerCORRECT ANSWER

Count the complete inference input with CountTokens for the selected destination model. Cache the count using both the destination model identifier and a hash of the complete inference input, and apply the admission gate after model selection.

Token boundaries and vocabularies differ between models, especially for multilingual text and code. Counting the complete input for the actual destination model provides the relevant input-token count. Including the model identifier and complete input in the cache key prevents reuse of a count calculated for a different model or request.

Practice Tokenization in a full exam

Questions on Tokenization in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Tokenization

3 min read

Generative language models process sequences of numerical identifiers rather than raw text. A tokenizer splits text into units called tokens and maps those units to identifiers from a vocabulary. Depending on the tokenizer, a familiar word might be one token, while an unfamiliar technical term might become several tokens. Punctuation, spaces, source code, and non-English text also contribute to token counts. A token is therefore not necessarily a word, character, or byte.

The tokenizer must match the model. Common approaches include byte-pair encoding, WordPiece, and unigram tokenization, but their vocabularies and segmentation rules differ. Identical text can produce different token counts across models. Rules of thumb relating English words or characters to tokens are useful for early estimates, not for enforcing production limits or predicting exact charges. Use the selected model’s tokenizer or a provider-supported token-counting interface when available.

Tokenization is distinct from embeddings and chunking. Tokenization identifies the units a model processes; an embedding represents an input, such as a token or document passage, as a numerical vector. Chunking divides documents into larger passages for storage, retrieval, or processing. In retrieval-augmented generation, an application might split a manual into passages, embed those passages for search, and insert selected passages into a prompt. The generation model then tokenizes that complete prompt.

Cloud inference APIs commonly meter input and output tokens separately. Input includes more than the latest user question: system instructions, conversation history, retrieved documents, and tool definitions can all consume tokens. Generated answers consume output tokens. Some models also account for reasoning tokens or use separate rules for cached input and multimodal content. Engineers should inspect the selected model’s pricing and response usage fields rather than assume a universal billing formula.

For a hypothetical model charging $1 per million input tokens and $4 per million output tokens, a request with 2,000 input tokens and 500 output tokens costs $0.004 before any other charges. At 100,000 such requests, token charges total $400. These are illustrative rates, not a cloud service price quote. Provisioned capacity, self-hosted endpoints, and other deployment options may instead charge primarily for allocated compute or throughput.

Models have finite context windows, usually expressed in tokens. For many models, the combined input and generated output must fit within that window, while separate input or output limits may also apply. Reserve room for the answer when constructing a prompt. A maximum-output setting caps generation; it does not guarantee an answer of that length, and setting it too low can truncate useful output.

Token volume also affects performance. Longer inputs generally increase prompt-processing work and memory requirements, while longer outputs generally increase completion time because autoregressive models generate tokens sequentially. Streaming improves perceived responsiveness but does not inherently reduce token usage. Reduce unnecessary history, retrieve only relevant passages, and request concise answers, while testing that these changes preserve accuracy. Removing essential context merely to save tokens can produce worse answers and additional requests.

Finally, language-model tokenization is not security tokenization, which replaces sensitive values with surrogate values. Converting text to token identifiers does not anonymize or encrypt it. Multimodal models may also represent images, audio, or video using model-specific units or token equivalents; text-token estimates alone do not reliably predict their context usage or cost.

Learn Tokenization in lessons

8 short lessons on Tokenization in 4 certification courses.

and 2 more lessons

Open the lessons