Generative language models process sequences of numerical identifiers rather than raw text. A tokenizer splits text into units called tokens and maps those units to identifiers from a vocabulary. Depending on the tokenizer, a familiar word might be one token, while an unfamiliar technical term might become several tokens. Punctuation, spaces, source code, and non-English text also contribute to token counts. A token is therefore not necessarily a word, character, or byte.
The tokenizer must match the model. Common approaches include byte-pair encoding, WordPiece, and unigram tokenization, but their vocabularies and segmentation rules differ. Identical text can produce different token counts across models. Rules of thumb relating English words or characters to tokens are useful for early estimates, not for enforcing production limits or predicting exact charges. Use the selected model’s tokenizer or a provider-supported token-counting interface when available.
Tokenization is distinct from embeddings and chunking. Tokenization identifies the units a model processes; an embedding represents an input, such as a token or document passage, as a numerical vector. Chunking divides documents into larger passages for storage, retrieval, or processing. In retrieval-augmented generation, an application might split a manual into passages, embed those passages for search, and insert selected passages into a prompt. The generation model then tokenizes that complete prompt.
Cloud inference APIs commonly meter input and output tokens separately. Input includes more than the latest user question: system instructions, conversation history, retrieved documents, and tool definitions can all consume tokens. Generated answers consume output tokens. Some models also account for reasoning tokens or use separate rules for cached input and multimodal content. Engineers should inspect the selected model’s pricing and response usage fields rather than assume a universal billing formula.
For a hypothetical model charging $1 per million input tokens and $4 per million output tokens, a request with 2,000 input tokens and 500 output tokens costs $0.004 before any other charges. At 100,000 such requests, token charges total $400. These are illustrative rates, not a cloud service price quote. Provisioned capacity, self-hosted endpoints, and other deployment options may instead charge primarily for allocated compute or throughput.
Models have finite context windows, usually expressed in tokens. For many models, the combined input and generated output must fit within that window, while separate input or output limits may also apply. Reserve room for the answer when constructing a prompt. A maximum-output setting caps generation; it does not guarantee an answer of that length, and setting it too low can truncate useful output.
Token volume also affects performance. Longer inputs generally increase prompt-processing work and memory requirements, while longer outputs generally increase completion time because autoregressive models generate tokens sequentially. Streaming improves perceived responsiveness but does not inherently reduce token usage. Reduce unnecessary history, retrieve only relevant passages, and request concise answers, while testing that these changes preserve accuracy. Removing essential context merely to save tokens can produce worse answers and additional requests.
Finally, language-model tokenization is not security tokenization, which replaces sensitive values with surrogate values. Converting text to token identifiers does not anonymize or encrypt it. Multimodal models may also represent images, audio, or video using model-specific units or token equivalents; text-token estimates alone do not reliably predict their context usage or cost.
Amazon SageMaker