Token Budgeting

Token budgeting treats model input and output as resources that must be allocated deliberately. A useful budget specifies how many tokens a request may consume, how much output to reserve, and how much the complete task may cost across multiple model calls. These are different constraints: a request can fit inside a model's context window while exceeding the application's spending target. Budgets should reflect the task, model, pricing structure, and required answer quality rather than simply using the largest supported context.

Before invoking a model, estimate the tokens in the complete request, including system instructions, user messages, conversation history, retrieved passages, tool definitions, and tool results. Use the target model's tokenizer or a supported token-counting API where available. Character counts and word counts are only rough approximations because tokenization varies across models, languages, code, and data formats. Multimodal inputs and reasoning-capable models can introduce additional accounting rules, so follow the selected model's usage definitions.

Reserve space for generation before filling the input context. For a model with a shared 16,000-token context limit, an application might reserve 2,000 tokens for output and 500 for a safety margin, leaving at most 13,500 for the complete input. This example assumes the model counts input and output against a shared limit; separate input, output, or reasoning limits may also apply. Allocate the input allowance among mandatory instructions, the current question, relevant history, and retrieved evidence. When mandatory content alone exceeds the allowance, reject or restructure the request instead of silently deleting essential instructions.

Context-window optimization selects useful information rather than merely truncating text. In retrieval-augmented generation, retrieve candidate passages, rerank them, remove duplicates, and admit the most relevant evidence within the remaining budget. For conversations, retain recent exchanges and important decisions while summarizing older material. Prune obsolete tool results and unrelated messages, but preserve facts, citations, constraints, and dependencies needed to answer correctly. For example, a support assistant should retain the customer's software version and diagnostic findings even when it removes repetitive troubleshooting dialogue.

Prompt compression reduces redundant wording, repeated examples, unnecessary formatting, and verbose tool schemas. Removing JSON whitespace can help, but changing structured data into ambiguous shorthand can make the model less reliable. Summarization and model-based compression also consume tokens and can introduce errors. Evaluate whether their additional cost is justified by savings over subsequent requests, and test that compressed prompts preserve task accuracy and instruction adherence.

Control response size through concise-answer instructions, appropriate output-token limits, and supported stopping mechanisms. Asking for three sentences guides behavior but does not enforce a hard token ceiling. Conversely, a small output limit can truncate a valid answer or leave structured output incomplete. Inspect completion status and validate outputs before accepting them. Shortening a response after generation does not recover the tokens already generated, and cancelling a stream should not be assumed to eliminate charges for work already performed.

Track actual input and output usage, latency, truncation rates, quality scores, and estimated cost by task, tenant, and model. Include retries, agent iterations, tool-call follow-ups, and summarization in the task budget; otherwise, a cheap individual request can become an expensive workflow. Reconcile estimates with returned usage metadata and applicable pricing, distinguishing cached and uncached input where relevant. Use representative evaluations to tune budgets and graceful fallback behavior. The objective is lower cost per successful task, not the fewest tokens regardless of correctness.