Context Pruning

Context pruning is an input optimization technique for applications that invoke foundation models. Instead of sending every available conversation turn, document passage, or tool result, the application selects the information necessary for the current request. The objective is not simply to minimize tokens: it is to reduce unnecessary input while maintaining task effectiveness. This matters in cloud applications because repeated model invocations can repeatedly incur charges for the same irrelevant history or oversized retrieval results.

A practical implementation starts with a token budget. Determine the selected model’s context-window rules, reserve capacity for the intended response, and account for system instructions, tool definitions, message formatting, and other request overhead. For a model with a shared 32,000-token input-and-output window, reserving 4,000 tokens for output and 1,000 for overhead leaves at most 27,000 tokens for the remaining input. This is a ceiling, not a target. Tokenization varies by model, so use a compatible tokenizer or a provider-supported token-counting API rather than assuming a fixed number of characters per token.

For conversational applications, pruning often removes obsolete exchanges while retaining the current request, recent relevant turns, persistent user constraints, and unresolved decisions. A travel assistant might discard discussion of a rejected destination but retain travel dates, accessibility requirements, and the agreed budget. Removing the oldest messages unconditionally is risky because an early message may contain the most important constraint. Preserve valid message structure and any tool-call and tool-result relationships required by the model API.

For retrieval-augmented generation, prune retrieved material before constructing the final prompt. Apply authorization and metadata filters, retrieve candidate passages, remove duplicates, rerank for relevance, and include useful passages until the input budget is reached. A support assistant answering a question about a specific software version should exclude unrelated versions and repeated excerpts. However, relevance ranking alone can omit exceptions or complementary evidence, so selection should consider coverage and retain source identifiers for citations.

Tool outputs are another common source of waste. An application investigating an incident can select relevant log fields and a bounded time interval instead of inserting a complete log export. Store the full results outside the model context when needed, and allow additional retrieval if the retained information is insufficient. Removing content from a prompt does not delete it from storage or change its retention policy.

Pruning differs from compression and summarization. Pruning removes selected material; compression or summarization represents information more compactly and can introduce its own information loss. These techniques can be combined, but neither guarantees equivalent answers. Likewise, setting a maximum output length or asking for a concise response controls generated tokens rather than removing input tokens. A larger context window also does not eliminate the cost of sending unnecessary content.

Evaluate pruning policies against representative tasks and difficult cases before deployment. Track estimated and actual input tokens, output tokens, latency, answer correctness, citation quality, and follow-up retrieval frequency. Include the cost and latency of reranking or summarization when calculating savings. Aggressive pruning can remove critical evidence, cause additional calls, or increase unsupported answers. Preserve trusted instructions and authorization constraints, treat retained external text as untrusted, and provide a fallback that retrieves more evidence or requests clarification when the remaining context is insufficient.