Prompt Compression

Prompt compression is an application-level technique for reducing the token footprint of a foundation model request without materially reducing its usefulness. A prompt includes more than the user's question: system instructions, examples, conversation history, retrieved documents, and tool results can all consume input tokens. Compression targets unnecessary wording and redundant representations while preserving requirements, evidence, and relationships that the model needs to complete the task.

In a cloud application, compression usually runs before model invocation. An application can replace repeated instructions with a concise canonical template, convert verbose tool responses into compact records, or summarize earlier conversation turns. More advanced implementations use a separate model to rewrite context or a model-based algorithm to select informative text. These techniques do not require a foundation model service to provide a dedicated compression feature.

Start with measurement rather than character counts. Tokenization varies by model, language, and data format, so shorter text does not always mean fewer tokens. Use a compatible tokenizer or a provider token-counting API where supported, then record actual usage from inference responses. Track original and compressed input tokens, output tokens, compression processing costs, latency, and task success. Establish a request budget that accounts for the model's context rules, reserved output capacity, and any additional model-specific token requirements.

For example, a support assistant might receive a long incident history containing repeated status messages. A compressed representation can preserve the affected resource identifiers, timestamps, error codes, attempted fixes, and current unresolved question while condensing repeated commentary. Keep exact error strings and configuration values when the answer depends on them. Preserve source identifiers so the assistant can still cite evidence, and retain access controls when storing or reusing compressed context.

Compression works alongside context pruning but is not identical to it. Pruning removes irrelevant documents, obsolete turns, or unnecessary fields; compression represents retained information more compactly. In retrieval-augmented generation, first retrieve and rerank relevant passages, then compress selected context if needed. Summarizing a large collection of irrelevant material is usually less effective than excluding that material before compression.

Response controls address a different part of the token bill. Ask for a concise answer or a specific output schema, and configure the model's supported output-token limit. A hard limit is a ceiling, not a guarantee of a complete response: setting it too low can truncate explanations or produce invalid structured output. Input compression alone does not ensure shorter outputs, so measure both sides independently.

Evaluate compression against an uncompressed baseline using representative tasks and difficult edge cases. Check factual accuracy, instruction adherence, citation quality, tool selection, and structured-output validity. Aggressive compression can remove negation, policy exceptions, temporal ordering, or units. Model-generated summaries can also introduce unsupported claims, so preserve authoritative text for high-risk decisions and provide a fallback when quality checks fail.

Finally, optimize total cost rather than compression ratio. An extra model call can cost more than the input tokens it saves and can increase end-to-end latency. Under fixed-capacity pricing, fewer tokens may improve throughput without immediately reducing the bill. Prompt caching is complementary but distinct: it can reduce repeated-prefix processing costs without shortening the logical prompt. Transport compression such as gzip reduces network bytes, not the tokens processed by the model.