A foundation model's context window defines the information available to it during an inference request. For text models, this capacity is typically expressed in tokens: units that may represent words, parts of words, punctuation, or other text fragments. The context includes more than the user's latest question. System instructions, previous conversation messages, examples, retrieved documents, tool definitions, and tool results can all consume capacity. Multimodal models also account for images, audio, or video according to model-specific processing and limits.
For many generative models, input tokens and generated output must fit within a shared context limit. Providers may also impose separate maximum input and output lengths, so a large context window does not imply an equally large possible response. Applications should check the selected model version and inference API rather than assume every model uses identical accounting. A request can fit the input allowance but still leave insufficient room for the answer the application needs.
Consider a hypothetical model with a 32,000-token shared context window. If instructions and conversation history consume 6,000 tokens and the application reserves 4,000 tokens for the response, no more than approximately 22,000 tokens remain for retrieved evidence and other request material. An engineering implementation should allow additional headroom for message formatting, tool interactions, and estimation differences. Depending on the API or application framework, oversized requests may be rejected or truncated; applications should implement an explicit policy rather than rely on implicit behavior.
In cloud applications, context management usually happens in the application layer surrounding a managed inference endpoint. A support assistant might retain recent conversation turns, summarize older exchanges, and retrieve only the passages relevant to the current question. Retrieval-augmented generation, or RAG, lets the application search a corpus much larger than the context window and insert selected evidence into each request. Chunk size, retrieval count, and reranking influence how effectively the available capacity is used. Summaries conserve space but may omit facts, so important details may need to remain available through retrieval or structured state.
Context length is one foundation-model selection criterion, not a substitute for evaluating the entire workload. A larger window can support long-document analysis or complex conversations, but processing more input generally increases resource use and can raise latency and cost. Model size and context capacity are different properties: a model with more parameters does not necessarily support a longer context. Compare task quality, modality support, multilingual performance, customization options, price, and response time alongside input and output limits. Tokenization varies across languages, so equal-length documents in different languages may require different token budgets.
A larger window also does not guarantee that the model will reliably use every detail. Relevant information can be overlooked, particularly when mixed with substantial unrelated text. Test representative long inputs and measure whether answers use the required evidence, rather than treating the advertised maximum as a quality guarantee. Fine-tuning can adapt behavior but does not inherently expand the supported context window. Likewise, a conversation stored in a database is not automatically visible to the model; the application must supply the relevant information during inference.
Prompt caching can reduce repeated input-processing cost and latency for supported models when requests reuse eligible prompt prefixes. It does not enlarge the context window, and cached material still occupies logical context capacity. Cache eligibility, minimum prefix sizes, expiration, and pricing vary by service and model. For practical model selection, choose a window that accommodates realistic prompts and required answers, then validate token usage, truncation behavior, answer quality, time to first token, and total response time under expected traffic.
Amazon OpenSearch Service