Cloud concept · Updated

Context Window - A context window is the maximum amount of information a foundation model can consider within a single inference, typically measured in tokens.

In short

The context window limits how much instruction, conversation history, retrieved information, and generated output a model can handle together. Engineers use this limit when selecting foundation models and designing prompts, retrieval pipelines, and conversation management so requests fit without sacrificing relevant information or exceeding cost and latency targets.

Cloud Concepts Hub →

6 key aspects of Context Window

Effective context-window design balances model-specific capacity, answer space, relevant evidence, and measured cost, latency, and quality.

01
Finite inference capacity

Instructions, conversation messages, retrieved evidence, and tool-related content share a bounded capacity. Budget for the complete request rather than measuring only the user's question.

02
Input and output budgeting

Many models require input and output to fit within a shared context limit while also enforcing separate output caps. Reserve space for the expected answer and verify the exact limits of the selected model and API.

03
Model-specific token accounting

Tokens are not equivalent to words, and token counts vary across languages and tokenizers. Multimodal inputs introduce additional accounting rules and limits that must be checked before deployment.

04
Selective context assembly

Retrieval, reranking, history pruning, and summarization help retain useful information within the window. These techniques require quality testing because removing or compressing content can discard necessary evidence.

05
Capacity versus effectiveness

A model's maximum supported context length does not guarantee reliable reasoning over every included detail. Evaluate relevant-information retrieval and answer accuracy at the input lengths the application will actually use.

06
Cost, latency, and caching

Longer inputs generally require more processing and can increase cost and latency. Supported prompt caching can reduce repeated processing, but it neither removes cached tokens from the context budget nor expands the window.

Cloud services for Context Window

An application can use Amazon OpenSearch Service to retrieve relevant passages, assemble them with instructions and conversation history, and invoke a foundation model through Amazon Bedrock or an endpoint hosted on Amazon SageMaker AI. The application must enforce a token budget that leaves room for the answer and any tool interactions. Bedrock offers model-dependent prompt caching, while SageMaker AI provides control over model serving configurations. Across either inference path, engineers should measure token usage, response quality, latency, and cost rather than assume that a larger context window improves every workload.

Amazon Bedrock

Amazon Bedrock provides access to foundation models with different context windows, output limits, modalities, and pricing. Applications must apply the selected model's limits, and supported models offer prompt caching for eligible repeated prompt content.

Amazon OpenSearch ServiceAmazon OpenSearch Service

Amazon OpenSearch Service supports vector and lexical retrieval to select relevant document passages before model invocation. This helps applications search large corpora while placing only selected evidence into the model's finite context window.

Context Window exam questions

Real practice exam questions on Context Window, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIB-C01 (beta)

A company's employee assistant uses Knowledge Bases for Amazon Bedrock to search an archive that has grown to millions of tokens. For each question, the application retrieves relevant passages and sends them to a foundation model with a documented 32,000-token context window shared between input and output. Monitoring confirms that complete requests, including instructions, conversation history, and retrieved passages, use at most 12,000 input tokens. Responses are limited to 4,000 tokens, and answer quality meets requirements. A business sponsor asks whether the growing archive requires a model with a larger context window. What is the BEST recommendation?

Show answerCORRECT ANSWER

Keep the current model and monitor assembled request sizes as the archive grows, because the supplied input and allowed output fit within its context window.

The current maximum input of 12,000 tokens plus up to 4,000 output tokens fits comfortably within the 32,000-token shared limit. Archive growth alone does not require a larger window, although changes in retrieval or conversation length should continue to be monitored.

MLA-C02 (beta)

A company is expanding an Amazon Bedrock document-rewriting application from English to Japanese and German. Documents have the same maximum character count in all three languages. The application must preserve the complete document and allow a response of up to 4,000 tokens without truncation.

Several candidate foundation models meet the rewriting-quality requirements and advertise the same shared input-and-output context-window size. However, production-sized documents that work with one candidate produce input-length errors with another. Short documents work with both, and testing shows no throttling.

Which approach should an ML engineer use to determine which candidate models can support the application's document-length requirements?

Show answerCORRECT ANSWER

Qualify each candidate using its supported token-counting method on complete, production-sized requests in every supported language, and check its input, output, and shared context limits.

Character count does not reliably predict token count across languages or models. Model-specific counting of the complete request, including instructions and formatting, establishes input usage. Reserving the required response capacity and checking any separate input and output limits determines whether each candidate can support the workload.

AIP-C01

A company exposes a document-analysis API backed by Amazon Bedrock. When its preferred model is unavailable, an application adapter must select an approved fallback model before invoking it. The API contract requires the complete document and instructions to be processed in one inference request and permits an answer of up to 8,000 tokens. Splitting the analysis across invocations, shortening the input, and reducing the output allowance are not permitted.

For a particular request, model-specific token counting confirms that the complete serialized input consumes 58,000 tokens for each fallback model. The adapter must also leave 2,000 tokens of unused context capacity as safety headroom. All fallback models meet the application's quality requirements and support streaming. Their documented limits are:

Fallback Shared input-and-output context limit Separate maximum input Maximum output
Analysis-A 64,000 tokens 64,000 tokens 8,000 tokens
Analysis-B 128,000 tokens 128,000 tokens 4,000 tokens
Analysis-C 96,000 tokens 96,000 tokens 8,000 tokens
Analysis-D 128,000 tokens 48,000 tokens 8,000 tokens

Which fallback selection and invocation configuration meets the API contract?

Show answerCORRECT ANSWER

Select Analysis-C and request 8,000 output tokens after verifying that the input, output allowance, and safety headroom fit its shared context limit.

The required capacity is 58,000 + 8,000 + 2,000 = 68,000 tokens, which fits Analysis-C's 96,000-token shared limit. The 58,000-token input and 8,000-token output allowance also satisfy its separate limits.

Practice Context Window in a full exam

Questions on Context Window in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Context Window

3 min read

A foundation model's context window defines the information available to it during an inference request. For text models, this capacity is typically expressed in tokens: units that may represent words, parts of words, punctuation, or other text fragments. The context includes more than the user's latest question. System instructions, previous conversation messages, examples, retrieved documents, tool definitions, and tool results can all consume capacity. Multimodal models also account for images, audio, or video according to model-specific processing and limits.

For many generative models, input tokens and generated output must fit within a shared context limit. Providers may also impose separate maximum input and output lengths, so a large context window does not imply an equally large possible response. Applications should check the selected model version and inference API rather than assume every model uses identical accounting. A request can fit the input allowance but still leave insufficient room for the answer the application needs.

Consider a hypothetical model with a 32,000-token shared context window. If instructions and conversation history consume 6,000 tokens and the application reserves 4,000 tokens for the response, no more than approximately 22,000 tokens remain for retrieved evidence and other request material. An engineering implementation should allow additional headroom for message formatting, tool interactions, and estimation differences. Depending on the API or application framework, oversized requests may be rejected or truncated; applications should implement an explicit policy rather than rely on implicit behavior.

In cloud applications, context management usually happens in the application layer surrounding a managed inference endpoint. A support assistant might retain recent conversation turns, summarize older exchanges, and retrieve only the passages relevant to the current question. Retrieval-augmented generation, or RAG, lets the application search a corpus much larger than the context window and insert selected evidence into each request. Chunk size, retrieval count, and reranking influence how effectively the available capacity is used. Summaries conserve space but may omit facts, so important details may need to remain available through retrieval or structured state.

Context length is one foundation-model selection criterion, not a substitute for evaluating the entire workload. A larger window can support long-document analysis or complex conversations, but processing more input generally increases resource use and can raise latency and cost. Model size and context capacity are different properties: a model with more parameters does not necessarily support a longer context. Compare task quality, modality support, multilingual performance, customization options, price, and response time alongside input and output limits. Tokenization varies across languages, so equal-length documents in different languages may require different token budgets.

A larger window also does not guarantee that the model will reliably use every detail. Relevant information can be overlooked, particularly when mixed with substantial unrelated text. Test representative long inputs and measure whether answers use the required evidence, rather than treating the advertised maximum as a quality guarantee. Fine-tuning can adapt behavior but does not inherently expand the supported context window. Likewise, a conversation stored in a database is not automatically visible to the model; the application must supply the relevant information during inference.

Prompt caching can reduce repeated input-processing cost and latency for supported models when requests reuse eligible prompt prefixes. It does not enlarge the context window, and cached material still occupies logical context capacity. Cache eligibility, minimum prefix sizes, expiration, and pricing vary by service and model. For practical model selection, choose a window that accommodates realistic prompts and required answers, then validate token usage, truncation behavior, answer quality, time to first token, and total response time under expected traffic.

Learn Context Window in lessons

6 short lessons on Context Window in 4 certification courses.

Open the lessons