Cloud concept · Updated

Recall@k - Recall@k measures the fraction of all known relevant items that a retrieval system returns within its first k results.

In short

Recall@k evaluates whether a search or retrieval pipeline finds the evidence needed for a query within a limited result budget. In retrieval-augmented generation (RAG), it helps engineers detect missing context, compare retrieval configurations, and balance evidence coverage against latency and context-window cost.

Cloud Concepts Hub →

6 key aspects of Recall@k

Recall@k provides a labeled measure of evidence coverage that supports retrieval tuning, stage-by-stage troubleshooting, and ongoing quality monitoring.

01
Coverage within a result budget

Recall@k measures how much of the known relevant set appears among the first k results. Choose k to reflect a real retrieval or context-selection budget, and compare configurations at the same cutoff.

02
Explicit relevance ground truth

The metric requires query-specific relevance judgments and a defined evaluation unit, such as documents or chunks. Version the labels and corpus together, and define how duplicates, access restrictions, and queries without relevant results are handled.

03
Stage-specific evaluation

Measure coverage at candidate retrieval, after reranking, and after final context selection. This identifies whether evidence was never found or was discarded later in the pipeline.

04
Recall, precision, and latency trade-offs

A larger result set can recover more evidence while introducing irrelevant context and additional processing cost. Select retrieval depth using recall, precision, latency percentiles, and downstream answer quality rather than recall alone.

05
Retrieval failure diagnosis

Use missed relevant items to investigate ingestion gaps, poor chunking, incompatible embeddings, restrictive filters, and approximate search losses. Separate relevance-based recall from approximate nearest-neighbor recall against exact search results.

06
Regression and drift monitoring

Run a versioned query suite after retrieval changes and periodically during production operation. Segment results to expose failures hidden by averages, and refresh judgments as the corpus and user questions evolve.

Cloud services for Recall@k

A retrieval application can use Amazon Bedrock Knowledge Bases backed by a supported Amazon OpenSearch Service configuration to retrieve evidence for generation. A custom evaluation workload on Amazon SageMaker AI can submit a versioned set of queries, compare returned identifiers with relevance labels, and calculate Recall@k at selected cutoffs. Publish aggregate and segmented results to Amazon CloudWatch alongside latency and operational metrics. This separates evidence-coverage testing from infrastructure health checks and helps determine whether a regression comes from retrieval configuration, data changes, or a later context-selection step.

Amazon Bedrock

Amazon Bedrock Knowledge Bases provides retrieval APIs that return source chunks for RAG applications. Use those results in a labeled evaluation harness to calculate Recall@k and compare retrieval depth, filtering, and reranking configurations rather than inferring recall from similarity scores.

Amazon OpenSearch ServiceAmazon OpenSearch Service

Amazon OpenSearch Service supports vector and lexical retrieval for building search and RAG pipelines. Evaluate its returned document or chunk identifiers against relevance labels while tuning hybrid retrieval, metadata filters, and approximate nearest-neighbor search parameters.

Amazon SageMakerAmazon SageMaker

Amazon SageMaker AI processing jobs can run custom retrieval evaluation workloads over versioned query datasets. Use repeatable jobs to compare embedding models, preprocessing strategies, and retrieval configurations while recording Recall@k alongside other metrics.

Amazon CloudWatchAmazon CloudWatch

Amazon CloudWatch can receive custom Recall@k metrics produced by an evaluation pipeline and display them alongside retrieval latency and error metrics. Configure alarms for quality regressions, recognizing that recall measurements require relevance judgments and are not automatically available for every live query.

Recall@k exam questions

Real practice exam questions on Recall@k, with the correct answer and its explanation. Answer them with every option in the exam simulator.

MLA-C02 (beta)

A company uses Knowledge Bases for Amazon Bedrock for an internal support assistant. For each request, the application calls Retrieve, applies an application-defined relevance gate to the returned chunks, and supplies up to five accepted chunks to a foundation model. If no chunks pass the gate, the application responds that supporting information is unavailable without invoking the model.

The company maintains two expert-labeled canary sets: answerable queries with complete sets of relevant chunk IDs, and unsupported queries confirmed to have no relevant chunks in the indexed corpus. Engineers periodically adjust the gate. They need nightly monitoring that detects both lost evidence for answerable queries and irrelevant evidence passing the gate for unsupported queries, independently of generated answers.

Which monitoring solution best meets these requirements?

Show answerCORRECT ANSWER

For answerable queries, calculate Recall@5 from the chunks accepted by the gate. For unsupported queries, calculate the fraction with at least one accepted chunk. Publish the two results as separate custom Amazon CloudWatch metrics and alarm on low recall or a high unsupported-query acceptance rate.

Recall@5 measures how much labeled evidence survives into the model's context for answerable queries. Unsupported queries have no relevant chunks, so recall is undefined for that set. The fraction of unsupported queries with nonempty accepted context directly measures the gate's false acceptance behavior. Separate metrics detect both failure modes without letting one conceal the other.

AIP-C01

A company operates a technical-support RAG assistant that uses Amazon Bedrock embedding and generation models. During the past three months, customers have increasingly used new product abbreviations and reused older product terms with different meanings. Answer relevance has declined, although the application's original retrieval benchmark remains unchanged at 92% Recall@10.

Engineers confirm that model versions and retrieval settings have not changed. Current documentation is fully indexed, and relevant procedures are preserved in complete chunks. On a manually labeled sample of recent queries, Recall@10 is 58%. Exact nearest-neighbor search performs no better than approximate search. Supplying the correct passages directly to the generation model restores answer quality.

Which approach should the developer take to address the decline and detect similar problems in the future?

Show answerCORRECT ANSWER

Create a versioned evaluation set from labeled recent queries alongside the historical set. Evaluate terminology-aware query rewriting or a better-suited embedding model on both sets, re-embedding documents if the embedding model changes. Monitor retrieval quality by product and time period after deployment.

The operating environment has changed even though the models have not: current terminology and intended meanings are no longer adequately represented by the historical evaluation set. Recent relevance labels expose the resulting retrieval degradation. Evaluating targeted retrieval adaptations against both recent and historical queries addresses the change while checking for regressions. Ongoing segmented monitoring helps detect subsequent drift.

Practice Recall@k in a full exam

Questions on Recall@k in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Recall@k

3 min read

Recall@k is a retrieval quality metric calculated as the number of relevant items in the first k results divided by the total number of relevant items for that query. If a labeled evaluation set identifies five relevant passages and the first ten retrieved results contain three of them, Recall@10 is 3/5, or 0.60. The denominator is the relevant set, not the number of returned results; dividing by the number of returned results measures precision instead.

In a cloud-hosted RAG application, retrieval supplies evidence to a foundation model before it generates an answer. Low recall means useful evidence is missing from the retrieved context, even if the model produces a fluent response. For example, a support assistant answering a question about account recovery might retrieve the general procedure but miss a separate policy covering locked administrator accounts. Increasing model size or changing the answer prompt does not reliably correct that retrieval failure.

Build an evaluation dataset containing representative queries and relevance judgments for documents or passages. Define the evaluation unit explicitly: document-level recall and chunk-level recall answer different questions. Use stable identifiers, version the corpus and judgments, and avoid giving duplicate or overlapping chunks credit for repeatedly retrieving the same evidence. Relevant items should also reflect the query's intended authorization and filtering scope. Queries with no relevant items have an undefined recall denominator; handle them separately when evaluating correct abstention or empty-result behavior.

Run the same queries against candidate configurations and calculate Recall@k for each query. A common aggregate is the mean across queries, which gives each query equal weight. Also examine results by language, topic, tenant, document age, and query difficulty, because an overall average can conceal severe failures in smaller groups. When relevance judgments are incomplete, the score measures recovery of known relevant items rather than guaranteed coverage of every useful item in the corpus.

For a fixed ranked result sequence, increasing k cannot decrease recall, but it can reduce precision and increase retrieval, reranking, and generation costs. More context can also distract the generator or exceed its input budget. Measure recall both before reranking and on the final context passed to the model: a large candidate pool may have good coverage while a later reranker or token-budget truncation removes essential evidence. Recall@k does not distinguish where relevant items appear within the first k positions, so complement it with ranking metrics such as nDCG or mean reciprocal rank.

When recall declines, trace failures through ingestion, preprocessing, chunking, embedding generation, indexing, filtering, and search. Check whether documents were indexed successfully, whether chunk boundaries separated questions from their answers, and whether query and document embeddings use compatible models and preprocessing. Confirm vector dimensions and the distance metric expected by the embedding model. Then investigate overly restrictive metadata filters, stale indexes, and approximate nearest-neighbor search settings. Compare approximate results with an exact vector search baseline to isolate search approximation losses; recovering exact nearest neighbors is not the same as retrieving semantically relevant evidence.

Production monitoring should combine periodic labeled retrieval tests with sampled relevance reviews, latency percentiles, empty-result rates, and index freshness indicators. Track changes in query distribution and embedding behavior to investigate drift. Similarity scores and model response relevance are useful diagnostics, but neither is a substitute for Recall@k against relevance judgments. Validate that retrieved passages actually support the requested facts, and evaluate answer correctness and faithfulness separately. Promote retrieval changes only after checking evidence coverage, response quality, latency, and cost together.

Learn Recall@k in lessons

4 short lessons on Recall@k in 2 certification courses.

Open the lessons