Recall@k

Recall@k is a retrieval quality metric calculated as the number of relevant items in the first k results divided by the total number of relevant items for that query. If a labeled evaluation set identifies five relevant passages and the first ten retrieved results contain three of them, Recall@10 is 3/5, or 0.60. The denominator is the relevant set, not the number of returned results; dividing by the number of returned results measures precision instead.

In a cloud-hosted RAG application, retrieval supplies evidence to a foundation model before it generates an answer. Low recall means useful evidence is missing from the retrieved context, even if the model produces a fluent response. For example, a support assistant answering a question about account recovery might retrieve the general procedure but miss a separate policy covering locked administrator accounts. Increasing model size or changing the answer prompt does not reliably correct that retrieval failure.

Build an evaluation dataset containing representative queries and relevance judgments for documents or passages. Define the evaluation unit explicitly: document-level recall and chunk-level recall answer different questions. Use stable identifiers, version the corpus and judgments, and avoid giving duplicate or overlapping chunks credit for repeatedly retrieving the same evidence. Relevant items should also reflect the query's intended authorization and filtering scope. Queries with no relevant items have an undefined recall denominator; handle them separately when evaluating correct abstention or empty-result behavior.

Run the same queries against candidate configurations and calculate Recall@k for each query. A common aggregate is the mean across queries, which gives each query equal weight. Also examine results by language, topic, tenant, document age, and query difficulty, because an overall average can conceal severe failures in smaller groups. When relevance judgments are incomplete, the score measures recovery of known relevant items rather than guaranteed coverage of every useful item in the corpus.

For a fixed ranked result sequence, increasing k cannot decrease recall, but it can reduce precision and increase retrieval, reranking, and generation costs. More context can also distract the generator or exceed its input budget. Measure recall both before reranking and on the final context passed to the model: a large candidate pool may have good coverage while a later reranker or token-budget truncation removes essential evidence. Recall@k does not distinguish where relevant items appear within the first k positions, so complement it with ranking metrics such as nDCG or mean reciprocal rank.

When recall declines, trace failures through ingestion, preprocessing, chunking, embedding generation, indexing, filtering, and search. Check whether documents were indexed successfully, whether chunk boundaries separated questions from their answers, and whether query and document embeddings use compatible models and preprocessing. Confirm vector dimensions and the distance metric expected by the embedding model. Then investigate overly restrictive metadata filters, stale indexes, and approximate nearest-neighbor search settings. Compare approximate results with an exact vector search baseline to isolate search approximation losses; recovering exact nearest neighbors is not the same as retrieving semantically relevant evidence.

Production monitoring should combine periodic labeled retrieval tests with sampled relevance reviews, latency percentiles, empty-result rates, and index freshness indicators. Track changes in query distribution and embedding behavior to investigate drift. Similarity scores and model response relevance are useful diagnostics, but neither is a substitute for Recall@k against relevance judgments. Validate that retrieved passages actually support the requested facts, and evaluate answer correctness and faithfulness separately. Promote retrieval changes only after checking evidence coverage, response quality, latency, and cost together.