Precision@k

Precision@k is an information retrieval metric calculated as the number of relevant results among the first k results divided by k. If a search returns five chunks and human reviewers judge three relevant to the query, Precision@5 is 3/5, or 0.60. The metric normally uses binary relevance judgments: each retrieved item either meets the defined relevance criteria or does not. A vector similarity score is not itself a relevance judgment or a Precision@k value.

In a cloud-hosted RAG application, a retriever searches an index and supplies selected passages to a foundation model. Precision@k evaluates the retrieval stage independently of the generated answer. Define the evaluation unit explicitly, such as a document, passage, or chunk, because chunk-level and document-level precision can differ significantly. Also specify whether evaluation occurs before reranking, after reranking, or after the final context selection step.

Build a representative test set containing queries and relevance judgments for candidate results. For an internal support assistant, a question about rotating database credentials should retrieve instructions applicable to the correct database engine, authentication method, and environment. A passage containing similar terminology but instructions for an incompatible configuration should not automatically count as relevant. Context matching therefore requires checking applicability, not just keyword overlap.

Run the same queries against alternative embedding models, chunking strategies, metadata filters, keyword-vector hybrid search configurations, and rerankers. Calculate Precision@k per query and commonly report the arithmetic mean across queries. Keep k, the evaluation dataset, and the labeling criteria fixed when comparing configurations. Define how to handle fewer than k returned results; under a fixed-k convention, unfilled positions receive no relevance credit. Record the number of results returned so filtering or retrieval failures remain visible.

Precision@k has important limitations. It does not measure how much relevant information was missed, which is the concern of recall. It also ignores ordering within the first k positions: moving a relevant result from fifth to first does not change Precision@5. Use metrics such as normalized discounted cumulative gain when graded relevance and ordering matter, and reciprocal rank when the position of the first relevant result matters. High precision can still leave a multi-part question without all the evidence needed to answer it.

Increasing k may retrieve additional useful evidence, but it can also introduce irrelevant context and increase prompt token usage. Precision does not necessarily decrease whenever k increases; its direction depends on the relevance of the additional results. Measure retrieval and reranking latency separately, including tail latency such as p95, rather than treating precision as a performance metric. Verify that the final context fits the model's input budget and retains the evidence needed for the task.

For production monitoring, log query identifiers, retrieved item identifiers, ranks, index versions, retrieval settings, and latency under appropriate privacy controls. Periodically label sampled traffic and calculate precision by query category, language, or tenant to detect regressions hidden by an overall average. Automated relevance judges can increase coverage, but their judgments require calibration against human-reviewed examples. Missing labels do not prove irrelevance, and high retrieval precision does not guarantee a grounded or correct generated answer; evaluate answer quality separately.