LLM-as-a-Judge is an evaluation method in which a large language model assesses another model's output, or outputs produced by the same model. The judge receives an evaluation prompt containing the original task, the candidate answer, and explicit criteria. Depending on the task, it may also receive a reference answer, retrieved documents, or other evidence. Its output can be a numerical score, a pass/fail decision, or a preference between two responses. Unlike a single standardized metric, LLM-as-a-Judge is a configurable evaluation approach whose meaning depends on the rubric and evaluation setup.
For foundation model evaluation, distinguish this approach from ROUGE, BLEU, and BERTScore. ROUGE measures overlap with reference text and commonly emphasizes recall, making it useful for evaluating summaries. BLEU measures modified n-gram precision with a brevity penalty and is traditionally associated with machine translation. BERTScore compares contextual token embeddings to estimate semantic similarity. These metrics provide useful signals, but they do not independently establish factual correctness or instruction compliance. An LLM judge can apply a task-specific rubric to these broader qualities, although it can also make mistakes.
A practical cloud workflow starts with a representative evaluation dataset, including ordinary requests, difficult cases, and known failure scenarios. Engineers generate answers using candidate models or prompt versions, then submit those answers to a managed evaluation service or an evaluator model endpoint. Store the dataset version, candidate model, judge model, rubric, inference settings, and individual judgments so results remain traceable. Run evaluations before deployment and on appropriately protected production samples to detect regressions.
For example, a retrieval-augmented support assistant might answer questions about a company's refund policy. Give the judge the customer question, the assistant's answer, and the retrieved policy passages. Ask it to score answer relevance and whether claims are supported by those passages as separate dimensions. A fluent answer can be relevant yet unsupported, so combining everything into one score can hide a serious defect. A reference-free evaluation does not require an ideal answer, but groundedness assessment still requires access to the evidence being checked.
Judging can use pointwise scoring, where each response is assessed independently, or pairwise comparison, where the judge chooses between two responses. Define score anchors and examples so that a rating has a consistent operational meaning. For pairwise comparisons, randomize or reverse answer order to measure position bias. Calibrate the judge against human-reviewed examples and inspect disagreement, especially for safety-sensitive or specialized tasks.
The main trade-offs are evaluation cost, latency, variability, and bias. Judges may favor longer answers, polished writing, or outputs resembling their own style. Low-temperature settings can reduce variation but do not guarantee reproducibility. Treat candidate text as untrusted data because embedded instructions can attempt to manipulate the evaluator. Protect sensitive evaluation data with appropriate access controls and retention policies. Finally, a judge's explanation is not proof that its score is correct: combine model judgments with deterministic checks, relevant similarity metrics, and targeted human review rather than treating automated scoring as an authoritative verdict.
Amazon SageMaker