Cloud concept · Updated

LLM-as-a-Judge - LLM-as-a-Judge uses a large language model to assess generated responses against explicit evaluation criteria.

In short

An evaluator LLM receives a task, a candidate response, and a scoring rubric, optionally with reference answers or supporting evidence, and returns a score, label, or preference. This approach helps engineers evaluate qualities such as relevance, groundedness, and instruction following at scale, but its judgments require calibration and human oversight.

Cloud Concepts Hub →

6 key aspects of LLM-as-a-Judge

Reliable LLM judging combines explicit rubrics, appropriate evidence, controlled comparisons, human calibration, and traceable evaluation runs.

01
Rubric-driven assessment

Define concrete criteria such as relevance, correctness, groundedness, and instruction following, with clear score anchors. Separate dimensions when a combined score could conceal an important failure.

02
Reference and evidence flexibility

Evaluation can use reference answers, supporting documents, or the task and response alone. Supply authoritative evidence when assessing whether claims are supported rather than relying only on the judge's internal knowledge.

03
Pointwise and pairwise judgments

Score individual responses against a rubric or compare two responses for a preference. Randomize pairwise answer order and allow ties when the task supports them.

04
Human calibration and bias controls

Compare judge decisions with human-reviewed examples before using scores as release criteria. Check for position, verbosity, and style biases, and review important disagreements.

05
Versioned, secure evaluation pipelines

Record dataset, rubric, model versions, and inference settings alongside results. Isolate untrusted answer text from evaluator instructions and apply access controls to sensitive prompts and outputs.

06
Complementary evaluation signals

Use LLM judgments alongside ROUGE, BLEU, BERTScore, deterministic validators, and human assessment where appropriate. A strong judge score does not guarantee factual accuracy, safety, or production readiness.

Cloud services for LLM-as-a-Judge

Managed evaluation capabilities in Amazon Bedrock and Vertex AI reduce the orchestration needed to score generated responses with evaluator models. Amazon SageMaker AI supports custom workflows when teams need control over the evaluator model, runtime, or scoring implementation. These services are alternative implementation paths and do not need to be deployed together. Across implementations, preserve the evaluation dataset, rubric, model configuration, and per-example results, then compare automated judgments with human-reviewed cases before using aggregate scores to approve a release.

Amazon Bedrock

Amazon Bedrock supports model evaluation using an LLM as a judge to assess generated responses against evaluation criteria. Engineers can also invoke supported foundation models through its APIs to build custom judging workflows.

Amazon SageMakerAmazon SageMaker

Amazon SageMaker AI can host an evaluator model and run processing jobs for a custom LLM-as-a-Judge pipeline. Engineers supply the judging prompts, scoring logic, calibration dataset, and result aggregation rather than assuming every hosted model is automatically a reliable evaluator.

LLM-as-a-Judge exam questions

Real practice exam questions on LLM-as-a-Judge, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIF-C01

An education company is comparing two foundation models that generate explanations for beginner students. The company uses another model to select the better explanation based on a rubric for age-appropriate language and teaching quality. During validation, the evaluator frequently changes its selection when the same two explanations are presented in the opposite order. Which change would best improve the reliability of this evaluation?

Show answerCORRECT ANSWER

Hide candidate model identities, evaluate each pair in both presentation orders using the same rubric, and check agreement with teacher judgments on a sample.

The changing selections indicate position bias. Evaluating both orders helps expose and reduce its effect, hiding identities limits model preference bias, and checking against teacher judgments calibrates the evaluator to the intended qualities.

MLA-C02 (beta)

A company uses Amazon Bedrock foundation models to summarize engineering incident reports. A nightly evaluation compares summaries from the production model and a candidate model. A separate foundation model receives the original report, both summaries, and a rubric for factual accuracy and completeness, then selects the better summary. During validation, engineers submit the same pairs twice with the presentation order reversed. The evaluator frequently prefers whichever summary appears first. The company needs a scalable comparison that reduces this bias while continuing to assess factual accuracy and completeness. Which change should the ML engineer make?

Show answerCORRECT ANSWER

Remove model-identifying labels, evaluate each pair in both presentation orders with the same rubric, and aggregate preferences by model identity while recording disagreements.

Blinding model identities and counterbalancing presentation order reduce identity and position effects in model-based evaluation. Mapping judgments back to the actual models before aggregation prevents first-position preferences from being mistaken for quality improvements. Order-dependent disagreements also identify cases that need further review.

AIP-C01

A company uses Amazon Bedrock for a customer-support assistant that answers questions in five languages. An evaluator foundation model scores each answer for policy correctness using the applicable policy excerpts. The evaluator has 97% agreement with specialists on a production-weighted audit sample. One language accounts for only 2% of requests.

Support specialists report that the evaluator frequently approves incorrect answers in that low-volume language. The team must determine whether automated evaluation is reliable for each supported language, independently of differences in the assistant's underlying answer quality. Qualified specialists are available for all five languages, but the additional review budget is limited.

Which evaluation change BEST meets these requirements?

Show answerCORRECT ANSWER

Allocate additional specialist reviews by language, oversampling low-volume languages and obtaining enough specialist-confirmed acceptable and unacceptable answers in each. Compare evaluator decisions with specialist labels to report false-acceptance and false-rejection rates, sample counts, and uncertainty separately for each language. Apply production weights only when also reporting population-wide estimates.

This measures evaluator errors against an independent reference within each language. False-acceptance rates condition on specialist-confirmed unacceptable answers, separating the evaluator's ability to detect errors from the assistant's underlying error prevalence. Targeted sampling improves evidence for low-volume languages, while counts and uncertainty prevent small samples from appearing conclusive. Production weighting remains appropriate for separate overall estimates.

Practice LLM-as-a-Judge in a full exam

Questions on LLM-as-a-Judge in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is LLM-as-a-Judge

2 min read

LLM-as-a-Judge is an evaluation method in which a large language model assesses another model's output, or outputs produced by the same model. The judge receives an evaluation prompt containing the original task, the candidate answer, and explicit criteria. Depending on the task, it may also receive a reference answer, retrieved documents, or other evidence. Its output can be a numerical score, a pass/fail decision, or a preference between two responses. Unlike a single standardized metric, LLM-as-a-Judge is a configurable evaluation approach whose meaning depends on the rubric and evaluation setup.

For foundation model evaluation, distinguish this approach from ROUGE, BLEU, and BERTScore. ROUGE measures overlap with reference text and commonly emphasizes recall, making it useful for evaluating summaries. BLEU measures modified n-gram precision with a brevity penalty and is traditionally associated with machine translation. BERTScore compares contextual token embeddings to estimate semantic similarity. These metrics provide useful signals, but they do not independently establish factual correctness or instruction compliance. An LLM judge can apply a task-specific rubric to these broader qualities, although it can also make mistakes.

A practical cloud workflow starts with a representative evaluation dataset, including ordinary requests, difficult cases, and known failure scenarios. Engineers generate answers using candidate models or prompt versions, then submit those answers to a managed evaluation service or an evaluator model endpoint. Store the dataset version, candidate model, judge model, rubric, inference settings, and individual judgments so results remain traceable. Run evaluations before deployment and on appropriately protected production samples to detect regressions.

For example, a retrieval-augmented support assistant might answer questions about a company's refund policy. Give the judge the customer question, the assistant's answer, and the retrieved policy passages. Ask it to score answer relevance and whether claims are supported by those passages as separate dimensions. A fluent answer can be relevant yet unsupported, so combining everything into one score can hide a serious defect. A reference-free evaluation does not require an ideal answer, but groundedness assessment still requires access to the evidence being checked.

Judging can use pointwise scoring, where each response is assessed independently, or pairwise comparison, where the judge chooses between two responses. Define score anchors and examples so that a rating has a consistent operational meaning. For pairwise comparisons, randomize or reverse answer order to measure position bias. Calibrate the judge against human-reviewed examples and inspect disagreement, especially for safety-sensitive or specialized tasks.

The main trade-offs are evaluation cost, latency, variability, and bias. Judges may favor longer answers, polished writing, or outputs resembling their own style. Low-temperature settings can reduce variation but do not guarantee reproducibility. Treat candidate text as untrusted data because embedded instructions can attempt to manipulate the evaluator. Protect sensitive evaluation data with appropriate access controls and retention policies. Finally, a judge's explanation is not proof that its score is correct: combine model judgments with deterministic checks, relevant similarity metrics, and targeted human review rather than treating automated scoring as an authoritative verdict.

Learn LLM-as-a-Judge in lessons

4 short lessons on LLM-as-a-Judge in 3 certification courses.

Open the lessons