Inter-Rater Agreement

Inter-rater agreement measures the extent to which independent human evaluators reach matching judgments about the same material. In machine learning, that material might be an image awaiting classification, a customer message requiring a safety label, or a foundation model response being evaluated for factual accuracy and relevance. Agreement is a property of the evaluation process, including the task, rubric, sampled items, and evaluator population; it is not simply a property of the evaluators themselves.

Start by defining the judgment precisely. For text generation, assess factual correctness, completeness, relevance, and tone separately rather than asking for one ambiguous quality score. Provide examples and anchored rating descriptions, such as what distinguishes a score of two from a score of three. Have multiple evaluators independently review an overlapping set of items. For model comparisons, hide model identities and randomize response order to reduce brand preferences and position bias.

Choose an agreement measure that matches the task. Percentage agreement is easy to explain but does not adjust for agreement expected by chance. Cohen's kappa supports categorical judgments from two raters, while Fleiss' kappa supports multiple raters under its rating-design assumptions. Weighted Cohen's kappa can account for the distance between ordered categories. Krippendorff's alpha supports different measurement levels and missing ratings when configured appropriately. For numerical scores, an intraclass correlation coefficient can be useful, but its formulation must match whether absolute agreement or consistency is required.

For example, two reviewers might agree on whether generated answers contain unsupported claims in 90 of 100 cases. That 90 percent agreement is informative, but if almost every answer receives the same label, chance-adjusted agreement may be much lower. Conversely, rare safety violations can make kappa difficult to interpret even when observed agreement is high. Inspect label prevalence, disagreement tables, sample size, and uncertainty alongside any summary coefficient. There is no universal agreement threshold that makes every evaluation trustworthy.

In a cloud workflow, store prompts, model outputs, rubric versions, and individual ratings with stable item identifiers. Route overlapping samples to a human workforce, then run a processing job to calculate agreement overall and by language, topic, risk category, and evaluation dimension. For example, an aggregate score might hide poor agreement on medical questions while routine customer-support answers receive consistent ratings. Restrict access to sensitive evaluation data and preserve an audit trail of decisions.

Human-in-the-loop workflows can escalate disputed items to an expert, request another independent review, or defer an automated decision. Calculate agreement from the original independent ratings before adjudication. Majority voting and expert adjudication can produce a final label, but neither demonstrates that the original judgments were consistent. Preserve both raw ratings and resolved labels so downstream users can distinguish uncertainty from a final decision.

Additional reviewers increase cost and latency, so a practical design often uses representative overlap across the evaluation set and extra review for high-risk cases. Measure agreement separately on these subsets because a disagreement-enriched sample does not represent the full workload. Use calibration rounds to improve instructions, then measure agreement again on fresh independently rated items. High agreement does not prove correctness: evaluators can share the same misconception or bias. Combine agreement analysis with expert reference cases, model-quality metrics, and periodic rubric reviews.