Confidence Intervals

Confidence intervals describe uncertainty around estimates calculated from a sample. A GenAI evaluation might report that a foundation model answered 90% of test questions correctly, but that percentage alone does not show how precisely the evaluation estimates performance on the target workload. An interval provides that context. If 900 of 1,000 independent, representative test cases pass, a 95% Wilson confidence interval for the underlying pass rate is approximately 88.0% to 91.7%.

The confidence level describes the long-run coverage of the interval-building procedure. If equivalent samples were repeatedly collected and intervals calculated using the same valid method, approximately 95% of the resulting 95% intervals would contain the true parameter. It does not mean that 95% of individual answers are correct, that future observations will fall inside the interval, or, under the usual frequentist interpretation, that the fixed parameter has a 95% probability of lying inside this particular calculated interval.

In cloud evaluation pipelines, retain case-level inputs, outputs, scores, and relevant metadata before calculating aggregate metrics. Record the model version, prompt template, retrieval configuration, inference settings, evaluator version, dataset version, and evaluation timestamp. Compute estimates and interval endpoints in a reproducible processing job, then publish the results to a reporting dataset. Dashboards can display error bars, interval bands, and comparison tables, while automated reports explain what changed and whether the evidence supports a deployment decision.

Choose a statistical method that matches the metric and sampling design. Wilson intervals are useful for binary pass rates, particularly where a simple normal approximation performs poorly. A t-based interval can be suitable for a mean when its assumptions are reasonable. Bootstrap intervals can support complex metrics, such as aggregate quality scores or latency percentiles, but require enough representative observations. When multiple questions belong to the same conversation or customer, resample independent clusters rather than treating correlated records as independent observations. Repeated generations for one prompt do not substitute for broad prompt coverage.

For model comparisons, evaluate candidates on the same test cases and calculate an interval for the paired difference. For example, a candidate might improve task success by 2 percentage points, with a 95% interval for the improvement from -0.5 to 4.5 percentage points. The results remain compatible with no improvement, even though the candidate's point estimate is higher. Overlap between separate models' intervals is not a reliable substitute for analyzing the difference directly. Also distinguish statistical evidence from practical value: a tiny improvement might not justify substantially higher cost or latency.

Interval width reflects sample size, variability, the chosen confidence level, and the analysis method. More independent observations generally narrow intervals; under common assumptions, halving the margin of error requires roughly four times as many observations. Higher confidence produces wider intervals. These trade-offs affect evaluation budgets because model calls, human assessments, and processing all incur costs. Narrow intervals do not correct biased datasets, unreliable judging, data leakage, or production drift.

Stakeholder reports should show the estimate, confidence level, interval, sample size, metric definition, calculation method, and important limitations. Break out critical workload segments without hiding their smaller sample sizes. Define release thresholds in advance; for example, require an appropriate one-sided lower confidence bound on success rate to exceed a business minimum. Account for repeated comparisons and repeated checking of results, which can undermine nominal coverage when handled naively. Distinguish confidence intervals from prediction intervals, Bayesian credible intervals, model-generated confidence scores, and raw standard deviations.