LLM Eval Confidence Intervals That Matter

LLM eval confidence intervals answer a narrower question than an accuracy leaderboard: could the measured gain plausibly be sampling noise? This guide builds a paired bootstrap around the same prompts, exposes cohort uncertainty, and returns ship, reject, or collect-more instead of manufacturing certainty.

LLM eval confidence intervals start with paired evidence

Two aggregate scores hide the most useful unit of evidence: the prompt-level difference between candidate and baseline. If both models answer the same 240 prompts, each prompt forms a pair, so subtract the baseline score from the candidate score before resampling. LLM eval confidence intervals built from those deltas preserve shared prompt difficulty instead of pretending the two score columns are independent samples.

Keep the row contract explicit: prompt ID, cohort, frozen input hash, baseline output, candidate output, judge version, both scores, and their delta. Missing or retried rows must follow a declared rule before analysis begins, because silently dropping candidate failures can turn evaluation uncertainty into a flattering result. A paired model comparison is credible only when every retained row represents the same task under comparable conditions.

Keep failures as outcomes rather than preprocessing inconveniences. A timeout, refusal, parser error, or judge abstention needs a predeclared score or separate endpoint so the candidate cannot improve simply by producing fewer scorable answers.

Archive the excluded-row ledger beside every interval: prompt ID, exclusion reason, retry count, and the decision made before scores were revealed. That receipt lets another reviewer reconstruct the analysis and detect whether missing candidate failures quietly changed the comparison.

Freeze the measurement contract before looking at gains

Write down the target behavior, prompt population, scoring scale, judge, sampling policy, exclusion rules, and release threshold before opening the candidate results. The AI eval measurement contract is the right companion because it makes the estimand legible: perhaps the mean change in a bounded rubric score across current support prompts. Without that sentence, LLM significance testing can answer a mathematically neat question nobody intended to ask.

Separate model randomness from prompt sampling. If each model produces one frozen response per prompt, the bootstrap describes uncertainty over the sampled prompt population, not every response the stochastic model might emit. Multiple seeds can be nested within prompts, but flattening all generations into independent rows exaggerates evidence. State which randomness LLM eval confidence intervals cover, and label all other uncertainty as unmeasured rather than implying a universal guarantee.

Review the contract with product, domain, and operations owners before a run. Their disagreement is valuable evidence that the metric, prompt population, or practical floor still lacks the precision needed for an automated release gate.

Bootstrap the mean delta without losing structure

The nonparametric bootstrap repeatedly samples prompt rows with replacement, calculates the mean paired delta for each sample, and reads percentile bounds from that empirical distribution. The NIST bootstrap handbook explains the resampling idea, while the SciPy bootstrap reference documents paired inputs and interval choices. The fixture below stays dependency-free so its assumptions remain visible.

Choose the statistic deliberately. A mean supports additive score changes, a win rate answers how often the candidate wins, and a severe-error rate may deserve its own upper bound. LLM eval confidence intervals should not average unlike harms into one friendly number. When the scoring function is an LLM judge, first apply LLM judge calibration so interval precision does not disguise a drifting or position-biased measurement instrument.

Archive the raw paired-delta histogram beside the interval. It reveals ties, outliers, ceilings, and discrete scoring artifacts that a smooth bootstrap density can conceal, giving reviewers a direct view of what the resampler repeatedly recombines.

Add a practical-effect floor to the decision

An interval entirely above zero says the observed mean gain is inconsistent with zero under the chosen resampling model; it does not say the gain pays for added latency, cost, or risk. Define a minimum practical effect, such as 0.03 rubric points, before evaluation. Ship only when the lower interval bound exceeds that floor, reject when the upper bound is below it, and collect more evidence when the interval overlaps the decision boundary.

That abstain region is a feature, not a statistical failure. It prevents a tiny but precisely measured gain from being marketed as a meaningful improvement and prevents a promising but noisy candidate from being discarded prematurely. LLM eval confidence intervals become a product gate only after statistical confidence and practical significance occupy separate columns in the release receipt. Include cost and latency guardrails beside them rather than smuggling those tradeoffs into the score.

Give the floor an owner and unit. “Three points” is meaningless until the rubric scale, baseline, affected traffic, cost of error, and acceptable tradeoff have been recorded in a decision that can survive personnel changes.

This deterministic fixture resamples paired prompt deltas, reports a 95% interval, applies a practical-effect floor, and abstains when the evidence crosses a boundary.

Runnable artifact — paired-bootstrap-decision.test.mjs

import assert from "node:assert/strict";
const deltas=[.2,.1,.2,0,.1,.3,-.1,.2,.1,.2,.3,0];let state=27082026;const random=()=>((state=Math.imul(state,1664525)+1013904223>>>0)/2**32);
const means=[];for(let b=0;b<6000;b++){let sum=0;for(let i=0;i<deltas.length;i++)sum+=deltas[Math.floor(random()*deltas.length)];means.push(sum/deltas.length)}means.sort((a,b)=>a-b);
const interval=[means[Math.floor(means.length*.025)],means[Math.floor(means.length*.975)]],floor=.12;
const decision=interval[0]>floor?"ship":interval[1]<floor?"reject":"collect-more";
assert.equal(decision,"collect-more");assert.ok(interval[0]<=floor&&interval[1]>=floor);console.log("PASS: paired bootstrap returns an evidence-bounded decision");

Run node paired-bootstrap-decision.test.mjs. Expected receipt: PASS: paired bootstrap returns an evidence-bounded decision.

Inspect cohorts before trusting the overall interval

A global improvement can conceal a severe regression in a small cohort. Compute the same paired deltas for language, task family, risk tier, prompt length, and any segment named in the measurement contract. Do not demand that every tiny slice achieve significance; report its sample size, interval width, and failure examples so sparse cohorts are visibly uncertain instead of quietly green.

Use Stanford HELM as a model for making scenarios, metrics, and limitations inspectable, not as evidence that one benchmark matches your users. LLM eval confidence intervals belong beside cohort composition and coverage. If the medical slice contains six prompts, its broad interval should trigger targeted collection or a conservative policy, not a confident claim that the candidate is equally safe across domains.

Predeclare the few cohorts tied to product risk and show exploratory slices separately. Testing dozens of ad hoc segments after seeing results almost guarantees a dramatic-looking interval somewhere, while making the original release question harder to see.

Diagnose interval width before buying more prompts

When the band is wide, first ask why. High prompt heterogeneity, judge inconsistency, rare catastrophic failures, and too few rows can all broaden LLM eval confidence intervals, but each calls for a different response. Plot the paired deltas and inspect the largest positive and negative rows before mechanically doubling the evaluation set. A bimodal distribution may mean two task families should never have shared one average.

New examples should come from the target population, not from whichever prompts make the candidate look stable. Preserve a holdout for later canary evals in AI releases and record any adaptive collection after seeing results. Otherwise the team can unknowingly tune both model and measurement set to the same surprises, producing narrow evaluation uncertainty around a decision that will not reproduce in traffic.

Estimate collection value before labeling more data. If the current interval barely overlaps the floor, targeted independent prompts may resolve the choice; if the candidate has a clear harmful cohort, more sampling only delays a product decision already supported by examples.

Report sensitivity instead of one magical interval

Recompute the verdict under defensible alternatives: include all failures as zero, vary the practical floor, use a trimmed statistic only when predeclared, and compare percentile bounds with another justified interval method. Sensitivity is not permission to select the happiest number. It is a map of which assumptions can reverse the decision and therefore deserve ownership, monitoring, or more data.

Keep bootstrap seeds, prompt hashes, model identifiers, judge versions, and the exact fixture with the release receipt. LLM eval confidence intervals are reproducible only when another reviewer can rebuild the paired rows and recover the same decision. Tie the result to test-time compute stop rules when inference budgets vary, because a candidate evaluated with more sampling is a different treatment from the baseline.

Run a code review on the resampling implementation itself. Off-by-one quantiles, accidental row shuffling, filtered nulls, or a non-seeded generator can change the reported boundary while leaving every chart visually plausible. Treat the bootstrap LLM evaluation code as release infrastructure.

Turn uncertainty into a release conversation

Present the point estimate, interval, practical floor, cohort exceptions, cost delta, and abstain reason on one page. The strongest statement is often conditional: under the frozen prompt population and judge version, the candidate's paired mean improvement is likely positive but not yet above the useful-effect floor. That sentence is more actionable than “accuracy rose two points,” because it shows exactly what additional evidence could change the conclusion.

Revisit LLM eval confidence intervals when the prompt mix, scorer, generation policy, or product risk changes. Never carry an old interval onto a materially new evaluation contract. A release culture that can say collect-more protects users and preserves promising experiments; it also makes a later ship decision traceable to evidence rather than confidence theater.

Publish the limits beside the verdict: population sampled, uncertainty represented, uncertainty omitted, and next trigger. That final box prevents a precise interval from traveling into presentations as a timeless guarantee about “the model” rather than evidence about one measured treatment.