LLM Eval Multiple Testing Without False Wins

LLM eval multiple testing separates a real release signal from the winner produced by enough comparisons. Freeze the hypothesis family, then keep adjusted decisions beside effects and intervals.

LLM eval multiple testing defines the family first

LLM eval multiple testing addresses false wins that appear when one release compares many models, prompts, slices, graders, or metrics. At a nominal five-percent threshold, inspecting enough null comparisons will eventually produce attractive small p-values even when no candidate has a real advantage.

Write the hypothesis family before opening results: candidate releases, datasets, prompt variants, languages, user segments, grader configurations, metrics, and directions. The family is the set of claims the decision process could promote, not merely the subset that looked interesting after analysis.

This guide uses eight committed synthetic arrays of ten paired baseline and candidate scores. The artifact derives effects, t-based intervals, exact sign-flip p-values, Holm and BH adjustments, decision flags, null-family incidence, CSV, and JSON from those arrays. It does not establish model superiority, prescribe one correction for every dependence structure, or replace review by someone who understands the sampling and decision context.

LLM eval multiple testing begins by writing the hypothesis family before results are visible. If a team tests every model, slice, prompt variant, and judge then reports only the smallest value, it has changed a planned evaluation into an unbounded search whose nominal threshold no longer means what readers assume.

Keep paired evidence and effect sizes visible

Begin with the design that generated each raw result. When the same examples are scored by two candidates, paired differences often preserve useful covariance; if data are clustered by user, conversation, or document, the estimator and interval should respect that structure before a correction is applied.

LLM eval multiple testing changes how a family of p-values is interpreted, not the observed effect. Retain the mean or robust effect estimate, confidence interval, sample size, missingness, grader version, and practical threshold beside raw and adjusted values so statistical rejection cannot stand in for product importance.

Use LLM eval confidence intervals to communicate magnitude and uncertainty. A result can survive multiplicity while remaining too small to matter, or miss a threshold while its interval still motivates a larger targeted study.

A paired fixture keeps item difficulty from dominating the comparison. Each candidate and baseline score the same generated examples, the test operates on within-item differences, and the exported table retains the pairing key so another reader can reproduce every statistic rather than trusting a summary row.

Plan stopping and reruns as part of the family. Peeking after each batch, adding examples until a threshold passes, or rerunning only unfavorable seeds expands the selection process even when the final table contains one p-value per row. Sequential methods can be appropriate, but they need their own error-spending contract. Otherwise freeze sample size, allow only documented data-quality reruns, and retain every attempted analysis in the release ledger.

Use Holm for strong family-wise control

Holm’s sequentially rejective procedure sorts raw p-values from smallest to largest and compares them with progressively less stringent family-wise thresholds. The 1979 primary paper establishes strong control while improving on a single-step Bonferroni rule.

For adjusted p-values, multiply each ordered value by the number of remaining hypotheses, cap at one, and take a cumulative maximum so the ordered result is monotone. Map values back to their original hypothesis IDs, and pin a known vector in tests because rank and off-by-one errors can quietly reverse release decisions.

LLM eval multiple testing often favors family-wise control when even one false promotion is expensive: a safety regression claim, a contractual capability statement, or a release gate with many searched slices. The cost is lower power, which should be acknowledged rather than hidden by redefining the family after results arrive.

Multiple comparisons in LLM evals require a family definition tied to one release decision. Separate exploratory diagnostics from confirmatory gates, name which hypotheses can approve the release, and freeze exclusions before calculation; otherwise moving a result between families becomes a hidden way to manufacture significance.

Use BH for a declared discovery program

Benjamini and Hochberg introduced false-discovery-rate control for settings where a collection of discoveries is expected and some bounded proportion of false discoveries is tolerable. The 1995 primary paper states the step-up procedure and its assumptions.

Compute adjusted values from the largest ordered p-value backward, multiply by family size divided by rank, cap at one, and take a cumulative minimum. Record the dependence assumptions behind the chosen variant; a convenient BH button does not make every correlated eval design valid.

Use this path for a real discovery workflow, such as prioritizing many candidate slices for follow-up, not as a way to rescue a preferred release. A BH rejection remains a statistical flag whose practical effect and replication plan must travel with it.

Holm correction controls the probability of at least one false rejection in the family and provides an interpretable sequential decision. Sort raw values, compare each with its shrinking threshold, preserve monotonic adjusted values, and retain the original hypothesis order in the final receipt.

Do not choose the correction after seeing winners

The error criterion is part of the evaluation contract. Decide whether the product needs family-wise protection, false-discovery control, hierarchical testing, a preregistered primary endpoint, or a confirmatory holdout before results are visible; choosing the least restrictive method afterward is another form of multiple searching.

LLM eval multiple testing also fails when analysts run several raw tests, alternative graders, filters, seeds, and aggregation rules but count only the final table as the family. Preserve an analysis ledger or immutable plan that names allowed transformations and identifies exploratory branches as exploratory.

The R p.adjust reference is useful for cross-checking named methods and their distinctions. Production code should pin library version, method name, hypothesis count, ordering, and expected vectors rather than rely on an unlabeled spreadsheet formula.

Benjamini-Hochberg answers a different operating question by controlling the false discovery rate under its assumptions. Its larger rejection set is not a better Holm result; publish both only when the article clearly states which error criterion maps to the actual release policy.

Runnable artifact — The data are synthetic and the tutorial does not prescribe one correction for every eval design, prove model superiority, or replace statistical review.

import assert from "node:assert/strict";
import { createHash } from "node:crypto";

const family = Object.freeze([
  Object.freeze({ id: "h1", baseline: Object.freeze([.61,.58,.64,.60,.63,.57,.62,.59,.65,.60]), candidate: Object.freeze([.68,.63,.69,.65,.69,.62,.67,.65,.70,.66]) }),
  Object.freeze({ id: "h2", baseline: Object.freeze([.54,.56,.52,.55,.53,.57,.51,.58,.55,.54]), candidate: Object.freeze([.58,.58,.55,.57,.56,.59,.54,.60,.57,.57]) }),
  Object.freeze({ id: "h3", baseline: Object.freeze([.72,.70,.71,.69,.73,.68,.74,.70,.72,.71]), candidate: Object.freeze([.75,.72,.73,.72,.75,.70,.77,.72,.74,.73]) }),
  Object.freeze({ id: "h4", baseline: Object.freeze([.44,.47,.45,.46,.43,.48,.42,.47,.45,.46]), candidate: Object.freeze([.45,.46,.47,.47,.44,.49,.43,.48,.46,.47]) }),
  Object.freeze({ id: "h5", baseline: Object.freeze([.82,.80,.81,.79,.83,.78,.84,.80,.82,.81]), candidate: Object.freeze([.83,.82,.80,.81,.84,.79,.85,.81,.83,.82]) }),
  Object.freeze({ id: "h6", baseline: Object.freeze([.33,.35,.34,.32,.36,.31,.37,.34,.33,.35]), candidate: Object.freeze([.32,.36,.35,.31,.36,.32,.36,.35,.34,.34]) }),
  Object.freeze({ id: "h7", baseline: Object.freeze([.66,.64,.67,.65,.68,.63,.69,.64,.66,.65]), candidate: Object.freeze([.67,.64,.68,.66,.68,.64,.70,.65,.65,.66]) }),
  Object.freeze({ id: "h8", baseline: Object.freeze([.49,.51,.50,.48,.52,.47,.53,.50,.49,.51]), candidate: Object.freeze([.49,.52,.49,.49,.51,.48,.52,.51,.50,.50]) }),
]);
const mean = (values) => values.reduce((sum, value) => sum + value, 0) / values.length;
const std = (values) => { const m = mean(values); return Math.sqrt(values.reduce((sum, value) => sum + (value - m) ** 2, 0) / (values.length - 1)); };
const exactSignFlipP = (differences) => {
  const observed = Math.abs(mean(differences));
  let extreme = 0, total = 2 ** differences.length;
  for (let mask = 0; mask < total; mask++) {
    const candidate = differences.map((value, index) => (mask & (1 << index)) ? value : -value);
    if (Math.abs(mean(candidate)) + 1e-15 >= observed) extreme++;
  }
  return extreme / total;
};
const adjustHolm = (values) => { const order = values.map((p, index) => ({ p, index })).sort((a,b) => a.p-b.p), out = Array(values.length); let previous = 0; order.forEach((entry, rank) => { previous = Math.max(previous, Math.min(1, (values.length-rank)*entry.p)); out[entry.index] = previous; }); return out; };
const adjustBH = (values) => { const order = values.map((p,index) => ({p,index})).sort((a,b) => a.p-b.p), out = Array(values.length); let next = 1; for (let rank=order.length-1; rank>=0; rank--) { next = Math.min(next, order[rank].p*order.length/(rank+1), 1); out[order[rank].index] = next; } return out; };
const rows = family.map((hypothesis) => {
  const differences = hypothesis.candidate.map((value,index) => value-hypothesis.baseline[index]);
  const effect = mean(differences), standardError = std(differences)/Math.sqrt(differences.length), margin = 2.2621571627409915*standardError;
  return { id: hypothesis.id, n: differences.length, effect, lo: effect-margin, hi: effect+margin, p: exactSignFlipP(differences), pairedScoresSha256: createHash("sha256").update(JSON.stringify([hypothesis.baseline,hypothesis.candidate])).digest("hex") };
});
const holm = adjustHolm(rows.map((row) => row.p)), bh = adjustBH(rows.map((row) => row.p));
const table = rows.map((row,index) => ({ ...row, holm: holm[index], bh: bh[index], holmReject: holm[index] <= .05, bhReject: bh[index] <= .05 }));
let state = 20260902;
const random = () => { state ^= state << 13; state ^= state >>> 17; state ^= state << 5; return (state >>> 0) / 4294967296; };
let rawFamilies = 0, holmFamilies = 0, bhFamilies = 0;
const simulationRuns = 500, hypothesesPerFamily = 12;
for (let run=0; run<simulationRuns; run++) {
  const pvalues = Array.from({ length: hypothesesPerFamily }, random), adjustedHolm = adjustHolm(pvalues), adjustedBH = adjustBH(pvalues);
  if (pvalues.some((p) => p <= .05)) rawFamilies++;
  if (adjustedHolm.some((p) => p <= .05)) holmFamilies++;
  if (adjustedBH.some((p) => p <= .05)) bhFamilies++;
}
const familyIds = family.map((entry) => entry.id), familySha256 = createHash("sha256").update(JSON.stringify(familyIds)).digest("hex");
const csv = ["id,n,effect,lo,hi,p,holm,bh,holm_reject,bh_reject", ...table.map((row) => [row.id,row.n,row.effect,row.lo,row.hi,row.p,row.holm,row.bh,row.holmReject,row.bhReject].join(","))].join("\n");
const jsonPayload = { fixture: "committed paired scores with exact sign-flip p-values", familyFrozen: Object.isFrozen(family) && family.every(Object.isFrozen), familyIds, familySha256, alpha: .05, table, familywiseNullSimulation: { seed: 20260902, runs: simulationRuns, hypothesesPerFamily, rawFamiliesWithAnyFalseRejection: rawFamilies, holmFamiliesWithAnyFalseRejection: holmFamilies, bhFamiliesWithAnyFalseRejection: bhFamilies }, csv, csvSha256: createHash("sha256").update(csv).digest("hex") };
const json = JSON.stringify(jsonPayload, null, 2);
assert.ok(jsonPayload.familyFrozen);
assert.equal(table.length, family.length);
assert.ok(table.every((row) => row.lo <= row.effect && row.effect <= row.hi));
assert.ok(rawFamilies >= holmFamilies);
console.log(JSON.stringify({ ...jsonPayload, jsonSha256: createHash("sha256").update(json).digest("hex") }, null, 2));
console.log("PASS: paired scores derive effects, intervals, p-values, familywise incidence, CSV, and JSON");

Build a ranked decision table

Give every hypothesis an immutable identifier and columns for family, endpoint, comparison, sample count, effect, interval, raw p-value, method, adjusted p-value, statistical disposition, practical disposition, and release action. Sort a view by raw p-value for calculation but retain stable IDs so ranks cannot detach results from their definitions.

The runnable lab starts from eight committed arrays of ten paired baseline and candidate scores. It derives mean effects, t-based intervals, exact sign-flip p-values, Holm and BH adjustments, decision flags, paired-score hashes, and CSV/JSON exports from those arrays rather than accepting a summary table as evidence.

LLM eval multiple testing becomes reviewable when a release manager can ask why one row passed and receive the family size, rank, threshold, effect, and decision rule. “Adjusted by the notebook” is not an adequate provenance record.

Effect size and interval remain visible even when an adjusted decision fails. A practically valuable shift may be underpowered, while a tiny shift can become statistically clear in a large suite; the release owner needs magnitude, uncertainty, sample count, and multiplicity decision on one row.

Simulate the false-win pressure

A seeded null simulation can make multiplicity tangible by drawing many families where every hypothesis has no effect, then counting unadjusted and adjusted flags. This exercise checks intuition and implementation behavior; it is not proof that a real eval’s p-values are uniform, independent, or correctly calibrated.

Run enough seeded repetitions for a stable teaching picture and retain the generator, seed, family size, alpha, correction, and result counts. If a method is changed, regenerate the receipt instead of preserving an old chart whose labels happen to match.

Pair simulation with measurement contracts for AI evals. The contract should explain sampling, labels, grader reliability, exclusions, family definition, error target, practical threshold, and who may authorize a confirmatory rerun.

The synthetic family includes known null and shifted comparisons so the artifact can test both restraint and sensitivity. Seed generation, store the paired differences, and assert the adjusted counts; those counts describe this committed fixture, not a universal expectation for production model quality.

Missing outcomes and grader failures must be resolved before ranking p-values. Define whether a missing score is excluded, imputed, treated as failure, or sent to adjudication, and apply that rule symmetrically across candidates. Report the count by hypothesis because a seemingly precise adjusted result can rest on a different evaluated population than its neighbors. Corrections cannot recover information discarded by a biased missingness rule.

Separate exploration from confirmation

Exploratory analysis can search widely for failure modes, subgroup patterns, and promising prompts, provided its outputs are labeled hypotheses rather than confirmed wins. Move selected findings into a new evaluation with fresh or properly held-out examples and a frozen analysis plan before using them as release claims.

Repeatedly checking the same benchmark after every prompt edit creates adaptation even if each individual table uses a correction. Track dataset exposure, candidate count, human decisions, and prior looks, then rotate or replenish confirmation sets according to a declared policy.

OpenAI trace grading for regressions can supply structured evidence at the run level, but multiple tested graders and trace slices still belong in the family. Tooling changes the observation format; it does not remove the inference problem.

LLM eval multiple testing should survive a changed display order. The lab keys hypotheses by stable IDs, computes from the frozen family, and joins labels afterward, preventing a table sort or renamed slice from changing correction order or disconnecting an interval from its decision.

Audit grader and slice multiplication

A model grader introduces prompts, models, temperature or sampling, reference material, and calibration thresholds that can multiply analytic choices. Freeze the grader configuration, assess agreement and directional bias on a rights-cleared labeled set, and treat alternative grader variants as hypotheses if their results can influence the release.

Likewise, slice definitions should come from product risk or an independent taxonomy rather than a scan for the smallest p-value. Calibrate the LLM judge before its scores enter a corrected table, and preserve human-review escalation for cases where an adjusted statistic hides substantive disagreement.

LLM eval multiple testing cannot repair contaminated data, label leakage, post-treatment filtering, dependent units treated as independent, or an invalid raw test. Put those validity checks before correction and retain their results in the same release evidence bundle.

Missing scores are part of the estimand, not housekeeping. Report how many pairs were excluded and why, reject asymmetric silent drops, and rerun sensitivity analysis when failures cluster by model or slice; otherwise the clean paired table can conceal a systematic serving defect.

Ship an adjusted evaluation receipt

The final receipt should include the immutable family manifest, data and grader digests, estimator, raw test method, correction and assumptions, alpha, practical threshold, source code version, pinned-vector tests, ranked table, effect intervals, exclusions, exploratory notes, approver, and planned replication. Hash the artifacts so a later report can reproduce the exact decision inputs.

Run the included Node lab and inspect the committed paired scores, derived intervals, exact sign-flip enumeration, Holm and BH vectors, family ID, and familywise null-simulation incidence. Before substituting production scores, a statistical reviewer still needs to approve the pairing, interval model, exchangeability assumption, family definition, and release criterion; one implemented fixture is not a universal test.

Revisit when a new slice, metric, grader, candidate, or assumption joins the decision family. A trustworthy process never lets a new comparison drift into an old denominator invisibly, and it never celebrates an adjusted win without showing the magnitude the product would actually ship.

Revisit LLM eval multiple testing when judge configuration, candidate roster, bootstrap method, or release gate changes. A compact decision ladder can introduce the idea on social channels, but the downloadable paired dataset and adjustment script remain the evidence that distinguishes a result from a false win.