LLM Eval Contamination Canary Tests

LLM eval contamination can make a benchmark score look more informative than the evidence warrants, but corpus overlap alone cannot prove what entered a model's training data. This tutorial builds exact, normalized, and fuzzy canaries around a wholly synthetic benchmark, then routes suspicious slices to quarantine without overstating the conclusion.

LLM eval contamination starts with lineage

Record who authored each item, when it became accessible, which benchmark version contains it, and which public corpus snapshot the scanner examined. Without that timeline, an overlap result cannot distinguish copied benchmark text, a common source, a paraphrase, or material published after model training.

The paper on proving test-set contamination in black-box models addresses model-level evidence with methods beyond simple corpus matching. Keep that stronger claim lane separate from a local text scanner.

The committed data contains invented sentences about moths, robots, and lanterns. Its LLM eval contamination receipt describes only overlap between those supplied strings; it knows nothing about any provider corpus, hidden training run, or deployed model.

The synthetic benchmark and comparison corpus now receive independent SHA-256 digests before scanning. Each benchmark row also carries its own hash, keeping later threshold changes distinguishable from an accidental fixture edit. The two corpus identities remain separate throughout every reported comparison.

Freeze benchmark and comparison corpus snapshots

Hash the complete benchmark manifest, individual items, normalization profile, public corpus list, retrieval date, and scanner version. Store content access rules too, because a later reviewer must know whether the comparison set was exhaustive, sampled, or constrained by licensing.

Fresh LLM benchmarks reduce exposure time but do not create immunity. The LiveBench paper is relevant to dynamic benchmark design; release cadence, objective grading, and contamination evidence still need an explicit measurement contract.

An LLM eval contamination scan should never mutate the benchmark while producing its report. Proposed removals or rewrites become a new version with an auditable mapping to the prior slice.

Exact-byte and NFKC-normalized matches stay separate in the raw overlap record. That distinction makes case and punctuation normalization visible instead of allowing one boolean to obscure which evidence triggered quarantine. Byte history remains reviewable. Neither normalization branch silently overwrites the original text or digest.

Use exact hashes as the narrowest canary

Exact byte hashes identify identical content under one encoding. They are cheap, deterministic, and easy to interpret, but line endings, punctuation, Unicode forms, or added wrappers can defeat them even when the semantic item was copied.

Preserve raw bytes before normalization so an exact match remains reproducible. A match should name benchmark item, corpus document, positions when available, both digests, and the snapshot identities; a global count without item-level receipts is difficult to investigate.

LLM eval contamination policy can quarantine an exact match immediately because the benchmark's independence is uncertain, yet the wording must remain precise. The result proves identical supplied text, not test set leakage into a particular model's weights.

Eight fixed seeds create deterministic minimum-hash signatures over token trigrams. The receipt publishes every seed and the signature threshold, while direct Jaccard remains available as a separately named calculation rather than a hidden proxy. Replaying those seeds must reproduce the full signature vector exactly.

Normalize conservatively and keep the raw evidence

A normalized pass may apply Unicode normalization, case folding, whitespace collapse, or a documented punctuation policy. Each transformation broadens recall and creates new false-positive opportunities, so store the profile and both original strings beside the match.

Domain-specific canonicalization needs special care. Removing code formatting, numeric separators, units, or answer labels may turn distinct questions into one normalized sequence; run mutation fixtures that demonstrate which differences the profile intentionally ignores.

Use metamorphic tests for LLMs to reason about controlled transformations separately from benchmark contamination. In this synthetic LLM eval contamination corpus, case and a small wording change are reported differently rather than flattened into one certainty label.

The evidence ladder stops at corpus overlap. Even an exact match raises LLM eval contamination risk but does not establish that a deployed model trained on the text, memorized it, or gained its score from exposure.

Runnable artifact — Scan a frozen synthetic benchmark and corpus with exact, normalized, direct-shingle, seeded-signature, threshold, and hash evidence.

import assert from "node:assert/strict";import{createHash}from"node:crypto";
const sha=value=>createHash("sha256").update(value).digest("hex"),normalize=value=>value.normalize("NFKC").toLowerCase().replace(/[^\p{L}\p{N}]+/gu," ").trim(),shingles=value=>{const words=normalize(value).split(/\s+/),out=[];for(let i=0;i<=words.length-3;i++)out.push(words.slice(i,i+3).join(" "));return new Set(out)},fnv=(text,seed)=>{let h=(2166136261^seed)>>>0;for(const ch of text){h^=ch.codePointAt(0);h=Math.imul(h,16777619)>>>0}return h},signature=(text,seeds)=>{const grams=[...shingles(text)];return seeds.map(seed=>grams.length?Math.min(...grams.map(x=>fnv(x,seed))):0)},estimate=(a,b)=>a.filter((v,i)=>v===b[i]).length/a.length,direct=(a,b)=>{const A=shingles(a),B=shingles(b);let hit=0;for(const x of A)if(B.has(x))hit++;return hit/(A.size+B.size-hit||1)};
const seeds=[11,29,47,71,101,131,173,211],thresholds={exact:true,normalized:true,directJaccard:.4,seededSignature:.375},benchmark=["Copper moths navigate by a violet harbor light","A ceramic robot counts rain on the station roof","The quiet river keeps seven paper lanterns"],corpus=["Copper moths navigate by a violet harbor light","A CERAMIC robot counts rain upon the station roof","Unrelated public sentence about orchard tools"];
const rows=benchmark.map(text=>{const candidates=corpus.map(source=>{const raw={sourceHash:sha(source),exact:sha(source)===sha(text),normalized:sha(normalize(source))===sha(normalize(text)),direct:+direct(source,text).toFixed(3),signature:+estimate(signature(source,seeds),signature(text,seeds)).toFixed(3)};return raw});const best=candidates.sort((a,b)=>Math.max(b.direct,b.signature)-Math.max(a.direct,a.signature))[0],quarantine=best.exact||best.normalized||best.direct>=thresholds.directJaccard||best.signature>=thresholds.seededSignature;return{benchmarkHash:sha(text),rawOverlap:best,decision:quarantine?"quarantine":"retain"}});
const receipt={fixture:"wholly synthetic corpus",seeds,thresholds,corpusHash:sha(JSON.stringify(corpus)),benchmarkHash:sha(JSON.stringify(benchmark)),rows,boundary:"corpus overlap signal is not proof of model training exposure"};assert.deepEqual(rows.map(x=>x.decision),["quarantine","quarantine","retain"]);assert.equal(new Set(rows.map(x=>x.benchmarkHash)).size,3);console.log(JSON.stringify(receipt,null,2));console.log("PASS: seeded signatures hashes thresholds and quarantine align");

Add fuzzy overlap without manufacturing certainty

Token shingles, edit similarity, or seeded MinHash-style signatures can surface near duplicates that hashes miss. Choose tokenization, shingle width, minimum length, threshold, and stop-word policy before seeing candidate outcomes, then retain the continuous score rather than only a red badge.

Short generic items collide easily, and formulaic instructions can create high overlap without copied answers. Review matched spans, source dates, domain specificity, and negative controls; route ambiguous candidates to inspection instead of automatically deleting valuable evaluation coverage.

The LLM eval contamination fixture uses bounded three-token shingles and a declared quarantine threshold. It is an educational detector, not an implementation of every technique in ConStat or a statistical statement about a black-box model.

Quarantine decisions are derived in the same row as exact, normalized, direct-overlap, and seeded-signature values. A reviewer can therefore reproduce the branch without consulting prose or trusting a postprocessed status badge. The row also preserves which threshold, rather than which narrative, fired.

Separate corpus signals from model-level tests

Corpus discovery says a benchmark item appears in material a training pipeline could potentially access. Model-level evidence asks whether behavior is unusually consistent with exposure, requiring carefully controlled hypotheses, baselines, uncertainty, and alternatives.

Do not write contaminated model in a dashboard when the system ran only search or string similarity. Labels such as public overlap found, benchmark slice quarantined, or exposure risk unresolved preserve the distinction and give downstream reviewers an actionable state.

Pair LLM eval contamination results with confidence intervals for evaluation comparisons so removing a slice does not create another overconfident score. Report coverage loss, uncertainty change, and whether the revised evaluation still answers its original decision.

The canary corpus contains an exact reuse, a paraphrase-like partial overlap, and a clean control. Those generated cases test contamination detection sensitivity and refusal boundaries; they do not estimate prevalence in private training data. Ambiguity stays owned. Each case is traceable to a rights-clear sentence in the committed corpus.

Quarantine, refresh, or retain by policy

Exact distinctive overlap may justify quarantine; normalized or fuzzy candidates may need review; clean items remain only clean relative to the scanned snapshot. A quarantine preserves evidence and prevents scoring use while a deletion erases investigation context.

Refresh an item with newly authored content when its capability target can be preserved and objective grading remains possible. Keep old and new identifiers distinct, rerun baselines, and avoid publishing answer keys or canaries in a distribution channel that defeats the intended freshness.

The canary evaluation release gate can consume these states before a model comparison is accepted. LLM eval contamination becomes a release input, not a post-hoc excuse for whichever score is inconvenient.

Threshold review should compare false positives and missed overlaps on a larger rights-clear calibration set. The current 0.4 direct-Jaccard and 0.375 signature cutoffs are teaching parameters frozen in the receipt, not general recommendations. Moving either cutoff creates a new decision-policy edition and reruns every row.

Publish the boundary beside every result

Archive benchmark digest, comparison corpus digest, timestamps, scanner code, exact and normalized hashes, fuzzy parameters, matched spans, thresholds, review outcomes, and the final included item set. Link the evaluation result to that receipt so a later corpus discovery can trigger reanalysis.

Write the broader evaluation measurement contract before choosing how much overlap changes a release decision. Different stakes may require different quarantine sensitivity, but none authorizes unsupported claims about hidden training data.

Run the synthetic scanner and inspect why its exact, near, and clean cases differ. The strongest LLM eval contamination workflow is epistemically modest: it makes local overlap undeniable, model exposure explicitly unknown, and next actions reproducible.

Fresh benchmark items require new benchmark and corpus hashes, not an in-place edit. That release discipline keeps LLM eval contamination evidence attributable when public discussion or documentation later repeats a once-novel prompt. The old slice stays identifiable. Its prior disposition remains available for comparison after the refresh.

Use negative controls to calibrate overlap alarms

Build comparison items that share domain vocabulary, formatting, answer length, and instruction style without copying the benchmark. A detector that flags every conventional phrase or code scaffold will quarantine whole domains, while a detector tuned only on obvious duplicates will miss small edits; negative controls reveal that operating tradeoff before policy touches a release score.

Stratify results by item length and content type because similarity behaves differently for a short arithmetic prompt, a long narrative question, source code, and multiple-choice answers. Review exact spans and candidate sources at the threshold boundary, then freeze the decision before scanning a new model result to avoid outcome-driven tuning.

LLM eval contamination monitoring should track alert precision on reviewed corpus pairs, not claim detector accuracy on hidden training exposure. Store false-positive rationales and use them to improve normalization or shingling only through a new versioned fixture whose changed quarantine set is independently inspectable.

A quarantined slice can still be inspected under a separate policy, but it cannot silently contribute to the headline score. The decision ledger preserves the raw match evidence and the owner who adjudicates ambiguous overlap.

Refresh scores without erasing the original record

When a benchmark slice is quarantined, recompute the aggregate on the retained set and publish both the original and revised denominators. Show which capabilities lost coverage, how uncertainty changed, and whether the release decision is robust to plausible outcomes for removed items; a cleaner number can be less useful if it no longer measures the intended task.

New items need independent authoring, answer verification, leakage review, and baseline calibration before replacing old ones. Keep identifiers and histories distinct so a refreshed prompt cannot inherit old scores or silently alter a public leaderboard under the same benchmark version.

LLM eval contamination is managed well when the organization can revisit an old decision after a new corpus discovery. The receipt should reconstruct exactly what was known at scoring time, which items were included, who reviewed alerts, and how the evaluation changed without converting uncertainty into an accusation.

The December review will rerun seeds, hashes, thresholds, and source citations. A ranking update alone does not refresh this article; the deterministic canary receipt must change or reaffirm the same boundary.