Item Response Theory for Better LLM Evals

Item response theory can separate model ability from prompt difficulty inside a declared LLM evaluation bank, where plain accuracy collapses both into one average. This tutorial fits a bounded 1PL/Rasch model, inspects residuals and information, and selects a domain-quota-constrained follow-up slice without calling the result general intelligence.

Why accuracy hides item difficulty

A benchmark average compresses two causes into one number. A model can miss because its capability is weak for the tested construct, because an item is unusually difficult, or because the item is ambiguous or broken. Comparing raw accuracy across a changing prompt bank silently treats every question as interchangeable. That makes a five-point movement hard to interpret and encourages teams to spend inference on repetitive easy cases.

Item response theory gives the response matrix a structure. Rows are model snapshots, columns are versioned items, and each cell is correct, incorrect, or missing. The model estimates a shared ability coordinate for each snapshot and a difficulty coordinate for each item. Those coordinates are local to this bank and population; they are not a declaration of general intelligence. The 2026 BEA study on LLM grading ability and response difficulty demonstrates why jointly reasoning about graders and responses is more revealing than one aggregate score.

Use that separation beside ordinary uncertainty estimates, not instead of them. Confidence intervals for LLM evals still describe sampling uncertainty around a reported metric. Item response theory addresses a different question: which mixture of model ability and prompt difficulty could have produced the pattern? The useful output is therefore a diagnostic map, not a new universal leaderboard.

Bound the 1PL Rasch model

The bounded lab uses a one-parameter logistic, or Rasch, model. For model m and item i, the probability of a correct answer is the logistic function of ability minus difficulty. One positive logit means the model is more likely than not to answer; one negative logit means less likely. Every item shares the same discrimination slope, so the fit cannot explain awkward data by inventing a different sharpness for every prompt.

That restriction is a feature for a small engineering receipt. It keeps the optimization deterministic, the parameters auditable, and the visual ruler interpretable. The lab clamps coordinates, adds light regularization, alternates ability and difficulty updates, and records convergence instead of claiming a perfect maximum-likelihood solution. A richer 2PL or multidimensional model may fit better, but it also needs more observations, stronger identification choices, and a much more careful story about what each dimension means.

The phrase adaptive LLM evaluation can tempt teams to overreach. Here, adaptation only means selecting informative items from a declared bank after fitting declared responses. It does not mean continuously probing a user, training on hidden outcomes, or generalizing beyond the domains represented in the matrix. Keep the bank, scoring rule, model snapshots, and fitting version together so the receipt can be reproduced when any one of them changes.

Version the response matrix

A useful matrix begins before the math. Give every item a stable identifier, domain label, prompt-version digest, scoring-rule version, and binary outcome definition. Give every model row a model identifier, provider or checkpoint version, decoding configuration, tool policy, and evaluation date. Missing must remain null. Converting an unavailable run into incorrect would make an operational gap look like a capability failure.

The artifact accepts only a bounded JSON object: models, items, domain labels, and a rectangular matrix containing zero, one, or null. It rejects duplicate identifiers, unknown domains, ragged rows, non-binary values, empty observed rows or columns, and oversized input before fitting. That contract makes LLM eval difficulty traceable to a concrete item edition. It also keeps a repaired rubric from overwriting the evidence collected with the old rubric.

Contamination deserves its own signal. A suspiciously easy item can reflect memorization, but difficulty alone cannot prove it. Pair this diagnostic with contamination canary tests and investigate prompt provenance. Likewise, keep judge behavior separate: LLM judge calibration tests the scorer, while the matrix here assumes the recorded binary decisions are the observations to model. Versioning lets those investigations meet without confusing their responsibilities.

Fit and identify one shared scale

Ability and difficulty are identifiable only up to a shared translation: adding the same constant to both leaves every probability unchanged. The lab resolves that ambiguity by centering item difficulties at zero after each alternating update and shifting abilities by the same amount. It then sorts identifiers only for computation and restores the requested presentation order. Permuting rows or columns therefore changes display order, not fitted values keyed by identity.

Alternating fit is easy to inspect. First update each model ability against its observed items; then update each item difficulty against its observed models. Regularization prevents all-correct or all-wrong margins from running toward infinity. A fixed iteration cap and tolerance bound the work. The receipt exposes iteration count, maximum parameter change, centered-difficulty mean, log likelihood, and a digest of normalized inputs and outputs.

This is the point where item response theory can earn trust or lose it. Never hide the identification convention, missing-value policy, or regularization strength. Compare the returned likelihood with an independent oracle, rerun a permuted matrix, and include degenerate margins in tests. The parameter values are meaningful only inside the declared model family and response population. Their value is practical: they make response patterns comparable enough to find weak items and plan a smaller follow-up slice.

Keep a raw-score table beside the fitted view. Item response theory explains structure in the matrix; it should never make the underlying successes, failures, or missing observations disappear. When someone disputes a coordinate, the team must be able to walk back from that point to the exact versioned responses that influenced it.

Read characteristic and item information curves

An item characteristic curve turns a position on the shared ruler into a predicted probability of success. At ability equal to difficulty, the bounded 1PL curve is 0.5. Move one logit above difficulty and the probability is about 0.731; move one below and it is about 0.269. Those exact checkpoints make a better visual contract than a decorative sigmoid because a reviewer can recalculate them.

Item information in this model is p times one minus p. It reaches 0.25 at the item's difficulty and falls symmetrically away from that point. An easy item is informative for weaker models and nearly uninformative once every candidate answers it correctly. A hard item behaves in the opposite direction. Item information is not importance: a safety-critical case can deserve permanent inclusion even when it adds little statistical separation near the current candidates. Item response theory supplies the curve, while release policy supplies the consequence.

The tinyBenchmarks method is useful context for evaluation with fewer examples, while the ACL work on unequal information in evaluation examples makes the broader point that examples do not contribute equally. The lab implements a deliberately smaller claim: calculate 1PL information at one declared target ability, then select under explicit domain quotas. It does not reproduce either paper's full method or guarantees.

Treat misfit and missingness as product signals

Residuals compare an observed binary result with its fitted probability. A large positive residual means a model succeeded where the shared scale expected difficulty; a large negative one means an unexpected miss. Aggregate residuals by item and inspect the largest absolute cases. They can reveal ambiguous wording, judge inconsistency, capability interactions, or a domain that one shared dimension cannot represent.

Do not automatically delete a misfitting prompt. Read it. Re-run it with preserved traces, inspect the rubric, and ask whether the surprising pattern exposes a valuable specialized skill. A one-dimensional Rasch model will misdescribe data when capabilities cross: one model may be stronger at code and weaker at multilingual reasoning despite the same total accuracy. Domain labels and quota selection keep that limitation visible, but they do not solve multidimensionality.

Missingness is equally substantive. If expensive tool-use items are missing mostly for one provider, the observed matrix is not missing at random. Report observed counts per row, item, and domain, then compare the selected slice with the full bank. Multiple-testing controls for LLM evals remain necessary when teams search many residuals and celebrate only the dramatic ones. Item response theory should narrow investigation, not license uncorrected fishing.

Choose an informative slice with domain quotas

Choose a target ability before ranking. It can be the median fitted ability, a release candidate, or a threshold model that represents the minimum acceptable capability. Compute every item's information at that coordinate, then satisfy each declared domain minimum before filling remaining slots by information. Stable identifier tie-breaks make the selection reproducible.

Quotas protect intent. Without them, five nearly identical reasoning items near the target could crowd out a low-frequency safety domain. With them, the slice preserves coverage while spending its flexible slots where the current comparison is most uncertain. The receipt lists selected and omitted items, domain counts, per-item information, and the rule used to break ties. A reviewer can see exactly why an item entered the small run.

The slice is a routing tool, not a permanent replacement for the full bank. Run the complete benchmark on a slower cadence, after meaningful model changes, and whenever residuals or domain outcomes drift. Use canary evals for AI releases to keep non-negotiable regression checks outside the optimization. The resulting adaptive LLM evaluation remains anchored to release policy: informative items improve resolution, while mandatory cases preserve risk coverage even when their item information is low.

Keep a reproducible item response theory receipt

A useful receipt answers four questions without rerunning the narrative: what data entered, what bounded model ran, what evidence came out, and what the result cannot establish. Record the input digest, ordered identities, observed and missing counts, fitting constants, convergence status, likelihood, centered coordinates, probabilities, residual summaries, target ability, quotas, selection, and output digest. Include a replay command and implementation version.

The downloadable lab checks stability by running the same fixture twice and permutation invariance by reordering identities. Its independent tests recompute likelihood and probabilities, exercise missing cells plus all-correct and all-wrong margins, reject malformed inputs, and kill a sign mutant and a selector that ignores quotas. Those controls are stronger than asserting that the program agrees with its own receipt. They also turn item response theory into an inspectable engineering decision rather than a spreadsheet ceremony.

Keep the conclusion narrow. Item response theory can separate ability and difficulty under a declared 1PL model well enough to diagnose items and plan a follow-up sample. It cannot measure general intelligence, prove benchmark validity, repair a biased bank, or guarantee that a smaller slice preserves every ranking. Publish that boundary beside the numbers. The honest artifact is valuable precisely because another engineer can replay its assumptions, challenge a residual, and restore the full benchmark when the small slice stops serving the decision.

Runnable artifact — Regularized one-dimensional 1PL diagnostics for this declared binary response bank; not general intelligence, benchmark validity, or guaranteed ranking preservation.

import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";

const MAX_BYTES=262144,MAX_MODELS=64,MAX_ITEMS=256,MAX_ITERATIONS=500,VERSION="irt-eval-lab-v1";
const builtIn={models:["baseline","candidate","specialist"],domains:["reasoning","code","safety"],items:[{id:"r1",domain:"reasoning"},{id:"r2",domain:"reasoning"},{id:"c1",domain:"code"},{id:"c2",domain:"code"},{id:"s1",domain:"safety"},{id:"s2",domain:"safety"}],responses:[[1,0,0,0,1,null],[1,1,1,0,1,0],[1,1,1,1,0,1]],targetAbility:0,sliceSize:4,domainQuotas:{reasoning:1,code:1,safety:1}};
const canonical=value=>JSON.stringify(normalize(value));
const normalize=value=>Array.isArray(value)?value.map(normalize):value&&typeof value==="object"?Object.fromEntries(Object.keys(value).sort().map(key=>[key,normalize(value[key])])):value;
const hash=value=>createHash("sha256").update(typeof value==="string"?value:canonical(value)).digest("hex");
function parseArgs(){const args=process.argv.slice(2);if(args.length===0)return structuredClone(builtIn);if(args.length!==2||args[0]!=="--fixture")throw new Error("invalid-arguments");const bytes=readFileSync(args[1]);if(bytes.length>MAX_BYTES)throw new Error("fixture-too-large");try{return JSON.parse(bytes.toString("utf8"));}catch{throw new Error("malformed-fixture");}}
function validate(input){if(!input||typeof input!=="object"||Array.isArray(input))throw new Error("invalid-fixture");const allowed=new Set(["models","domains","items","responses","targetAbility","sliceSize","domainQuotas"]);if(Object.keys(input).some(key=>!allowed.has(key)))throw new Error("unknown-field");const safe=value=>typeof value==="string"&&/^[A-Za-z0-9][A-Za-z0-9._ -]{0,63}$/.test(value);if(!Array.isArray(input.models)||input.models.length<2||input.models.length>MAX_MODELS||input.models.some(x=>!safe(x))||new Set(input.models).size!==input.models.length)throw new Error("invalid-models");if(!Array.isArray(input.domains)||input.domains.length<1||input.domains.length>32||input.domains.some(x=>!safe(x))||new Set(input.domains).size!==input.domains.length)throw new Error("invalid-domains");if(!Array.isArray(input.items)||input.items.length<2||input.items.length>MAX_ITEMS)throw new Error("invalid-items");const ids=new Set();for(const item of input.items){if(!item||typeof item!=="object"||Object.keys(item).some(k=>!["id","domain"].includes(k))||!safe(item.id)||ids.has(item.id)||!input.domains.includes(item.domain))throw new Error("invalid-item");ids.add(item.id);}if(!Array.isArray(input.responses)||input.responses.length!==input.models.length||input.responses.some(row=>!Array.isArray(row)||row.length!==input.items.length||row.some(v=>v!==0&&v!==1&&v!==null)))throw new Error("invalid-responses");for(let m=0;m<input.models.length;m++)if(input.responses[m].every(v=>v===null))throw new Error("empty-model-row");for(let i=0;i<input.items.length;i++)if(input.responses.every(row=>row[i]===null))throw new Error("empty-item-column");if(!Number.isFinite(input.targetAbility)||Math.abs(input.targetAbility)>8)throw new Error("invalid-target");if(!Number.isInteger(input.sliceSize)||input.sliceSize<1||input.sliceSize>input.items.length)throw new Error("invalid-slice-size");if(!input.domainQuotas||typeof input.domainQuotas!=="object"||Array.isArray(input.domainQuotas)||Object.keys(input.domainQuotas).some(key=>!input.domains.includes(key)))throw new Error("invalid-quotas");let quota=0;for(const domain of input.domains){const n=input.domainQuotas[domain]??0;if(!Number.isInteger(n)||n<0)throw new Error("invalid-quota");if(n>input.items.filter(item=>item.domain===domain).length)throw new Error("quota-unavailable");quota+=n;}if(quota>input.sliceSize)throw new Error("quota-exceeds-slice");return input;}
const logistic=x=>x>=0?1/(1+Math.exp(-x)):Math.exp(x)/(1+Math.exp(x));
const clamp=x=>Math.max(-6,Math.min(6,x));
function fit(input){const modelOrder=input.models.map((id,index)=>({id,index})).sort((a,b)=>a.id.localeCompare(b.id)),itemOrder=input.items.map((item,index)=>({...item,index})).sort((a,b)=>a.id.localeCompare(b.id));const response=modelOrder.map(model=>itemOrder.map(item=>input.responses[model.index][item.index]));let ability=modelOrder.map(()=>0),difficulty=itemOrder.map(()=>0),converged=false,delta=Infinity,iterations=0;const rate=.18,lambda=.08;for(;iterations<MAX_ITERATIONS;){iterations++;const previous=[...ability,...difficulty];for(let m=0;m<ability.length;m++){let grad=-lambda*ability[m],count=0;for(let i=0;i<difficulty.length;i++){const y=response[m][i];if(y===null)continue;grad+=y-logistic(ability[m]-difficulty[i]);count++;}ability[m]=clamp(ability[m]+rate*grad/Math.max(1,count));}for(let i=0;i<difficulty.length;i++){let grad=-lambda*difficulty[i],count=0;for(let m=0;m<ability.length;m++){const y=response[m][i];if(y===null)continue;grad+=logistic(ability[m]-difficulty[i])-y;count++;}difficulty[i]=clamp(difficulty[i]+rate*grad/Math.max(1,count));}const mean=difficulty.reduce((a,b)=>a+b,0)/difficulty.length;difficulty=difficulty.map(x=>x-mean);ability=ability.map(x=>x-mean);delta=Math.max(...[...ability,...difficulty].map((x,index)=>Math.abs(x-previous[index])));if(delta<1e-7){converged=true;break;}}const abilities=Object.fromEntries(modelOrder.map((model,index)=>[model.id,Number(ability[index].toFixed(8))])),difficulties=Object.fromEntries(itemOrder.map((item,index)=>[item.id,Number(difficulty[index].toFixed(8))]));let likelihood=0;const probabilities={},residuals={};for(let m=0;m<modelOrder.length;m++){for(let i=0;i<itemOrder.length;i++){const p=logistic(ability[m]-difficulty[i]),key=modelOrder[m].id+"/"+itemOrder[i].id;probabilities[key]=Number(p.toFixed(8));if(response[m][i]!==null){const y=response[m][i];likelihood+=y*Math.log(p)+(1-y)*Math.log(1-p);residuals[key]=Number((y-p).toFixed(8));}}}return{modelOrder,itemOrder,abilities,difficulties,probabilities,residuals,logLikelihood:Number(likelihood.toFixed(8)),iterations,iterationLimit:MAX_ITERATIONS,converged,stopReason:converged?"converged":"iteration-limit",maxDelta:Number(delta.toFixed(10))};}
function select(input,fitResult){const information=fitResult.itemOrder.map(item=>{const p=logistic(input.targetAbility-fitResult.difficulties[item.id]);return{id:item.id,domain:item.domain,probability:Number(p.toFixed(8)),information:Number((p*(1-p)).toFixed(8))};}).sort((a,b)=>b.information-a.information||a.id.localeCompare(b.id)),selected=[];for(const domain of input.domains){for(const item of information.filter(x=>x.domain===domain).slice(0,input.domainQuotas[domain]??0))if(!selected.some(x=>x.id===item.id))selected.push(item);}for(const item of information)if(selected.length<input.sliceSize&&!selected.some(x=>x.id===item.id))selected.push(item);return{targetAbility:input.targetAbility,domainQuotas:input.domainQuotas,items:selected,omitted:information.filter(item=>!selected.some(x=>x.id===item.id)),allInformation:information};}
const input=validate(parseArgs()),fitted=fit(input),selection=select(input,fitted),observed=input.responses.flat().filter(x=>x!==null).length;
const core={schema:VERSION,inputDigest:hash(input),fit:{abilities:fitted.abilities,difficulties:fitted.difficulties,iterations:fitted.iterations,iterationLimit:fitted.iterationLimit,converged:fitted.converged,stopReason:fitted.stopReason,maxDelta:fitted.maxDelta,logLikelihood:fitted.logLikelihood,centeredDifficultyMean:Number((Object.values(fitted.difficulties).reduce((a,b)=>a+b,0)/input.items.length).toFixed(10)),probabilities:fitted.probabilities,residuals:fitted.residuals},selection,counts:{models:input.models.length,items:input.items.length,observed,missing:input.models.length*input.items.length-observed},claimBoundary:"Regularized one-dimensional 1PL diagnostics for this declared binary response bank; not general intelligence, benchmark validity, or guaranteed ranking preservation."};
const receipt={...core,receiptHash:hash(core),replayCommand:"node irt-eval-lab.mjs"+(process.argv.length>2?" --fixture <same-path>":"")};
process.stdout.write(JSON.stringify(receipt,null,2)+"\n");