Bradley–Terry vs Elo for LLM Rankings

Bradley-Terry vs Elo is a choice between a batch fit and an online rating history, and neither should publish a global LLM order before the comparison graph and ranking fragility are visible. This comparison runs one declared vote bank through both methods, reorders the Elo history, bootstraps the Bradley-Terry model, removes influential votes, and returns ABSTAIN when the evidence cannot identify one scale.

Pairwise votes are a graph

Before comparing Bradley-Terry vs Elo, I draw the comparison graph. Models are nodes. Every judged matchup adds an edge, with win, loss, or tie evidence attached. That picture answers a question no leaderboard can repair later: is there enough shared evidence to place every model on one scale?

A connected graph does not need every possible pair, but every node needs a path to every other node. If models A, B, and C only face each other while D and E live in a separate pool, the data identifies two local orders, not one global order. Adding a constant to all scores in one component changes nothing about its likelihood. Any precise cross-component rank is therefore an editorial invention.

This is why my lab checks connectivity before fitting. It reports the components, comparison counts, and degree of every model. A disconnected fixture returns ABSTAIN for a global LLM model ranking while still preserving the evidence that is locally interpretable. That is a stronger result than filling a table with false precision.

The graph also exposes thin bridges. One comparison can technically connect two dense groups while carrying too much influence. I keep that bridge visible for later vote-removal stress instead of letting the word “connected” stand in for “well supported.” Item response theory for LLM evals makes a similar move with item banks: identify what the observed structure can support before naming a score.

What Elo updates online

Elo is an online update rule. It takes current ratings, converts their difference into an expected pairwise result, then moves each rating by a K-factor times the surprise. A stronger-than-expected win moves little; an upset moves more. That makes Elo operationally attractive when votes arrive as a stream and the product wants a rating after every match.

The convenience carries policy. Starting ratings anchor the early path. K controls responsiveness. The exact vote order changes which rating difference each later result encounters. Tie scoring must be declared. New-model admission and rating inactivity also become product decisions. None of these make Elo unusable; they make the preference leaderboard historical. Bradley-Terry vs Elo becomes useful only when that history is part of the comparison.

For Bradley-Terry vs Elo, order dependence is the most concrete contrast. The lab replays one fixed multiset of votes in two deterministic orders. Both runs contain identical wins and ties, yet their intermediate paths—and sometimes final ratings—separate because each update sees a different state. A small K narrows the separation while adapting slowly. A large K reacts quickly and preserves more of the path.

I would choose that behavior intentionally for a live ladder or routing system. I would not describe its last rating table as if it were a batch estimate independent of ingestion order. The receipt needs starting score, scale convention, K, tie policy, exact ordering key, and replay digest. Without those fields, another team cannot reproduce the number even when it has the same votes.

What Bradley–Terry fits in batch

The Bradley-Terry model treats each pairwise outcome as evidence about latent positive strengths, usually expressed as unconstrained log abilities. The probability that model A beats model B is a logistic function of their ability difference. Batch fitting asks for the parameter set that best explains all declared comparisons together.

That joint view is the reason I reach for a Bradley-Terry model when I have a frozen evaluation window and need a reproducible comparison. Row order should not matter because the likelihood aggregates the same sufficient evidence. The lab verifies this by shuffling the input rows and requiring the centered fitted scores to agree within a declared numerical tolerance.

Batch does not mean objective. The fit still inherits the prompts, raters, sampling plan, tie treatment, cohort mix, and missing matchups in the dataset. It also needs an identification rule because only score differences matter. I center the fitted log abilities to a mean of zero and report the convergence threshold and iteration limit. The generalized Bradley-Terry methods paper provides the optimization foundation; the lab remains a small educational fit, not a production statistical package.

I also avoid calling the maximum-likelihood order stable merely because the optimizer converged. Convergence says the procedure settled on this declared graph. It does not say nearby plausible datasets preserve the same order. That second question belongs to uncertainty and influence checks.

Run Bradley-Terry vs Elo on the same votes

A fair Bradley-Terry vs Elo comparison starts from one canonical vote table. Each record names a winner, loser, prompt group, and weight; a tie becomes two reciprocal half-weight outcomes under the declared policy. I validate identifiers, reject self-matches, bound weights, sort only for the canonical digest, and then hand the same validated records to both methods.

The batch result is one centered endpoint. The Elo result is a path plus an endpoint for each ordering policy. That difference should stay visible in the article and artifact. Flattening both into a rank column erases the mechanism the comparison is supposed to teach.

The Chatbot Arena paper is useful context because it connects pairwise human preferences to an operating leaderboard and describes the statistical work required around the ranking. My synthetic fixture is much smaller. It proves the mechanics and replay contract, not Arena-scale accuracy, rater agreement, or general model quality.

For the default fixture, I show scores and rank positions, then deliberately avoid crowning a winner in prose. The interesting output is where the methods agree, where Elo orderings separate, and which comparisons carry the narrowest support. That turns pairwise LLM evaluation into an inspectable decision instead of a decorative league table. It also complements confidence intervals for LLM evals: one article asks how uncertain a metric is, while this one asks how method and preference evidence shape an order.

Detect disconnected and fragile rankings

Connectivity is the first gate; fragility is the second. After the fit, the lab bootstraps votes with a fixed seed, refits the connected samples, and records each model’s observed rank band. The interval is a diagnostic over this empirical vote bank. It is not a confidence guarantee for future users, prompts, or products.

I then run a removal stress: drop each vote in turn, refit, and record whether the top pair flips. This is intentionally modest. Exhaustive worst-case subset removal is a different optimization problem, but even a one-vote scan can reveal a leaderboard leaning on a tiny bridge or an unusual upset. The 2026 paper on preference-removal ranking flips shows why influential preferences deserve inspection rather than being buried in an aggregate.

The output uses three statuses. PASS means the graph is connected and every requested diagnostic completed. FRAGILE means fitting succeeded but bootstrap bands overlap materially or a declared removal flips the top order. ABSTAIN means the data cannot identify a global ranking or validation failed. “Pass” is not a quality badge; it only says the Bradley-Terry vs Elo receipt is complete under the declared checks.

I prefer this language because it keeps the preference leaderboard proportional to its evidence. A close rank with overlapping bands should look close. A global order across disconnected pools should not render. The UI earns trust by preserving ambiguity rather than rounding it away.

Handle ties, cohorts, and position bias

Tie handling changes the likelihood and must be public. This lab uses half a win for each side in both methods so the comparison stays compact. A production system might model ties explicitly, ask for “both bad” separately from “both good,” or remove ambiguous votes after a declared review. Those are measurement choices, not cleanup details.

Aggregation can hide more than ties. A pairwise LLM evaluation may mix coding prompts, multilingual tasks, safety refusals, short answers, and long-form reasoning. A single Bradley-Terry model assumes one shared latent order is useful across that mix. If cohorts disagree, the aggregate can be mathematically neat and operationally misleading. Bradley-Terry vs Elo does not resolve that missing cohort model.

I would publish cohort slices beside the overall LLM model ranking when sample sizes support them. I would also randomize left/right presentation, retain position, capture rater or judge version, and audit whether position predicts the vote after controlling for the pair. LLM judge calibration helps with the judge layer; it does not erase prompt sampling or preference-definition problems.

The lab therefore accepts a prompt-group field and reports its counts, but it does not estimate a hierarchical model. That boundary is deliberate. A compact artifact can show missing strata and invite a better analysis without pretending to solve heterogeneity. My rule is simple: if the ranking will drive release, routing, or marketing claims, the receipt should reveal whose preferences and which tasks the order summarizes.

Choose a method by operating need

I use Bradley-Terry vs Elo as an operating decision, not a tournament about which formula wins.

Choose Elo when the product genuinely needs sequential updates: a live ladder, online matchmaking, or a visible score after each accepted vote. Freeze the K-factor, starting ratings, tie rule, and ingestion order. Recompute from the event log when policy changes instead of silently continuing a rating history under new rules.

Choose a Bradley-Terry model when the evaluation window can close before publication and the team wants a batch estimate over all declared comparisons. Preserve the raw graph, identification constraint, convergence receipt, and uncertainty diagnostics. Refit when the dataset changes. Do not compare disconnected components.

Use both when operations and analysis have different jobs. Elo can power the live surface while a periodic batch fit audits whether the order is path-sensitive or thinly connected. If the two disagree, that is a prompt for investigation, not permission to select the preferred answer.

In every case, start with the evaluation contract. AI evals need measurement contracts explains why construct, sampling, decision threshold, and failure policy should exist before the score. A ranking method cannot rescue vague criteria or biased matchups. The method only organizes the evidence it receives. The responsible choice is the one whose history, assumptions, and failure states match the decision you need to make.

Publish a ranking receipt

A ranking becomes reviewable when it travels with a receipt. Mine starts with the dataset digest, model identifiers, prompt-group counts, tie policy, and comparison components. It records the Bradley-Terry model constraint and convergence, plus the Elo K-factor, starting score, scale, and two ordering digests.

Then it adds the evidence that is easiest to omit: bootstrap seed and repetitions, rank bands, influential vote removals, method-specific scores, and the final PASS, FRAGILE, or ABSTAIN status. A Bradley-Terry vs Elo receipt should be downloadable as JSON and readable in the page. A screenshot of sorted names is not enough.

Versioning matters. If raters, judge prompts, sampling weights, deduplication, or tie policy changes, create a new receipt. Keep the previous one so the team can explain why a preference leaderboard moved. If the same canonical dataset produces a different batch result outside tolerance, treat that as a regression. If an Elo replay differs with the same order, treat that as a broken history.

This is the practical answer to Bradley-Terry vs Elo: choose the update model that matches the operating need, then make fragility impossible to hide. The lab gives you a small, deterministic place to practice that discipline. It does not certify external validity. It makes the next review sharper by showing exactly what the rank could—and could not—learn from the votes.

Runnable browser artifact — Declared pairwise fixture and deterministic stress tests only; not proof of universal model quality, unbiased preferences, or cross-cohort validity.

<!doctype html><html lang="en"><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Bradley–Terry vs Elo ranking lab</title><style>:root{color-scheme:light dark;font-family:system-ui;background:#071812;color:#eef7f2}body{max-width:72rem;margin:auto;padding:2rem}button,select{font:inherit;min-height:44px;padding:.65rem 1rem}button{background:#6ee7a8;color:#062017;border:0;border-radius:.6rem;font-weight:700}button:disabled{opacity:.55}section{background:#10261f;border:1px solid #315046;border-radius:1rem;padding:1rem;margin:1rem 0}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(14rem,1fr));gap:1rem}pre{white-space:pre-wrap;overflow-wrap:anywhere;background:#06130f;padding:1rem;border-radius:.7rem}.path{display:flex;gap:.3rem;align-items:end;height:8rem}.step{width:1.5rem;background:#6ee7a8;min-height:.4rem}.muted{color:#aac3b8}</style><main><h1>Bradley–Terry vs Elo ranking lab</h1><p>Replay one bounded synthetic vote bank, inspect connectivity, compare Elo order paths, and stress a batch fit. No model is globally ranked when the comparison graph is disconnected.</p><section aria-labelledby="controls"><h2 id="controls">Replay controls</h2><label for="order">Elo vote order</label> <select id="order"><option value="forward">Forward</option><option value="reverse">Reverse</option></select> <button id="run" type="button">Run ranking receipt</button> <button id="download" type="button" disabled>Download JSON receipt</button></section><div class="grid"><section><h2>Method summary</h2><dl id="summary"></dl></section><section><h2>Elo score path</h2><div id="path" class="path" aria-label="Elo score path"></div></section></div><section><h2>Machine-readable receipt</h2><pre id="receipt" tabindex="0"></pre></section><p class="muted">Synthetic teaching fixture. Rank bands describe this vote bank only.</p></main><script>(()=>{"use strict";const MAX_MATCHES=1000,MAX_MODELS=64,MIN_WEIGHT=.01,MAX_WEIGHT=10,builtIn={models:["Atlas","Birch","Cinder","Dune"],matches:[{winner:"Atlas",loser:"Birch",group:"reasoning"},{winner:"Cinder",loser:"Dune",group:"code"},{winner:"Atlas",loser:"Cinder",group:"writing"},{winner:"Dune",loser:"Birch",group:"reasoning"},{winner:"Birch",loser:"Cinder",group:"code"},{winner:"Atlas",loser:"Dune",group:"writing"},{winner:"Cinder",loser:"Atlas",group:"code"},{winner:"Birch",loser:"Dune",group:"writing"}],tiePolicy:"half-win",k:24,seed:918,bootstrapRuns:80};
const canonical=value=>Array.isArray(value)?value.map(canonical):value&&typeof value==="object"?Object.fromEntries(Object.keys(value).sort().map(key=>[key,canonical(value[key])])):value,stable=value=>JSON.stringify(canonical(value)),hash=value=>{let h=2166136261,s=typeof value==="string"?value:stable(value);for(let i=0;i<s.length;i++){h^=s.charCodeAt(i);h=Math.imul(h,16777619);}return(h>>>0).toString(16).padStart(8,"0");},rng=seed=>{let state=seed>>>0||1;return()=>((state=Math.imul(state,1664525)+1013904223>>>0)/4294967296);};
function validate(input){if(!input||typeof input!=="object"||Array.isArray(input))throw Error("invalid-input");const f=structuredClone(input);if(!Array.isArray(f.models)||f.models.length<2||f.models.length>MAX_MODELS||new Set(f.models).size!==f.models.length||f.models.some(x=>typeof x!=="string"||!/^[A-Za-z0-9][A-Za-z0-9 ._-]{0,39}$/.test(x)))throw Error("invalid-models");if(!Array.isArray(f.matches)||!f.matches.length||f.matches.length>MAX_MATCHES)throw Error("invalid-matches");const names=new Set(f.models);for(const row of f.matches){if(!row||typeof row!=="object"||!names.has(row.winner)||!names.has(row.loser)||row.winner===row.loser||typeof row.group!=="string"||!row.group.length||row.group.length>40)throw Error("invalid-match");if(row.weight!==undefined&&(!Number.isFinite(row.weight)||row.weight<MIN_WEIGHT||row.weight>MAX_WEIGHT))throw Error("invalid-weight");row.weight=row.weight??1;}if(f.tiePolicy!=="half-win"||!Number.isFinite(f.k)||f.k<=0||f.k>100||!Number.isSafeInteger(f.seed)||!Number.isSafeInteger(f.bootstrapRuns)||f.bootstrapRuns<20||f.bootstrapRuns>300)throw Error("invalid-policy");return f;}
function components(f){const next=Object.fromEntries(f.models.map(x=>[x,new Set()]));for(const row of f.matches){next[row.winner].add(row.loser);next[row.loser].add(row.winner);}const seen=new Set(),out=[];for(const model of f.models){if(seen.has(model))continue;const stack=[model],part=[];seen.add(model);while(stack.length){const node=stack.pop();part.push(node);for(const peer of next[node])if(!seen.has(peer)){seen.add(peer);stack.push(peer);}}out.push(part.sort());}return out;}
function fitBT(f,matches=f.matches){const score=Object.fromEntries(f.models.map(x=>[x,0])),tolerance=1e-6;let maxUpdate=Infinity,iterations=0;for(let iter=0;iter<1000;iter++){const gradient=Object.fromEntries(f.models.map(x=>[x,0]));for(const row of matches){const p=1/(1+Math.exp(-(score[row.winner]-score[row.loser]))),weight=row.weight??1;gradient[row.winner]+=weight*(1-p);gradient[row.loser]-=weight*(1-p);}const rate=.06/Math.sqrt(1+iter/30),updates=Object.fromEntries(f.models.map(model=>[model,rate*(gradient[model]-.01*score[model])]));maxUpdate=Math.max(...Object.values(updates).map(Math.abs));for(const model of f.models)score[model]+=updates[model];const mean=f.models.reduce((sum,m)=>sum+score[m],0)/f.models.length;for(const model of f.models)score[model]-=mean;iterations=iter+1;if(maxUpdate<tolerance)break;}return{scores:score,converged:Number.isFinite(maxUpdate)&&maxUpdate<tolerance,iterations,tolerance,maxUpdate:Number(maxUpdate.toExponential(3))};}
function elo(f,matches=f.matches){const rating=Object.fromEntries(f.models.map(x=>[x,1000])),path=[];for(const row of matches){const expected=1/(1+10**((rating[row.loser]-rating[row.winner])/400)),delta=f.k*(row.weight??1)*(1-expected);rating[row.winner]+=delta;rating[row.loser]-=delta;path.push(Number(rating[f.models[0]].toFixed(4)));}return{rating,path};}
const ranks=(models,scores)=>Object.fromEntries([...models].sort((a,b)=>scores[b]-scores[a]||a.localeCompare(b)).map((m,i)=>[m,i+1]));
function run(input=builtIn,options={}){const f=validate(input),parts=components(f),digestMatches=[...f.matches].sort((a,b)=>stable(a).localeCompare(stable(b))),datasetDigest=hash({...f,matches:digestMatches}),groupCounts=Object.fromEntries([...new Set(f.matches.map(row=>row.group))].sort().map(group=>[group,f.matches.filter(row=>row.group===group).length])),forwardDigest=hash(f.matches),reverseDigest=hash([...f.matches].reverse()),order=options.order==="reverse"?[...f.matches].reverse():f.matches;if(parts.length>1)return{schema:"ranking-lab-v1",datasetDigest,promptGroupCounts:groupCounts,connectivity:{connected:false,components:parts},status:"ABSTAIN",claimBoundary:"Disconnected comparison components are not globally comparable."};const btFit=fitBT(f),bt=btFit.scores,eloA=elo(f,order),eloOther=elo(f,[...order].reverse()),random=rng(f.seed),samples=Object.fromEntries(f.models.map(m=>[m,[]]));if(!btFit.converged)return{schema:"ranking-lab-v1",datasetDigest,promptGroupCounts:groupCounts,connectivity:{connected:true,components:parts},tiePolicy:f.tiePolicy,k:f.k,seed:f.seed,bt:btFit,status:"ABSTAIN",claimBoundary:"The declared batch-fit tolerance was not reached, so this fixture is not ranked."};let skippedDisconnected=0,skippedUnconverged=0;for(let i=0;i<f.bootstrapRuns;i++){const rows=Array.from({length:f.matches.length},()=>f.matches[Math.floor(random()*f.matches.length)]);if(components({...f,matches:rows}).length>1){skippedDisconnected++;continue;}const fit=fitBT(f,rows);if(!fit.converged){skippedUnconverged++;continue;}const r=ranks(f.models,fit.scores);for(const model of f.models)samples[model].push(r[model]);}const accepted=f.bootstrapRuns-skippedDisconnected-skippedUnconverged;if(!accepted)return{schema:"ranking-lab-v1",datasetDigest,promptGroupCounts:groupCounts,connectivity:{connected:true,components:parts},tiePolicy:f.tiePolicy,k:f.k,seed:f.seed,bt:btFit,bootstrap:{requested:f.bootstrapRuns,accepted,skippedDisconnected,skippedUnconverged},status:"ABSTAIN",claimBoundary:"No connected, converged bootstrap resample remained, so uncertainty bands were not synthesized."};const bands=Object.fromEntries(f.models.map(m=>[m,{min:Math.min(...samples[m]),max:Math.max(...samples[m])}])),baseRank=ranks(f.models,bt),top=Object.keys(baseRank).sort((a,b)=>baseRank[a]-baseRank[b]),removals=f.matches.map((_,index)=>{const rows=f.matches.filter((__,i)=>i!==index),partsAfter=components({...f,matches:rows}),connected=partsAfter.length===1;if(!connected)return{index,connected,components:partsAfter,converged:false,topFlipped:null};const fit=fitBT(f,rows);if(!fit.converged)return{index,connected,converged:false,topFlipped:null};const rank=ranks(f.models,fit.scores);return{index,connected,converged:true,topFlipped:rank[top[0]]>rank[top[1]]};}),fragile=skippedDisconnected>0||skippedUnconverged>0||Object.values(bands).some(x=>x.max>x.min)||removals.some(x=>x.topFlipped===true||!x.connected||!x.converged),core={schema:"ranking-lab-v1",datasetDigest,promptGroupCounts:groupCounts,matchWeight:{minimum:MIN_WEIGHT,default:1,maximum:MAX_WEIGHT},connectivity:{connected:true,components:parts},tiePolicy:f.tiePolicy,tieEncoding:"two reciprocal weight-0.5 records",k:f.k,seed:f.seed,bt:{...btFit,ranks:baseRank},elo:{startingRating:1000,scale:400,selectedOrder:options.order||"forward",selectedOrderDigest:hash(order),forwardOrderDigest:forwardDigest,reverseOrderDigest:reverseDigest,scores:eloA.rating,path:eloA.path,reverseEndpoint:eloOther.rating},bootstrap:{requested:f.bootstrapRuns,accepted,skippedDisconnected,skippedUnconverged,bands},bootstrapBands:bands,removalStress:removals,status:fragile?"FRAGILE":"PASS",claimBoundary:"Synthetic pairwise diagnostics; not external validity, an unbiased preference population, or a universal model-quality ranking."};return{...core,receiptHash:hash(core)};}
let lastReceipt=null;function render(){const download=document.querySelector("#download");download.disabled=true;lastReceipt=null;try{const receipt=run(builtIn,{order:document.querySelector("#order").value}),summary=document.querySelector("#summary"),path=document.querySelector("#path");summary.innerHTML="<dt>Status</dt><dd>"+receipt.status+"</dd><dt>Connectivity</dt><dd>"+receipt.connectivity.components.length+" component</dd><dt>Tie policy</dt><dd>"+(receipt.tiePolicy||"not applicable")+"</dd>";path.replaceChildren(...((receipt.elo?.path)||[]).map(value=>{const el=document.createElement("span");el.className="step";el.style.height=Math.max(8,48+(value-1000)*1.5)+"px";el.title=String(value);return el;}));document.querySelector("#receipt").textContent=JSON.stringify(receipt,null,2);lastReceipt=receipt;download.disabled=false;}catch(error){document.querySelector("#receipt").textContent=JSON.stringify({schema:"ranking-lab-v1",status:"ABSTAIN",error:String(error.message)});}}function downloadReceipt(){if(!lastReceipt)return;const blob=new Blob([JSON.stringify(lastReceipt,null,2)],{type:"application/json"}),a=document.createElement("a");a.href=URL.createObjectURL(blob);a.download="bradley-terry-elo-ranking-receipt.json";a.click();setTimeout(()=>URL.revokeObjectURL(a.href),1000);}document.querySelector("#run").addEventListener("click",render);document.querySelector("#download").addEventListener("click",downloadReceipt);globalThis.JPJournalLab={run,validate,builtIn,stable,hash};render();})();</script></html>