Federated vs Centralized Fine-Tuning

Federated vs centralized fine-tuning starts with whether raw examples are allowed to move. This guide holds the objective constant, then compares skew, cohort quality, communication, privacy controls, and operations.

Federated vs centralized fine-tuning begins with custody

The deciding question is whether raw examples are allowed to cross the boundary where they were produced. Centralized model training copies examples into one controlled training environment. Federated learning sends a model or training program to participating clients and aggregates their updates. Federated vs centralized fine-tuning therefore starts with a data-flow diagram, not an accuracy chart.

Write down the protected unit: a row, conversation, device history, institution cohort, or regional dataset. Then name who controls each location, how long examples remain, and whether deletion can be honored. Federation is justified when those constraints forbid or seriously burden central collection. If data can move safely and lawfully, centralization usually offers simpler debugging, batching, and evaluation.

Updates are not harmless substitutes for examples. Gradients or parameter deltas can reveal information under some threat models, while participation metadata can expose client activity. The federated path changes custody; it does not erase privacy work. The first figure keeps raw examples, local updates, aggregated updates, checkpoints, and evaluation data on separate lanes so “the data stays local” cannot become an imprecise blanket claim.

Define why examples cannot move

A credible constraint is specific. A hospital may prohibit patient-level export; an on-device keyboard may avoid collecting personal text; separate companies may cooperate without pooling records. “Privacy” by itself is too vague to select an architecture. Identify the rule, contractual promise, latency need, or trust boundary that prevents a central dataset, and document exceptions such as opt-in evaluation samples.

Federated vs centralized fine-tuning also changes deletion and incident response. Central training needs lineage from collected examples into datasets and checkpoints. Federated training needs lineage from client eligibility, local software, round participation, update aggregation, and retained checkpoints. Neither provides perfect unlearning merely by choosing the architecture.

If the reason is a formal privacy target, budget differential privacy separately from federation. Clipping, noise, adjacency, privacy accounting, and protected-unit definitions are additional mechanisms. If the reason is lower bandwidth, measure the actual update traffic; repeated model transfers can exceed a compact upload of curated examples. A useful custody decision contains the constraint and the cost it creates.

Hold the learning objective constant

Comparisons become meaningless when the model, initialization, preprocessing, loss, or evaluation split changes with the training topology. Freeze those elements first. A centralized baseline should concatenate the same generated client examples, use the same linear model, begin from the same seeded weights, and optimize the same objective. The federated path may differ in batching and aggregation because those are the mechanisms under study.

The federated vs centralized fine-tuning artifact uses a small synthetic classifier so every update is inspectable. It is not a language-model benchmark. Central and federated runs share data, initialization, learning rate, and logistic loss. The receipt identifies the seed and prints per-client metrics, preventing a convenient global average from hiding a weak cohort.

Freeze one evaluation contract too. Use the same measurement contract for both pipelines: dataset version, cohort definitions, metrics, uncertainty method, and release thresholds. A central baseline is not a straw man; give it reasonable optimization and operating conditions. The purpose is to price locality honestly, not to make the preferred architecture win.

Model FedAvg on non-IID clients

Federated Averaging selects clients, lets each perform local optimization, then averages updates weighted by their example counts. The original FedAvg paper studies the communication setting and non-identically distributed data that makes this useful—and difficult. Weighting prevents a tiny client from counting like a giant one, but it does not guarantee fairness across cohorts.

Non-IID data means clients do not resemble random slices of one common distribution. One region may contain different vocabulary, one device tier different behavior, or one institution a rare class. A FedAvg non-IID data test needs an IID control under the same objective. More local epochs reduce communication but allow each client model to drift farther toward its own distribution. Federated vs centralized fine-tuning should therefore report both global quality and spread across clients.

The round-braid figure makes sample weight and dropout visible at every checkpoint. A missing strand means a client did not participate; it does not mean its users vanished from the product. Track which cohorts repeatedly miss selection or fail to finish. A stable aggregate can coexist with a deteriorating minority cohort, so the release gate needs worst-client or named-cohort thresholds beside the mean.

Price communication, participation, and dropout

A federated round moves model state or an equivalent training package outward and updates inward. Multiply bytes per transfer by selected clients and rounds, then include retries, protocol overhead, and checkpoint distribution. The result is an architecture estimate, not merely “number of rounds.” Centralized training moves examples once or repeatedly through an ingestion pipeline and then pays internal storage and compute traffic.

Participation is a sampling process constrained by availability, power, network, eligibility, and policy. Dropout after selection wastes capacity and can interact with aggregation protocols. Federated vs centralized fine-tuning should record invited, accepted, completed, rejected, and aggregated clients. Stragglers need deadlines and a recovery policy; silently waiting forever is not coordination.

The synthetic lab varies participation and deterministic dropout patterns, then records communication bytes and per-round contributors. Those quantities are generated by the fixture, not field measurements. Use them to verify accounting and explore shape. Replace them with a product-specific envelope before choosing. If one client cannot reliably finish a local epoch within the window, reducing model size, local work, or cohort scope may matter more than tuning the aggregator.

Separate federation from privacy controls

Federation limits routine raw-example centralization, yet the server still observes updates unless another mechanism protects them. Secure aggregation lets a server recover an aggregate without reading each surviving client contribution in the clear under its stated protocol assumptions. The secure aggregation paper also treats client dropout as a core systems concern.

Secure aggregation does not prevent everything. The aggregate may leak information, the final model may memorize, compromised clients may poison updates, and metadata may remain visible. Differential privacy can bound an explicitly defined leakage channel, while authentication, attestation, anomaly controls, and governance address other threats. Federated vs centralized fine-tuning needs a threat table rather than one “private” badge.

The artifact records secure aggregation as an explicitly unimplemented architecture boundary: individual fixture updates stay visible so weighting and norm rejection can be audited, and its receipt says that no cryptography ran. That honesty matters: a toy masking trick would create false assurance. For an implementation vocabulary, the TensorFlow Federated guide separates federated data, client computation, and server aggregation. Production selection should add a reviewed protocol, key lifecycle, dropout threshold, and incident plan.

Evaluate the global model and client cohorts

Global loss answers whether the population-weighted objective improved. It does not answer whether every cohort improved, whether calibration changed, or whether the model became unsafe for a rare group. Report per-client or policy-defined cohort metrics with sample counts. Include the worst cohort, a lower percentile, and spread rather than publishing only a league table of averages.

For federated vs centralized fine-tuning, compare checkpoints on one frozen evaluation corpus when policy allows, then supplement it with local evaluation for data that cannot move. The aggregation server should receive only the evaluation statistics allowed by the privacy design. Small cohorts may require suppression or uncertainty intervals. Report uncertainty instead of one winning average so random client selection is not mistaken for architecture quality.

Also evaluate operations: round completion, dropout, update norms, communication, rollback, and reproducibility. A model that reaches a similar mean while failing unpredictable cohorts may be unacceptable. A central model that scores well but violates custody is equally unacceptable. The decision combines hard governance constraints with a quality and operations envelope.

Run the seeded federation lab

The Node artifact generates four client cohorts from a pinned pseudo-random seed. It trains the same two-feature linear classifier with a bias centrally and with weighted FedAvg under IID and skewed assignments. Scenarios vary local epochs, participation, and dropout. A hand-calculated aggregation fixture confirms that sample weights are applied correctly, while repeated runs verify identical output hashes.

Read the receipt in this order: data and initialization hash, scenario parameters, round contributors, global loss, client losses, spread, and bytes. The federated vs centralized fine-tuning numbers are synthetic optimization outputs. They do not predict a real model, prove privacy, or benchmark a framework. A labeled poisoned-update case is rejected by a norm policy to demonstrate the boundary, not to claim robust aggregation.

Change skew before changing rounds. Then change local epochs while holding participation fixed. That sequencing reveals which assumption moves the result. If the experiment only exposes a final score, extend it until every client and round is visible. Distill behavior only after choosing the training custody model; otherwise compression can obscure which topology created the behavior.

Runnable artifact — A small synthetic optimization simulator, not a privacy proof, production FL framework benchmark, or claim about a real model's quality.

import assert from "node:assert/strict";
import { createHash } from "node:crypto";
const hash=(value)=>createHash("sha256").update(JSON.stringify(value)).digest("hex");
function rng(seed){let s=seed>>>0;return()=>{s^=s<<13;s^=s>>>17;s^=s<<5;return(s>>>0)/4294967296}}
const sigmoid=(z)=>1/(1+Math.exp(-z));
function makeClients(seed=904,skew=.82){const random=rng(seed);return Array.from({length:4},(_,client)=>{const n=40+client*8;return Array.from({length:n},(_,i)=>{const cohort=client<2?-1:1;const x1=(random()*2-1)+cohort*skew;const x2=(random()*2-1)-cohort*skew*.35;const p=sigmoid(1.4*x1-.9*x2+.15);return{x:[x1,x2,1],y:random()<p?1:0,id:client+":"+i}})})}
const loss=(w,rows)=>rows.reduce((sum,row)=>{const p=Math.min(.999999,Math.max(.000001,sigmoid(w.reduce((s,v,i)=>s+v*row.x[i],0))));return sum-(row.y*Math.log(p)+(1-row.y)*Math.log(1-p))},0)/rows.length;
function step(w,rows,rate=.18){const g=[0,0,0];for(const row of rows){const d=sigmoid(w.reduce((s,v,i)=>s+v*row.x[i],0))-row.y;g.forEach((_,i)=>g[i]+=d*row.x[i]/rows.length)}return w.map((v,i)=>v-rate*g[i])}
function central(clients,rounds=12){let w=[0,0,0],rows=clients.flat();for(let r=0;r<rounds;r++)w=step(w,rows);return w}
const l2=(values)=>Math.hypot(...values);
function fedavg(clients,{rounds=12,localEpochs=2,participation=1,drop=[],poison=null,normThreshold=Infinity}={}){let global=[0,0,0],trace=[],bytes=0,rejections=[];for(let round=0;round<rounds;round++){const count=Math.max(1,Math.ceil(clients.length*participation)),participants=clients.map((_,i)=>(i+round)%clients.length).slice(0,count).filter(i=>!drop.includes((round+":"+i))).sort((a,b)=>a-b);const proposed=participants.map(i=>{let w=[...global];for(let e=0;e<localEpochs;e++)w=step(w,clients[i]);const update=w.map((value,j)=>value-global[j]);if(poison&&poison.round===round&&poison.client===i)update[0]*=poison.multiplier;const updateNorm=l2(update),accepted=updateNorm<=normThreshold;if(!accepted)rejections.push({round,client:i,reason:"update-norm-exceeds-threshold",updateNorm,normThreshold});return{i,n:clients[i].length,w:global.map((value,j)=>value+update[j]),updateNorm,accepted}});const local=proposed.filter(row=>row.accepted),total=local.reduce((s,x)=>s+x.n,0);if(local.length)global=global.map((_,j)=>local.reduce((s,x)=>s+x.w[j]*x.n/total,0));bytes+=participants.length*global.length*8*2;trace.push({round,participants,acceptedClients:local.map(row=>row.i),rejectedClients:proposed.filter(row=>!row.accepted).map(row=>row.i),weights:local.map(x=>x.n/total),updateNorms:proposed.map(row=>({client:row.i,norm:row.updateNorm,accepted:row.accepted})),checkpoint:[...global]})}return{w:global,trace,bytes,rejections}}
const clients=makeClients(),initial=[0,0,0],initializationHash=hash(initial),centralized=central(clients),federated=fedavg(clients,{drop:["2:1","5:3"]});
const metrics=(w)=>{const clientLosses=clients.map(rows=>loss(w,rows));return{globalLoss:loss(w,clients.flat()),clientLosses,clientLossSpread:Math.max(...clientLosses)-Math.min(...clientLosses)}};
const handClients=[[{x:[1,0,1],y:1}],[{x:[-1,0,1],y:0},{x:[-1,0,1],y:0},{x:[-1,0,1],y:0}]],hand=fedavg(handClients,{rounds:1,localEpochs:1}),locals=handClients.map(rows=>step(initial,rows)),expected=initial.map((_,j)=>locals[0][j]*.25+locals[1][j]*.75);expected.forEach((v,i)=>assert.ok(Math.abs(v-hand.w[i])<1e-12));
const repeated=fedavg(makeClients(),{drop:["2:1","5:3"]});assert.deepEqual(repeated,federated);
const poisonPolicy={name:"l2-update-norm-v1",normThreshold:1,fixture:{round:0,client:0,multiplier:100}},poisoned=fedavg(clients,{rounds:1,localEpochs:2,poison:poisonPolicy.fixture,normThreshold:poisonPolicy.normThreshold});assert.deepEqual(poisoned.trace[0].rejectedClients,[0]);assert.equal(poisoned.rejections[0].reason,"update-norm-exceeds-threshold");
const iid=makeClients(904,0),iidFed=fedavg(iid),summarize=(run)=>({weights:run.w,...metrics(run.w),bytes:run.bytes,participantCounts:run.trace.map(row=>row.participants.length)}),sweeps={localEpochs:[1,2,4].map(value=>({value,...summarize(fedavg(clients,{localEpochs:value}))})),participation:[.5,.75,1].map(value=>({value,...summarize(fedavg(clients,{participation:value}))})),dropout:[{name:'none',drop:[]},{name:'bounded',drop:['2:1','5:3']},{name:'repeated-client-2',drop:['1:2','4:2','7:2']}].map(scenario=>({name:scenario.name,...summarize(fedavg(clients,{drop:scenario.drop}))}))};
const secureAggregationBoundary={implemented:false,cryptographyExecuted:false,visibleUpdatePolicyOnly:true,note:"This deterministic fixture exposes individual updates to test weighting and norm rejection; it does not execute secure aggregation."};
const receipt={fixture:"seeded-linear-federation-v2",seed:904,synthetic:true,model:"two-feature linear classifier with bias",privacyProof:false,secureAggregationImplemented:false,secureAggregationBoundary,initialization:initial,initializationHash,dataHash:hash(clients),poisonPolicy,poisonedUpdate:{rejections:poisoned.rejections,round:poisoned.trace[0]},scenarios:{centralized:{weights:centralized,...metrics(centralized)},federated:{weights:federated.w,...metrics(federated.w),bytes:federated.bytes,rounds:federated.trace},iidFederated:{...metrics(iidFed.w),bytes:iidFed.bytes}},sweeps,handAggregation:{weights:hand.w,expected,pass:true}};
assert.ok(receipt.scenarios.federated.rounds.every(row=>row.weights.reduce((a,b)=>a+b,0)>.999999));
console.log(JSON.stringify({...receipt,receiptHash:hash(receipt)},null,2));console.log("PASS: matched synthetic objective, weighted FedAvg, poison rejection, IID/skew cohorts, dropout, communication, and per-client loss spread reconcile");

Use a frontier instead of a winner table

The decision frontier plots six concerns without collapsing them into a fake universal score: raw-data custody, worst-cohort quality, communication, privacy-control burden, operational ownership, and evaluation observability. Centralization usually offers stronger observability and simpler control. Federation can satisfy a non-movable-data requirement, but adds distributed software, sampling, update protection, and recovery work.

Federated vs centralized fine-tuning becomes a defensible choice when one option satisfies every hard constraint and its remaining costs fit a declared envelope. If both fit, prototype the simpler centralized path unless locality provides a concrete product or governance benefit. If only federation satisfies custody, fund its security and operations as first-class product work.

Avoid weighting every axis into one number unless stakeholders have agreed on the weights and veto conditions. A hard residency rule cannot be traded away by a small loss improvement. Likewise, a severe worst-cohort regression cannot be hidden by lower communication. Keep the frontier and the raw receipt together so decision-makers can challenge assumptions rather than debate a mysterious aggregate.

Choose the pipeline by evidence

Select centralized training when examples may move into a governed environment and the team values simple debugging, stable batching, and complete evaluation. Select federation when raw-example locality is a hard requirement or a meaningful trust promise, and the organization can operate client selection, versioning, update protection, cohort evaluation, dropout recovery, and rollback.

The release decision for federated vs centralized fine-tuning should include the custody map, matched objective, product-shaped client distribution, quality spread, communication estimate, threat model, and ownership roster. Schedule a revisit when frameworks, secure aggregation protocols, policy, or client distributions change. Distribution drift can reverse a result even when implementation is untouched.

Take one action: replay one product-shaped client distribution before choosing federated training. Replace the lab’s generated cohorts with non-sensitive statistics that reflect actual sizes and skew, keep the same central control, and publish both aggregate and cohort receipts. If locality is not a hard constraint and the extra system has no measured benefit, choose the smaller operating surface. If locality is mandatory, the receipt becomes the beginning of the engineering plan.