LLM Inference Roofline: Find the Bottleneck

An LLM inference roofline makes arithmetic and byte movement argue on the same axes. Model prefill and decode separately before naming a bottleneck or buying an optimization.

An LLM inference roofline asks one bounded question

An LLM inference roofline estimates whether a declared prefill or decode workload is limited first by peak arithmetic, memory bandwidth, or the shape of work being offered. It is a transparent upper-bound model built from FLOPs, bytes, peak FLOP/s, and sustained bandwidth assumptions, not a benchmark result.

The original Roofline paper plots attainable performance against operational intensity. Its central ceiling is the lower of peak compute and bandwidth multiplied by FLOPs per byte, which turns an optimization conversation into an explicit claim about movement or arithmetic.

The browser lab uses editable synthetic values and labels every unit. It neither profiles a GPU nor predicts tokens per second; it gives reviewers a calculation they can replace with measured operator traffic and credible hardware ceilings.

A roofline starts with a workload unit. Prefill can be measured per prompt or token batch, while decode is naturally per generated token and active sequence; mixing those denominators creates an attractive point whose FLOPs and bytes never occurred in the same interval.

Separate prefill from decode before counting

Prefill processes a prompt’s tokens with parallel matrix work, while autoregressive decode advances one or a few new tokens and repeatedly touches model state plus the growing KV cache. Mixing the phases into one arithmetic intensity can hide the exact bottleneck an architecture decision is meant to address.

An LLM inference roofline therefore creates one point per phase and workload shape. Record batch size, prompt length, generated length, model dimensions, precision, attention method, cache layout, and concurrency, because each changes reuse or traffic even when the model name is constant.

The phase distinction also informs disaggregated LLM inference. Placement should follow measured needs and service objectives; a stylized compute-bound prefill point is not permission to split infrastructure without accounting for transfer and queueing.

Count useful operations and compulsory byte movement from an explicit model, then disclose exclusions. Kernel fusion, cache residency, quantization metadata, speculative paths, and parallel communication can move the observed system away from the teaching estimate without making the arithmetic internally inconsistent.

Count useful FLOPs with stated formulas

Choose a formula boundary that another engineer can reproduce. Dense projections are commonly approximated from matrix dimensions and batch-token counts, attention adds sequence-dependent work, normalization and elementwise operations contribute smaller terms, and mixture-of-experts layers require an explicit active-expert assumption.

The transformer inference optimization survey organizes algorithm, compiler, and hardware concerns that affect these counts. Cite the formula source or derive it beside the model; avoid copying a headline parameter count into FLOPs without declaring multiply-add convention and active paths.

For the teaching LLM inference roofline, FLOPs are direct inputs so no hidden architecture formula can masquerade as fact. That simplicity is useful during review: disagreement about FLOPs can be resolved by editing one number and watching the point move.

LLM arithmetic intensity is FLOPs divided by bytes moved at the chosen memory boundary. State whether bytes refer to high-bandwidth memory, a cache level, or interconnect traffic; one kernel may be compute-bound at HBM and bandwidth-bound across a slower device link.

Finally, keep queueing outside the arithmetic roof but inside the product decision. Measure arrival rate, service-time distribution, batching wait, preemption, and cancellation. If a modeled phase has abundant hardware headroom while users wait, scheduling or admission may be the actual constraint. The roofline has succeeded when it prevents an irrelevant kernel optimization, not only when it names compute or memory.

Build a byte-traffic ownership ledger

Bytes need more care than tensor size. Name weight reads, activation reads and writes, KV-cache history reads, current-token writes, temporary buffers, collective communication, host-device transfer, and output transfer, then state which layer of the memory hierarchy the bandwidth ceiling represents.

An LLM inference roofline that assumes every weight is fetched from high-bandwidth memory should say so; batch reuse or cache residency may reduce traffic, while spills and kernel boundaries may add it. KV-cache optimization deserves its own line because decode traffic grows with context and precision.

Do not reverse-engineer bytes from observed time and then use them to explain that same time. Start from an independent model or profiler counters, preserve uncertainty ranges, and mark uncounted traffic so the chart remains falsifiable.

Memory-bound decoding appears when the weight and key-value-cache traffic per token keeps intensity below the machine balance point. Larger batches can reuse weights across sequences and move the point right, but padding, scheduling gaps, and cache growth prevent batch size from being a free multiplier.

Runnable artifact — The calculator is a first-order model, not a benchmark or capacity promise. Hardware values are declared inputs and every optimization requires measurement on the target stack.

<!doctype html><html lang="en"><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>LLM inference roofline lab</title><style>:root{color-scheme:dark}*{box-sizing:border-box}body{font:16px/1.45 system-ui;background:#091318;color:#f4f7f6;max-width:980px;margin:auto;padding:24px}main{display:grid;gap:16px}fieldset,.panel{border:1px solid #8aa0aa;border-radius:12px;padding:14px}fieldset{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px}label{display:grid;gap:5px}button,input,select,a,textarea{font:inherit;padding:9px}button,a{min-height:44px}textarea{width:100%;min-height:210px;background:#071014;color:#f4f7f6}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));gap:12px}.status{padding:10px;border-left:5px solid #46e0c1;background:#10242b}svg,canvas{max-width:100%;height:auto}.sr{position:absolute;left:-9999px}@media(prefers-reduced-motion:reduce){*{animation:none!important;transition:none!important;scroll-behavior:auto!important}}#stage svg{width:100%;border:1px solid #8aa0aa}table{border-collapse:collapse;width:100%}th,td{padding:8px;border:1px solid #8aa0aa;text-align:left}</style><main><h1>Phase-aware inference roofline</h1><p>This first-order teaching model uses editable assumptions; it is not a hardware benchmark.</p><fieldset><legend>Current assumptions</legend><label>Peak TFLOP/s<input id="peak" type="number" min="1" value="312"></label><label>Bandwidth GB/s<input id="band" type="number" min="1" value="1555"></label><label>Prefill TFLOPs<input id="pf" type="number" min=".001" value="480"></label><label>Prefill traffic GB<input id="pb" type="number" min=".001" value="1600"></label><label>Decode TFLOPs<input id="df" type="number" min=".001" step=".1" value="1.2"></label><label>Decode traffic GB<input id="db" type="number" min=".001" value="240"></label></fieldset><p><button id="run">Recalculate current inputs</button> <a id="svgExport" download="roofline.svg">Export current SVG</a> <a id="jsonExport" download="roofline.json">Export current JSON</a></p><div id="stage" class="panel" aria-live="polite"></div><table id="table"><caption>Current computed roofline points</caption><thead><tr><th>Phase</th><th>Intensity FLOP/B</th><th>Ceiling TFLOP/s</th><th>Bound</th></tr></thead><tbody></tbody></table><textarea id="receipt" readonly aria-label="Execution receipt"></textarea></main><script>const byId=(id)=>document.getElementById(id),number=(id)=>Number(byId(id).value),sha=async(value)=>[...new Uint8Array(await crypto.subtle.digest('SHA-256',new TextEncoder().encode(value)))].map(v=>v.toString(16).padStart(2,'0')).join('');
function calculate(assumptions){const calc=(name,phase)=>{const intensity=phase.flops/phase.bytes,bandCeiling=intensity*assumptions.band/1000,ceiling=Math.min(assumptions.peak,bandCeiling),sensitivity=[.8,1,1.2].map(factor=>{const changedIntensity=phase.flops/(phase.bytes*factor);return{trafficFactor:factor,intensity:changedIntensity,ceiling:Math.min(assumptions.peak,changedIntensity*assumptions.band/1000)}});return{name,intensity,bandCeiling,ceiling,bound:bandCeiling<assumptions.peak?'memory bandwidth':'compute',sensitivity}};return[calc('prefill',assumptions.prefill),calc('decode',assumptions.decode)]}
function makeSvg(assumptions,rows){const width=860,height=430,left=72,right=826,top=48,bottom=360,knee=assumptions.peak*1000/assumptions.band,xMin=.5,xMax=Math.max(1000,knee*1.6,...rows.map(row=>row.intensity*1.6)),yMax=assumptions.peak*1.12,logMin=Math.log10(xMin),logMax=Math.log10(xMax),x=value=>left+(Math.log10(Math.max(xMin,value))-logMin)/(logMax-logMin)*(right-left),y=value=>bottom-value/yMax*(bottom-top),roofStart=[x(xMin),y(xMin*assumptions.band/1000)],roofKnee=[x(knee),y(assumptions.peak)],roofEnd=[x(xMax),y(assumptions.peak)],points=rows.map(row=>({name:row.name,x:x(row.intensity),y:y(row.ceiling),intensity:row.intensity,ceiling:row.ceiling}));const pointMarkup=points.map((point,index)=>'<circle data-phase="'+point.name+'" cx="'+point.x.toFixed(3)+'" cy="'+point.y.toFixed(3)+'" r="12" fill="'+(index?'#d4ff72':'#ff8d76')+'"/><text x="'+(point.x+16).toFixed(3)+'" y="'+(point.y+6).toFixed(3)+'">'+point.name+'</text>').join('');const svg='<svg xmlns="http://www.w3.org/2000/svg" width="860" height="430" viewBox="0 0 860 430" role="img" aria-labelledby="roof-title roof-desc"><title id="roof-title">Current inference roofline</title><desc id="roof-desc">Two current phase points, bandwidth slope, compute roof, and computed knee.</desc><rect width="860" height="430" fill="#10242b"/><g stroke="#8aa0aa" fill="none"><path d="M'+left+' '+top+'V'+bottom+'H'+right+'"/><path d="M'+roofStart[0].toFixed(3)+' '+roofStart[1].toFixed(3)+'L'+roofKnee[0].toFixed(3)+' '+roofKnee[1].toFixed(3)+'L'+roofEnd[0].toFixed(3)+' '+roofEnd[1].toFixed(3)+'" stroke="#46e0c1" stroke-width="7"/></g><g fill="white" font-family="system-ui" font-size="16">'+pointMarkup+'<text x="74" y="402">log arithmetic intensity (FLOP/B); knee '+knee.toFixed(3)+'</text><text x="80" y="70">current roof '+assumptions.peak.toFixed(3)+' TFLOP/s</text></g></svg>';return{svg,plot:{bounds:{xMin,xMax,yMax},knee,roof:{start:roofStart,knee:roofKnee,end:roofEnd},points}}}
async function execute(){try{const assumptions={peak:number('peak'),band:number('band'),prefill:{flops:number('pf')*1e12,bytes:number('pb')*1e9},decode:{flops:number('df')*1e12,bytes:number('db')*1e9}};for(const value of[assumptions.peak,assumptions.band,assumptions.prefill.flops,assumptions.prefill.bytes,assumptions.decode.flops,assumptions.decode.bytes])if(!Number.isFinite(value)||value<=0)throw Error('inputs must be finite and positive');const rows=calculate(assumptions),rendered=makeSvg(assumptions,rows),svgSha256=await sha(rendered.svg),data={model:'first-order roofline teaching calculation',units:{flops:'FLOP',bytes:'byte',peak:'TFLOP/s',bandwidth:'GB/s'},assumptions,rows,plot:rendered.plot,svgSha256,invariants:{positive:true,twoCurrentPoints:rendered.plot.points.length===2,ceilingNeverExceedsPeak:rows.every(row=>row.ceiling<=assumptions.peak),coordinatesFinite:rendered.plot.points.every(point=>Number.isFinite(point.x)&&Number.isFinite(point.y))}};if(!Object.values(data.invariants).every(Boolean))throw Error('roofline invariant failed');byId('stage').innerHTML=rendered.svg;byId('table').tBodies[0].innerHTML=rows.map(row=>'<tr><th>'+row.name+'</th><td>'+row.intensity.toFixed(3)+'</td><td>'+row.ceiling.toFixed(3)+'</td><td>'+row.bound+'</td></tr>').join('');byId('svgExport').href=URL.createObjectURL(new Blob([rendered.svg],{type:'image/svg+xml'}));byId('jsonExport').href=URL.createObjectURL(new Blob([JSON.stringify(data,null,2)],{type:'application/json'}));byId('receipt').dataset.execution=JSON.stringify(data);byId('receipt').value='PASS: '+JSON.stringify(data,null,2)}catch(error){byId('receipt').dataset.execution=JSON.stringify({unexpectedError:error.name+': '+error.message});byId('receipt').value='FAIL: unexpected '+error.message}}byId('run').onclick=()=>void execute();void execute();</script></html>

Calculate the two ceilings consistently

Operational intensity is FLOPs divided by bytes. The bandwidth ceiling equals intensity times bandwidth after unit conversion, the compute ceiling is declared peak or a defensible sustained ceiling, and attainable performance is their minimum; the ridge point occurs where those lines meet.

The lab asserts that no point exceeds peak compute and exports formulas, inputs, units, results, plot SVG, and a hash. Its example gives prefill high intensity and decode low intensity solely to illustrate the geometry, not to characterize a particular accelerator.

An LLM inference roofline should show uncertainty as bands when traffic or sustained bandwidth is approximate. A point near the ridge cannot support a categorical label without sensitivity analysis, because modest changes in batching, precision, or fusion can cross the boundary.

The GPU roofline model supplies two ceilings: peak compute and bandwidth times arithmetic intensity. The smaller ceiling is the optimistic bound; multiplying it by a guessed utilization factor may help planning, but that factor must be labeled as an assumption rather than a measured hardware capability.

Use sensitivity to choose the next experiment

Vary one assumption at a time: batch size for weight reuse, context length for KV traffic, precision for storage, fused kernels for intermediate movement, peak arithmetic, sustained bandwidth, and interconnect transfer. The slopes reveal which measurement could change the decision instead of encouraging a generic optimization list.

If decode remains bandwidth-limited across plausible inputs, test cache layout, quantization, batching, or memory locality. If prefill presses the compute roof, examine kernel efficiency and FlashAttention’s I/O-aware approach while checking whether the modeled attention bytes match the implementation.

Sensitivity does not prove the proposed change will help latency. It produces a ranked hypothesis that still needs an operator profile and end-to-end workload measurement under the same request mix.

Prefill reuses weights across many token operations and often reaches higher intensity, yet attention intermediates and sequence length reshape its byte ledger. Keep prefill and decode on separate traces so an averaged request point does not hide opposite bottlenecks that require different optimizations.

Model communication explicitly when a phase crosses devices. Tensor parallel collectives, pipeline bubbles, expert routing, host staging, and prefill-to-decode KV transfer each introduce bytes and latency on a different interconnect. Add a separate roof or transfer term rather than folding network time into HBM bandwidth. A disaggregated design can improve utilization while creating a new bottleneck whose units the original single-device chart cannot express.

Compare roofline position with profiler evidence

Collect achieved FLOP/s, relevant memory bandwidth, kernel duration, occupancy, cache behavior, and communication from a supported profiler. Map counters to the same phase and byte boundary used by the model, then explain gaps between the theoretical ceiling, an empirical machine ceiling, and achieved performance.

An LLM inference roofline is most honest when measured points fall below both roofs and the unused headroom remains visible. Kernel launch overhead, dependencies, small matrices, synchronization, fragmentation, scheduling, and service batching can keep work far from either headline limit.

The prefill/decode-aware evaluation paper reinforces phase-sensitive evaluation. Treat new research as evidence to inspect and reproduce, not a source of universal constants for a different system.

A quantization proposal changes both axes. Lower precision reduces bytes and may change supported compute throughput, while dequantization, scale loads, and kernel availability add costs; update the entire worksheet instead of sliding the old point horizontally and declaring the new bottleneck solved.

Connect hardware limits to service objectives

Tokens per second, time to first token, inter-token latency, throughput, p95 completion, and cost are service metrics, while the roofline describes a compute kernel or phase boundary. Build a conversion ledger that names useful tokens, batch occupancy, padding, speculative work, queue delay, and non-model overhead before projecting product impact.

Continuous batching can improve utilization while worsening tail latency for some workloads. Plot several arrival-rate and batch scenarios rather than using maximum throughput as the only operating point.

An LLM inference roofline should inform, not replace, admission control and capacity planning. A memory-bound phase can still be economically acceptable, and a compute-bound phase can still miss latency because requests wait before execution.

Use an LLM inference roofline to reject impossible explanations before profiling deeper. If the predicted bound is far above observation, scheduling or kernel efficiency deserves investigation; if observation approaches the bound, reducing the constrained resource has a clearer route to impact.

Avoid false precision and universal bottlenecks

Round estimates according to their inputs and display assumptions beside the plot. Peak vendor specifications, effective bandwidth, quantization packing, sparsity, cache residency, and operator fusion all carry qualifications; four decimal places cannot convert uncertain traffic into measurement.

Never publish “decode is memory-bound” without the modeled model, context, batch, hardware, precision, and software path. The same phase can move as those dimensions change, and transfer between disaggregated stages can introduce a ceiling absent from a single-device diagram.

The artifact deliberately calls itself a first-order calculator. It rejects nonpositive inputs, preserves units, distinguishes phases, and exports a reproducible SVG, giving future measurements a stable worksheet rather than an authoritative benchmark label.

This artifact is a deterministic capacity worksheet, not a benchmark. Its hardware values are labeled fixture assumptions, its phase receipts reproduce exactly, and no throughput number should be presented as a claim about an unmeasured GPU, provider, compiler, or production serving stack.

Ship a phase-aware performance worksheet

Preserve model revision, workload histogram, formulas, FLOP convention, byte owners, memory level, hardware and software versions, peak and sustained ceilings, phase points, sensitivity ranges, profiler captures, service metrics, and decision. Link every edited input to its evidence source.

Run the browser worksheet with the included synthetic values, export its JSON, then substitute one measured deployment trace. If the profiler and model disagree, investigate the boundary before buying hardware or redesigning serving; the disagreement is often more useful than the original label.

Revisit the LLM inference roofline when workload shape, model architecture, precision, kernel implementation, cache policy, or accelerator changes. Its value is a durable habit: count arithmetic and movement separately, then test the bottleneck the counts actually predict.

Refresh the plot when architecture, precision, batching, cache layout, or hardware ceilings change. The two-phase postcard is useful distribution material only if it retains units and fixture labels, then directs readers to the full FLOP-and-byte ledger for an auditable decision.

Validate sustained ceilings with a microbenchmark that resembles the access pattern, precision, and concurrency under study. Vendor peaks are useful orientation, but ECC, clocks, thermal state, power limits, kernel shape, and concurrent tenants affect usable rates. Preserve the microbenchmark command, warm-up, sample distribution, and device configuration; never tune the ceiling backward until it explains application time.