LLM Admission Control for Stable Latency

LLM admission control decides whether a request should enter GPU service, wait with an honest promise, or be rejected while retry still has value. Without that boundary, overload becomes a growing queue where every user waits longer, cancellations waste work, and the slowest requests collapse together.

This guide derives a token-aware policy, exercises it in a small discrete-event simulator, and specifies the waiting and rejection states. It is for serving and product teams protecting predictable response time when demand exceeds finite accelerator capacity.

The scheduler joins continuous batching, a token budget, queue fairness, and tail latency in one policy so overload cannot be displaced onto an invisible cohort.

LLM admission control makes overload explicit

LLM admission control exists because accelerators have finite memory and compute while request cost varies by prompt length, generated length, model, adapter, and decoding features. Accepting every arrival does not remove scarcity; it converts scarcity into unbounded wait and cancellation waste.

Separate admission from execution scheduling. Admission decides whether the service can make a useful promise under current policy. Scheduling orders accepted work and may combine requests. The two share estimates, but a sophisticated scheduler still needs a boundary when offered load exceeds recoverable capacity.

My position is that a fast explicit rejection is kinder than a timeout disguised as progress. The boundary is product-specific: interactive work needs a short start deadline, while asynchronous analysis can accept a durable queue position. Both must tell the truth about current state.

Define protected service objectives per journey: predicted start, first useful output, completion, and successful cancellation. Then decide how much work may be queued before those objectives become implausible. Queue length alone is inadequate because one long prompt can cost more than many short ones.

Estimate work in tokens and memory

LLM admission control needs an upper-bound estimate for prompt processing, expected generation, cache state, model footprint, adapter overhead, and special features. User-supplied maximum output is useful but often pessimistic; a workload-specific percentile can guide planning while a hard cap protects capacity.

Record estimated and actual costs for every completed request. Calibrate by model and workload class, then track underestimation separately from overestimation. Underestimation threatens service objectives; excessive overestimation wastes capacity by rejecting work that would have fit.

Treat memory and compute as separate budgets. A request may fit arithmetic capacity but exceed key-value memory, or fit memory while saturating prompt processing. A two-dimensional feasibility check is clearer than compressing incompatible resources into one unexplained score.

Unknown or malformed inputs receive a conservative estimate or a preflight rejection before GPU allocation. Do not ask the model to estimate its own resource need. The serving layer can count input tokens, apply configured bounds, and identify features deterministically.

Illustrative TypeScript — a reviewable design sketch, not a compiled production implementation.

type AdmissionRequest = {
  promptTokens: number;
  outputUpperBound: number;
  memoryPages: number;
  workloadClass: "interactive" | "batch" | "background";
  deadlineMs: number;
};

Signal	Decision	Proof
Capacity inside budget	Admit	Predicted start meets SLO
Short recoverable delay	Defer	Queue position and expiry shown
Budget cannot recover	Reject	Retry guidance is bounded

Simulate an overload burst

LLM admission control can be tested with a discrete-event fixture containing arrival time, prompt tokens, output estimate, actual output, class, deadline, and cancellation. Model a finite prompt and decode capacity, update active work each tick, and preserve every decision with its predicted start.

The worked burst sends ten short interactive requests, two long background reports, and one oversized prompt into a service already at seventy percent capacity. The policy admits the short requests, defers one report, preserves one background slot, and rejects the oversized request before allocation.

Compare three policies: admit all, first-come first-served with a fixed count, and token-aware admission with aging. The inspectable output reports start and completion distributions per class, rejected work, wasted work after cancellation, and maximum queue age. No invented production numbers are required.

Vary prediction error and arrival burstiness. A useful policy degrades gradually: guarded reserves shrink throughput slightly during calm periods but prevent interactive collapse during bursts. Save the event trace and decision ledger so a surprising schedule can be replayed exactly.

Balance fairness, age, and cost

LLM admission control must prevent cheap work from starving long requests and prevent one account from occupying every slot. Use per-class reserves, per-tenant concurrency caps, and aging that increases priority while remaining subject to a maximum feasible cost and deadline.

Shortest-estimated-job-first can reduce average completion time but punish large legitimate tasks. Pure first-come order protects age but lets one large request block a busy interactive lane. Weighted fair queues with explicit classes make the tradeoff reviewable and provide a place for product commitments.

Never infer priority from prompt wording or a model's assessment of importance. Priority comes from authenticated plan, product journey, durable job class, or operator policy. Otherwise an injected document can claim urgency and steal protected capacity.

Measure service by cohort, including the rejected cohort. A low latency chart computed only from admitted winners can improve while the product becomes unusable. Report admission rate, predicted wait accuracy, deadline success, and completion outcome for every class and tenant band.

Design waiting as a real product state

LLM admission control should return a durable request ID when delay is acceptable. The interface can show queued, estimated start range, changing position band, cancel, and notification options. Avoid a precise countdown when service time is stochastic; explain the estimate and update it at bounded intervals.

Interactive requests should not sit behind a spinner past their useful deadline. Offer retry, smaller scope, switch to an asynchronous path, or cancel. Choose one primary recovery action for the context. Do not simulate streaming text before the request has actually entered generation.

Cancellation must remove queued work immediately and signal active work promptly. Measure cancellation success and capacity released. A cancel button that only hides the UI makes overload worse while teaching users to submit duplicates.

Use semantic status text and restrained live-region announcements. Queue movement every second is not meaningful to a screen-reader user or a visual reader. Announce state transitions such as queued, started, needs action, and completed; keep core status readable without client-side animation.

1. EstimateEstimate

   Bound prompt and requested generation work.

2. BudgetBudget

   Read live memory, queued work, and protected reserves.

3. ClassifyClassify

   Apply priority, age, fairness, and deadline.

4. RespondRespond

   Admit, defer with status, or reject with safe retry.

Shed work with bounded retry guidance

LLM admission control rejects when no credible schedule meets the request's deadline or when safety reserves are exhausted. Return a stable reason category and retry guidance only when capacity is expected to recover. Random immediate retry instructions amplify a burst into a retry storm.

Use server-provided backoff with jitter, idempotent request identity, and a retry budget. Clients must stop after the budget or move to a durable asynchronous path. Preserve prior queue age when appropriate so reconnecting clients do not lose fairness through a transient network failure.

A named failure mode is synchronized retries after a regional capacity event. The consequence is repeated overload even after recovery begins. Mitigate it with randomized retry windows, admission tokens or leases for large jobs, and circuit-breaker behavior at callers.

Do not return an overload response after beginning a consequential tool effect. Admission belongs before expensive generation and effects. Once external work starts, the workflow needs durable execution and reconciliation rather than a generic request retry.

Observe policy and scheduler together

LLM admission control receipts should contain policy version, estimates, live budget band, workload class, tenant band, predicted start, decision, reason, and later actual cost. Join them with scheduler traces to discover whether a correct admission prediction was undermined by execution behavior.

The vLLM paper explains the memory-management approach behind a widely used serving engine. Version-specific vLLM scheduler controls document available knobs; validate the deployed server rather than treating this article as flag documentation.

Dashboard arrival rate, admitted work units, active work, queued work units, estimated error, queue-age distribution, deadline misses, cancellation waste, rejection rate, and retry return rate. Break them down by class so average health cannot hide starvation.

Alert on protected-objective breach and estimator drift, not every rejection. Rejection can be the policy working correctly during overload. The incident begins when the system cannot honor its declared outcomes or when the policy distributes scarcity contrary to its fairness contract.

Roll out from shadow decisions to enforcement

LLM admission control should begin by logging what the new policy would decide while the current system continues. Replay historical bursts through the simulator, compare shadow predictions with actual starts, and calibrate cost estimates. Then enforce only on one low-consequence class.

Use a safety ladder: cap oversized inputs, bound queue work, reserve an interactive lane, add fair sharing, then tune cost-aware ordering. Each step has an observable reason and rollback. Launching a clever unified score first makes causal diagnosis much harder.

Release gates include bounded queue work, protected cohort deadlines inside budget, successful cancellation, no starvation in long-run fixtures, stable retry return rate, and an operator switch to conservative fixed caps. Rehearse the switch under load before calling it a rollback plan.

Preserve the simulator, burst traces, policy file, UI state specimen, shadow comparison, and final decision receipt. Those artifacts align engineering and design around one claim: scarcity is handled deliberately before the product asks users to wait.

Put the method into practice

LLM admission control protects service quality by estimating variable work, comparing it with live capacity and deadlines, and making an explicit admit, defer, or reject decision before expensive execution. Fairness and truthful UI are part of the policy, not downstream polish.

Start with the thirteen-request overload fixture. Compare admit-all with token-aware enforcement, inspect every cohort, and keep the policy only if it protects interactive deadlines without starving durable work or provoking a retry storm.

The narrow method connects to four existing Journal notes: backpressure and flow control, SLOs that follow user journeys, LLM routing by cost, risk, and latency, AI streaming UX without jitter. They cover adjacent implementation boundaries without changing this article's single search intent. Preserve the worked fixture, its visual evidence, and the release receipt so a later update can compare behavior instead of relying on memory.