Inference Gateway Routing by Live Model Load

Inference gateway routing fails when equal pod weights hide unequal model work. A replica with a warm prefix and resident adapter may beat an idle-looking replica that must load both, while a short queue can still conceal one enormous decode.

This guide builds a live scorer from queue cost, cache affinity, adapter residency, health, and fairness, then tests what happens when its telemetry becomes stale.

Inference gateway routing sees work, not pods

Imagine three replicas serving the same base model. Pod A has four short decodes and the requested prefix in cache. Pod B has one medium decode, the requested adapter resident, and no prefix hit. Pod C is empty but is draining after a health signal. Round robin treats them as equivalent slots; least requests may pick C or overvalue B. Inference gateway routing needs a model of remaining work and preparation cost rather than a count of HTTP connections.

The request descriptor should name model, estimated prompt tokens, maximum output tokens, adapter identity, cache-scope key, priority class, and deadline class. The endpoint snapshot should name queued prefill tokens, queued decode tokens, active sequences, resident adapters, reusable prefix blocks, memory pressure, and health. A route is a scored match between those two records at a particular observation time.

The first dashboard should plot estimated work against observed start time for each request class. Inference gateway routing is only credible when operators can see where token estimates, batching, or preparation assumptions diverged.

Use the extension as a contract boundary

The Gateway API Inference Extension repository defines the evolving implementation and APIs, while the Kubernetes introduction to the Inference Extension explains why generic request balancing lacks model-serving context. Pin an entry from the project's live tag history in the deployment receipt. Names and maturity can change, so manifests should record the exact CRD and controller images under test.

Conceptually, an InferencePool describes a pool of serving endpoints and an Endpoint Picker makes request-aware choices. Keep policy outside the serving engine: the picker consumes bounded metrics and returns a candidate, while the gateway still owns traffic behavior and the backend owns execution. That separation lets routing evolve without teaching every client about cache blocks or adapter slots.

Keep the InferencePool and Endpoint Picker responsibilities legible in configuration and telemetry. The pool describes eligible model endpoints; the picker interprets request and live-serving evidence without quietly redefining availability. Store both selected object versions beside every sampled routing decision.

Estimate queue cost in model time

Raw queue length confuses a two-token completion with a long generation. Approximate queued work from prompt tokens awaiting prefill, remaining decode tokens, batch compatibility, and measured throughput for the endpoint's hardware and model revision. The estimate can be imperfect; it must be calibrated and monotonically related to user waiting. Compare predicted start time with observed queue delay by workload cohort, then update coefficients only from held-out traces.

Inference gateway routing should also reserve capacity for short interactive requests when long offline jobs arrive. A bounded aging term can improve fairness without allowing an old low-priority job to violate an interactive deadline class. Record both the raw execution score and the policy adjustment. Otherwise an operator cannot tell whether latency came from model load, a business priority, or a bad predictor.

Normalize queue units before adding them. Waiting requests, queued tokens, and predicted decode milliseconds answer different questions, so the scorer should convert each signal into expected delay with a timestamped calibration.

Price prefix affinity without crossing tenants

A prefix hit can avoid repeated prefill work, but the routing key must include the full cache trust boundary. Model revision, tokenizer, prompt bytes, relevant decoding configuration, tenant or isolation salt, and cache format all affect safe reuse. Never send a request toward an endpoint merely because an unsalted hash resembles another tenant's prefix. The expected savings should use reusable token count and measured prefill rate, not a boolean cache-hit badge.

The failure mode is a large cache bonus that overwhelms queue growth. Its consequence is a hot endpoint accumulating work because every related request follows yesterday's affinity. Cap the bonus at the estimated work actually avoided, decay it with telemetry age, and stop granting it under memory pressure. Inference gateway routing should prefer a warm endpoint only while its predicted completion remains better than a cold alternative.

Prefix cache routing must include model revision, tokenizer, tenant boundary, and cache policy in its compatibility key. A high hit estimate is negative evidence when any one of those dimensions is unknown.

Treat adapter residency as a preparation receipt

For multi-LoRA serving, a resident adapter can avoid storage fetch, host transfer, device copy, and activation. The endpoint snapshot needs adapter ID, exact version digest, rank or memory footprint, state such as loading or ready, and last verified time. A “loaded” string without version can route a request to incompatible weights. Score the preparation cost from measured cold-load distributions and the probability that current pressure evicts the adapter before execution.

Do not let residency create permanent owner stickiness. Add per-tenant queue ceilings and a fallback endpoint set, then separate locality benefit from fairness policy. When all resident endpoints are saturated, a cold load elsewhere may finish sooner. The route receipt should show queue estimate, cache savings, adapter savings, policy adjustment, and the final chosen endpoint so engineers can challenge each term independently.

LoRA adapter load is similarly conditional. Count residency only after the serving process confirms the requested adapter revision, then charge an explicit cold-load estimate when memory pressure may evict it before execution.

Fail closed on health and stale observations

Health is a gate before it is a penalty. Endpoints that are draining, model-mismatched, out of memory, or beyond an error threshold leave the candidate set. For remaining endpoints, snapshots older than a short workload-specific limit lose cache and adapter bonuses; very stale data triggers a conservative fallback such as gateway-level load balancing across known healthy members. A stale zero queue is more dangerous than an honest unknown.

Carry observation timestamps end to end and monitor the age at selection, not just collection success. If the picker cannot reach its telemetry source, the gateway should have explicit behavior with bounded duration. Inference gateway routing is an availability component, so degraded mode must be rehearsed under controller restarts, missed watches, and partial endpoint discovery rather than invented during an incident.

Stale observations should age into pessimism rather than remain attractive. After the freshness window, remove the endpoint or add a documented uncertainty penalty until health and load signals resume.

Build one score with explainable units

Convert every term to predicted milliseconds or another common cost unit: queued execution, request prefill, expected decode, cold adapter preparation, and cache savings. Apply health as exclusion and policy as a separately labeled adjustment. Avoid a bag of dimensionless weights whose meaning disappears after tuning. The selected endpoint should be the lowest eligible predicted completion time, with randomized tie-breaking among near-equal candidates to prevent synchronization.

Start with a simple linear score and test residuals before adding complexity. A learned router may eventually improve predictions, but it still needs monotonic safety constraints and an explanation surface. My position is that an interpretable five-term scorer is the right default until trace evidence proves a specific nonlinear interaction matters. Operational debugging speed is part of model-serving performance.

Write score components to the route receipt in their original units and after normalization. That makes a surprising choice explainable without reverse-engineering a single blended number during an incident.

Inject load patterns the scorer dislikes

Replay synchronized long prompts, many tiny chats, one popular prefix, alternating adapters, a cold-start wave, telemetry lag, and a draining endpoint. Measure time to first token, inter-token latency, deadline misses, cold loads, cache reuse, fairness by tenant, route churn, and prediction error. Compare round robin, least requests, queue-only, and the full live scorer on the same arrivals. Averages cannot qualify a router; retain tails by workload class.

The neighboring systems matter. Prefix cache isolation defines safe reuse keys, multi-LoRA serving explains adapter pressure, continuous batching shapes queue cost, and PagedAttention fragmentation reveals memory that raw utilization hides. Route evidence is credible only when these serving contracts agree on identity and capacity.

Stress the scorer with alternating long and short prompts, adapter bursts, cold prefixes, and one silent endpoint. The fixture should expose route churn, starvation, and optimistic telemetry before production traffic does. This keeps inference gateway routing measurable under the workload shapes most likely to defeat a tidy average.

Operate the route decision as evidence

Emit a compact receipt for sampled requests: request cohort, candidate endpoint IDs, snapshot ages, exclusion reasons, each cost term, selected endpoint, actual start time, completion outcome, and model revision. Use it to calibrate predictions and to answer support questions without logging prompt content. Aggregate miss reasons such as stale telemetry, no resident adapter, and all endpoints unhealthy; they point to different remedies.

Inference gateway routing succeeds when live evidence improves user latency without hiding policy or weakening isolation. Prefer preparation savings only up to the work they avoid, express queues in model time, remove unhealthy candidates, and degrade deliberately when observations age. Revisit the scorer when hardware, batching, cache format, adapter loading, or workload mix changes, because every coefficient belongs to that measured environment.

Runnable artifact: The deterministic scorer rejects an unhealthy endpoint and selects the lowest live execution cost instead of the smallest raw queue. Save it as live-load-router.test.mjs and run node live-load-router.test.mjs. Expected final line: PASS: live route selected.

import assert from "node:assert/strict";
const score=x=>x.queue*5+x.adapterLoad*3-x.prefixHit*4+x.unhealthy*100;
const pods=[{id:"a",queue:4,adapterLoad:0,prefixHit:1,unhealthy:0},{id:"b",queue:1,adapterLoad:1,prefixHit:0,unhealthy:0},{id:"c",queue:0,adapterLoad:0,prefixHit:1,unhealthy:1}];
assert.equal([...pods].sort((a,b)=>score(a)-score(b))[0].id,"b");
console.log("PASS: live route selected");

The initial production scorer should remain boring enough to calculate during an incident. Inference gateway routing adds predicted queue work and request cost, subtracts only preparation work demonstrably saved by compatible cache or adapter state, excludes unhealthy endpoints, and labels fairness adjustments separately. That formula creates an evidence trail that can evolve from real prediction error without handing traffic to an opaque learner prematurely.

Launch behind a sampled comparison that records the route the old policy would have selected, but never duplicates the user request. Review tail latency, deadline misses, cache safety, adapter loads, route churn, tenant fairness, and telemetry age by workload cohort. A median improvement cannot justify one isolated tenant or one long-prompt class becoming unstable. Recalibrate after model, hardware, batch scheduler, cache format, adapter catalog, or arrival mix changes, because every millisecond estimate belongs to that environment.