LLM Routing by Cost, Risk, and Latency

LLM routing can reduce cost and latency, but a cheaper response is not a win when the router silently downgrades quality on the request that matters. A credible router needs a consequence-aware policy, versioned evidence, and a fallback rule that a reviewer can reconstruct.

This article builds a routing receipt across four workload classes and shows how to shadow-test candidate models before they receive production authority. The outcome is an optimization system with an explicit safety boundary, not a hidden switch statement.

The operating vocabulary connects model routing, AI cost optimization, inference latency, and model fallback so savings can never be evaluated separately from consequence and recovery.

LLM routing begins with workload classes

Classify requests by the job and consequence visible to the product. Brainstorming, extraction, grounded support, code modification, and account mutation are not interchangeable token streams. Each class needs its own quality rubric, latency objective, privacy boundary, and fallback behavior. The class should be stable enough to analyze but narrow enough to explain.

Do not let the router infer consequence only from prompt wording. The calling feature knows whether the output will remain a draft, reach a customer, or drive an external effect. Pass that context as typed policy input. A user saying ‘just do it’ should not convert a preview-only workflow into an automatic write.

The original artifact is a workload card containing owner, examples, disallowed models, evaluation set, p95 latency budget, maximum unit cost, failure behavior, and review date. LLM routing becomes operable when a team can review that card without reading provider-specific branching code.

Include residency, modality, context-window need, tool support, and provider retention when those factors constrain eligibility. A model that is inexpensive and accurate on text may still be invalid for a regional workload or unable to preserve a required tool contract. Treat these as hard gates, not small deductions in one blended score.

Reference note. RouteLLM paper is the primary standard, model, or research source for the implementation claim immediately above.

Build a versioned routing receipt

For every decision, record request class, policy version, eligible models, expected quality band, estimated cost, recent latency, selected model, fallback reason, and evaluation cohort. Hash or redact sensitive prompt material; the receipt needs decision context, not a second copy of every customer conversation.

Include whether the selection was primary, fallback, experiment, or manual override. If a provider times out and the system chooses another model, the interface and metrics should not treat that as the normal path. LLM routing receipts make degradation visible before it becomes the default through a long-running incident.

The trade-off is telemetry volume. Sample low-consequence successes while retaining complete aggregate counts, but keep full decision receipts for high-consequence paths and anomalies. Never sample away denials, unsafe disagreements, repeated fallbacks, or cases that cross a policy threshold.

Receipts need price provenance too. Store the estimator or provider-price version used by the decision, because token accounting and rates change. Without that snapshot, a later analyst may reconstruct quality correctly but reach the wrong conclusion about why one eligible model won the route.

Illustrative TypeScript — an interface sketch, not a compiled implementation.

type RoutingReceipt = {
  requestClass: "draft" | "grounded" | "code" | "consequential";
  policyVersion: string;
  candidates: Array<{ model: string; quality: number; cost: number; p95Ms: number }>;
  selected: string;
  reason: "primary" | "fallback" | "experiment" | "override";
  cohort: string;
};

Evaluate the dimensions separately

A single win rate hides why a model is useful. Measure task quality on a reviewed dataset, p50 and p95 time to a useful result, input and output cost, refusal behavior, groundedness, tool-call validity, and disagreement on safety-critical cases. Report uncertainty when the sample is small rather than turning three examples into a precise score.

In the worked example, model A costs one unit, answers drafts in 700 milliseconds, and scores 86 on copy acceptance. Model B costs four units, takes 1.8 seconds, and scores 94. For draft headlines, A is eligible. For a policy-grounded refund explanation, only B clears the citation and contradiction thresholds. One global ranking would choose badly for at least one class.

Two observations follow. Cost optimization happens inside an eligibility boundary, and a slower model can still improve perceived time when it produces the first usable answer with fewer retries. LLM routing should optimize completed user work, not raw response latency detached from correction effort.

Add a baseline that sends every fixture through the current production model. The router must beat that simple path on cost or latency without crossing the class quality floor. Comparing only candidate routes can make a complicated policy appear valuable even when the stable baseline remains the better product decision.

Shadow traffic before granting authority

Replay scrubbed production requests to candidate models without exposing their output or allowing tools. Preserve the same retrieval snapshot and tool descriptions so the comparison is meaningful. Have deterministic checks score schemas, citations, and policy, then sample disagreements for human review. The candidate should not learn from data it is not permitted to process.

Use paired analysis: compare candidates on the same request, not unrelated daily averages. Segment by language, input length, customer tier, and boundary cases. A model can look equivalent overall while failing a small class that carries most consequence. That class should remain excluded even when the global cost curve is attractive.

The failure mode is silent downgrade. During provider stress, a fallback model receives a task it has never passed. The consequence is lower quality precisely when monitoring is noisy. Mitigate it with per-class fallback allowlists and a fail-closed or human-review state when no eligible model remains.

Route with thresholds, not vibes

A simple policy can be explicit: choose the lowest-cost eligible model whose lower confidence bound clears the class quality threshold and whose recent p95 meets the latency objective. If no model qualifies, select the declared degraded state. The policy should be deterministic given its inputs even when the model responses are not.

LLM routing can use a learned classifier, but its output is another measured component. Track confusion by request class and keep high-consequence overrides deterministic. When classification confidence is weak, route upward or ask for clarification instead of gambling on the cheapest path.

Version thresholds and feature inputs. A route chosen under last month’s benchmark should remain reconstructable after models and prices change. Store the snapshot used by the decision, then refresh policy on a schedule or when drift signals cross a named boundary.

Apply hysteresis near a threshold so small measurement noise does not switch models on adjacent requests. The router can require a meaningful margin or a sustained window before promoting a new path. This reduces inconsistent tone, cache fragmentation, and operational confusion without locking the product to an inferior model forever.

Design fallback as a product state

Fallback changes the material of the experience. A slower high-quality model may need a progress state. A cheaper degraded model may require a draft label and block external actions. A complete outage may preserve the user’s input and offer manual continuation. The interface should not claim normal service while behavior operates under a different contract.

Avoid provider names unless they help the user act. Explain capability: ‘Draft mode is available; policy checks need review’ is more useful than a vendor error. Operators can see provider details in the receipt. LLM routing becomes trustworthy when internal complexity maps to a clear user consequence.

Give people a retry only when the next attempt can differ, and make cancellation propagate across hedged requests. Otherwise a timed-out primary may continue generating after fallback completed, wasting cost or attempting a duplicate tool call. Idempotency and abort signals belong in the routing design.

Reference note. FrugalGPT paper is the primary standard, model, or research source for the implementation claim immediately above.

Release with drift and rollback signals

Canary one workload class, one traffic slice, and one policy version. Compare quality, latency, cost, fallback rate, correction behavior, and user abandonment against the baseline. Predeclare rollback triggers, especially unsafe disagreement and no-eligible-model frequency. A cost improvement cannot override a safety regression.

Monitor provider updates, dataset drift, language mix, and changes in request length. Re-run shadow evaluation when a model alias changes even if the API name stays the same. The receipt should carry a concrete model revision when the provider exposes one and record unknown when it does not.

Review the router quarterly and retire dead branches. Complexity creates accidental policy, especially after emergency fallbacks. A concise current matrix is stronger than years of conditionals nobody can explain. The release is complete when operators can answer which classes changed, why, and how to reverse them.

Reference note. NIST AI RMF Resource Center is the primary standard, model, or research source for the implementation claim immediately above.

Conclusion and implementation references

LLM routing should classify the user-visible job, restrict eligibility by consequence, compare cost and latency inside that boundary, and leave a replayable receipt. The optimization is credible only when the router can also say no eligible model is available.

Start with four workload cards and a paired shadow test. Keep consequential requests on the baseline until a candidate clears their specific checks. That narrow rollout will show whether the router improves completed work or merely lowers an invoice.

This method also sits beside five related Journal notes: AI eval measurement contracts, architecture decision baselines, consequence-shaped review, journey-level SLOs, cancellable fetch pipelines. Each expands one boundary that this article deliberately keeps narrow, so the links are supporting material rather than competing actions. The three authoritative references are placed beside the specific claims they support above; the framework, examples, failure modes, and implementation judgments are my synthesis.

Use the conclusion as a release boundary: reproduce the named fixture, preserve its evidence, and record any exception before extending the pattern to a higher-consequence workflow. That final receipt makes the method reviewable by someone who did not build it and gives a future update a concrete point of comparison.