vLLM vs SGLang for Agent Workloads

vLLM vs SGLang benchmarks become useful for agents only when the trace includes tool pauses, repeated prefixes, structured outputs, adapter churn, and the long tail hidden by aggregate tokens per second.

This guide defines one matched replay, the invariants that keep it fair, and a decision record that weighs serving behavior and operating cost together.

vLLM vs SGLang begins with an agent transcript

Build the benchmark from a real shape without claiming private production experience: a synthetic trace can model a planner, two tool waits, three short structured generations, one long synthesis, and a repeated system prefix. Give each turn prompt length, maximum output, actual output, pause duration, schema, adapter, deadline, and tenant. Preserve arrival order and concurrency.

Agent inference serving differs from chat completion serving because the GPU alternates between bursts of generation and externally imposed silence. A system that wins a steady decode test can lose when those gaps change batching and cache pressure.

For vLLM vs SGLang, the working artifact is a versioned synthetic agent trace. It records seed, arrivals, token estimates, tool waits, schemas, adapter IDs, and deadlines. I would stop the release when the benchmark contains only one continuous prompt-completion pair; that failure means the evidence cannot support this step's claim.

A burst of short tool turns should drive a versioned synthetic agent trace; capture seed, arrivals, token estimates, tool waits, schemas, adapter IDs, and deadlines. Stop when the benchmark contains only one continuous prompt-completion pair, because that outcome breaks the first boundary under test.

The exact implementation vocabulary here includes agent inference serving, so the term remains connected to a concrete decision rather than hidden in metadata.

• Freeze: Pin engines, model, hardware, and semantics.
• Replay: Preserve bursts, pauses, caches, and adapters.
• Break: Inject worker, telemetry, and cancellation faults.
• Decide: Publish raw evidence and workload weights.

Pin equivalent engine contracts

Use the official documentation and repositories to pin engine version, supported model revision, tokenizer, quantization, attention backend, tensor parallelism, GPU clocks, memory target, and server flags. Match request semantics before measuring: sampling parameters, stop behavior, structured output grammar, prefix cache scope, and adapter revision must mean the same thing. Record unsupported combinations instead of silently disabling a feature.

The comparison is not fair when one engine gets a warm cache, looser schema, or different maximum sequence length. Treat setup parity as a testable artifact, not a paragraph in the result.

The decision surface for vLLM vs SGLang is a two-column configuration lockfile. Its compact receipt contains container digest, command line, model files, hardware, request fields, and unsupported deltas. If similar-looking flags are assumed semantically equal, the route stays unresolved and returns to design before polish.

Adapter eviction pressure should enter a two-column configuration lockfile; an uninvolved reviewer must recover container digest, command line, model files, hardware, request fields, and unsupported deltas. Hold the next action when similar-looking flags are assumed semantically equal.

The primary references for this decision are vLLM documentation, vLLM repository, and SGLang documentation. Engine documentation establishes supported mechanisms, but only a matched agent trace reveals prefix reuse, tool pauses, structured decoding, and adapter churn together. Repository claims do not substitute for pinned versions, identical hardware, and raw request-level evidence.

The exact implementation vocabulary here includes continuous batching, so the term remains connected to a concrete decision rather than hidden in metadata.

Measure the complete latency shape

Report queue time, time to first token, inter-token latency, terminal latency, and deadline misses by turn type. Separate prefill-heavy planning, tiny tool arguments, structured extraction, and long synthesis. Include p50, p95, and p99; one slow tool-call argument can stall an otherwise responsive agent.

Throughput remains useful, but express it beside accepted requests and wasted work from cancellations. Continuous batching should improve device use without making a short control turn wait behind a long decode. The benchmark should therefore preserve user-visible sequence, not just aggregate emitted tokens.

I would review vLLM vs SGLang through a per-turn latency waterfall, not a slide assembled after implementation. The saved evidence is arrival, queue start, first token, last token, pause boundaries, cancellation, and cohort. The explicit rejection rule is simple: one tokens-per-second average hides control-turn tails.

Worker-loss recovery should interrupt a per-turn latency waterfall, with arrival, queue start, first token, last token, pause boundaries, cancellation, and cohort retained for comparison. Reopen the design if one tokens-per-second average hides control-turn tails.

The exact implementation vocabulary here includes prefix cache reuse, so the term remains connected to a concrete decision rather than hidden in metadata.

Test prefix reuse inside the trust boundary

Agent traces repeat system instructions, tool descriptions, policy text, and conversation prefixes. Measure reusable tokens and cache hit eligibility, then include model revision, tokenizer, tenant salt, prompt bytes, and cache policy in the key. Prefix cache reuse is valuable only when it cannot cross an authorization boundary or route stale tool descriptions.

Replay cold, warm, and memory-pressure phases for both servers. A high apparent hit rate can be negative evidence if requests cluster onto one saturated worker or incompatible prefixes are treated as equivalent.

This part of vLLM vs SGLang becomes testable through a cache eligibility and eviction ledger. Preserve requested prefix digest, isolation scope, reusable tokens, endpoint, hit, eviction, and saved prefill time. Treat the step as failed whenever cache hit rate is compared without tenant safety or latency consequence, even when the visual result appears convincing.

A cache-disabled control should accompany a cache eligibility and eviction ledger; the fallback receipt is requested prefix digest, isolation scope, reusable tokens, endpoint, hit, eviction, and saved prefill time. Treat cache hit rate is compared without tenant safety or latency consequence as an explicit failed state.

The exact implementation vocabulary here includes structured generation, so the term remains connected to a concrete decision rather than hidden in metadata.

Exercise schemas and tool-call bursts

Structured generation changes the decode path. Create JSON fixtures with nested objects, enums, optional branches, Unicode, long strings, and impossible schemas, then require equivalent accept and reject behavior. Measure compile or preprocessing cost separately from constrained decoding.

Follow each structured turn with a simulated tool pause and a short correction turn. The key product question is not whether the engines can emit JSON; it is whether schema latency, failure text, cancellation, and retry behavior remain predictable when many agents reach the same tool boundary together.

For vLLM vs SGLang, the working artifact is a golden structured-output corpus. It records schema digest, compile time, valid rate, rejection class, correction turns, and terminal latency. I would stop the release when a single easy JSON object stands in for structured generation; that failure means the evidence cannot support this step's claim.

A burst of short tool turns should drive a golden structured-output corpus; capture schema digest, compile time, valid rate, rejection class, correction turns, and terminal latency. Stop when a single easy JSON object stands in for structured generation, because that outcome breaks the first boundary under test.

Dimension	Measure	Why it matters
Control turns	TTFT p99	Tool responsiveness
Prefixes	Safe saved prefill	Repeated context
Schemas	Valid + correction	Tool reliability
Adapters	Cold wait + eviction	Product variety

Rotate adapters and memory pressure

If the product serves multiple fine-tunes, alternate adapters according to a measured or explicitly synthetic popularity curve. Record ready, loading, evicted, and incompatible states. Charge cold-load time to the request that experiences it, and include base-model KV cache pressure rather than reporting adapter load in isolation.

Multi-LoRA support is operationally useful only if version identity and residency are visible. A smaller cold-load median does not compensate for uncontrolled eviction storms or a route that starves a rare but urgent adapter.

The decision surface for vLLM vs SGLang is an adapter residency heatmap. Its compact receipt contains adapter digest, memory footprint, load transition, eviction cause, request wait, and fallback. If all benchmark requests use one permanently resident adapter, the route stays unresolved and returns to design before polish.

Adapter eviction pressure should enter an adapter residency heatmap; an uninvolved reviewer must recover adapter digest, memory footprint, load transition, eviction cause, request wait, and fallback. Hold the next action when all benchmark requests use one permanently resident adapter.

Include failure and recovery runs

Kill one worker during prefill, drain another during decode, delay telemetry, send an oversized prompt, cancel at the first token, and restart the router with warm clients. Observe whether requests fail once, retry safely, or duplicate visible output. Retain server and gateway identifiers so an operator can join the trace without prompt content.

The winning system is the one the team can repair under realistic pressure, not the one that never encounters a fault in a ten-minute run. Recovery time and explanation quality belong in the score.

I would review vLLM vs SGLang through a fault-injection transcript, not a slide assembled after implementation. The saved evidence is failure boundary, accepted identity, retry owner, duplicate output, recovery time, and operator action. The explicit rejection rule is simple: error rate is measured only while every worker is healthy.

Worker-loss recovery should interrupt a fault-injection transcript, with failure boundary, accepted identity, retry owner, duplicate output, recovery time, and operator action retained for comparison. Reopen the design if error rate is measured only while every worker is healthy.

1. 1Freeze

   Pin engines, model, hardware, and semantics.

2. 2Replay

   Preserve bursts, pauses, caches, and adapters.

3. 3Break

   Inject worker, telemetry, and cancellation faults.

4. 4Decide

   Publish raw evidence and workload weights.

Join serving mechanics to adjacent controls

Continuous batching explains scheduling pressure, prefix isolation defines safe reuse, multi-LoRA serving defines adapter identity, and constrained decoding defines output acceptance. Read those contracts alongside the benchmark rather than treating the engines as sealed binaries.

If a server's default conflicts with the product's cache or schema policy, the decision record must name the adapter layer or the reason to reject it. This keeps vLLM vs SGLang grounded in the complete agent system instead of a narrow serving leaderboard.

This part of vLLM vs SGLang becomes testable through a dependency-and-policy map. Preserve each serving feature's owner, configuration, neighboring invariant, and regression test. Treat the step as failed whenever the server choice silently redefines product policy, even when the visual result appears convincing.

A cache-disabled control should accompany a dependency-and-policy map; the fallback receipt is each serving feature's owner, configuration, neighboring invariant, and regression test. Treat the server choice silently redefines product policy as an explicit failed state.

Related implementation evidence lives in continuous batching for LLM inference, prefix cache tenant isolation, multi-LoRA serving, and constrained JSON decoding. Continuous batching, prefix isolation, cache optimization, and admission control each alter the benchmark envelope. Attach their active policies to every replay so a serving-engine comparison cannot quietly award one system a different workload.

The runnable score demonstrates why p99, adapter cold load, and prefix reuse can reverse a throughput-only ranking.

Runnable artifact — serving-workload-score.test.mjs

import assert from "node:assert/strict";
const score=(s,w)=>w.toolBursts*s.p99+w.adapters*s.adapterCold+w.prefixReuse*(1-s.cacheHit)*100;
const work={toolBursts:3,adapters:1,prefixReuse:4};
assert.ok(score({p99:90,adapterCold:40,cacheHit:.8},work)<score({p99:130,adapterCold:15,cacheHit:.3},work));
console.log("PASS: serving score preserves workload");

Run node serving-workload-score.test.mjs. Expected receipt: PASS: serving score preserves workload.

Choose from a weighted operating record

Weight latency cohorts, deadline misses, structured validity, prefix safety, adapter cold starts, memory headroom, fault recovery, upgrade work, observability, and team familiarity. Publish raw results before weights so another team can reach a different answer. Run at least three seeded repetitions after warm-up and keep the full distributions.

My recommendation rule is deliberately conditional: choose the engine that passes every contract and minimizes the workload-weighted tail and operating burden on the pinned hardware. Re-run after an engine, model, driver, scheduler, or trace change.

For vLLM vs SGLang, the working artifact is a workload-weighted serving decision record. It records raw measurements, exclusions, normalized weights, selected engine, dissent, and revisit trigger. I would stop the release when a universal winner is declared from one GPU and one synthetic throughput test; that failure means the evidence cannot support this step's claim.

A burst of short tool turns should drive a workload-weighted serving decision record; capture raw measurements, exclusions, normalized weights, selected engine, dissent, and revisit trigger. Stop when a universal winner is declared from one GPU and one synthetic throughput test, because that outcome breaks the first boundary under test.

Choose between vLLM vs SGLang from a weighted replay of the actual agent workload, not from an isolated throughput peak. Reopen the decision after engine releases, model changes, or traffic-shape drift, and retain enough raw traces for another team to reproduce the selection.