Prefix Cache Isolation for Multi-Tenant LLMs

Prefix cache isolation is the control that lets a shared LLM server reuse expensive prompt work without turning another tenant's traffic into a side channel. This guide shows how to define the cache key, salt boundary, observability, and eviction tests before enabling reuse.

The reader should leave with one answer: which requests may share cached KV blocks, and which identity, model, adapter, multimodal input, or policy change must force a miss.

The supporting vocabulary is tenant-scoped reuse policy, salted cache domains, shared prompt acceleration, isolated KV blocks. Each term serves the same search intent: prevent cross-tenant prefix reuse while preserving safe cache hits inside each trust boundary.

My position is that prefix caching is an authorization decision disguised as a performance feature. A cache hit should be explainable from an explicit trust boundary, never inferred from similar text alone.

1. Request identity
2. Salted block key
3. Allowed reuse
4. Forced miss

Prefix cache isolation starts with principals

A tenant is not always a user; it may be a workspace, legal entity, environment, or data-residency boundary. The vLLM automatic prefix caching design documents block hashes, extra identifiers, and per-request cache salts for multi-tenant isolation. For this prefix cache isolation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Inventory every caller identity
• Map workspace and environment ownership
• Separate public from private prefixes
• Write the reuse policy in plain language

The measurement I keep is cache hits sliced by source and destination principal. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is two principals outside the policy produce a shared block hit. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is a hit is allowed only when both requests resolve to the same declared cache domain. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Put every dependency in the key

Matching token IDs are insufficient when model weights, adapters, images, or system policy can change the resulting KV state. The SGLang system paper describes RadixAttention and reuse of KV cache across requests with shared prefixes. For this prefix cache isolation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Hash model and tokenizer revisions
• Include LoRA and prompt-template IDs
• Hash multimodal inputs
• Version safety and routing policy

The measurement I keep is a reconstructable cache-key ledger. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is an adapter or image changes while the text hash stays equal. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is any output-shaping dependency change must force a miss. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Salt the first block deliberately

A domain salt should enter the first block hash so every descendant block inherits the isolation boundary. The vLLM PagedAttention paper establishes block-based KV-cache sharing and copy-on-write as serving primitives. For this prefix cache isolation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Generate opaque domain salts
• Inject salt before block chaining
• Keep salts out of user prompts
• Rotate salts on policy change

The measurement I keep is hit and miss traces before and after salt rotation. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is a stale descendant block remains reusable after rotation. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is rotating the first-block salt must invalidate the complete prefix chain. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Option	Observed signal	Verdict
Global text hash	maximum hits; no tenant boundary	reject
User-level salt	safe but low team reuse	inspect
Tenant policy salt	bounded reuse with rotation	ship

Treat timing as observable surface

Even when cached bytes are never returned, latency differences can reveal whether another request warmed a prefix. For this prefix cache isolation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Replay guessed shared prompts
• Compare cold and warm latency distributions
• Add noisy-neighbor pressure
• Inspect per-tenant hit counters

The measurement I keep is timing separability between authorized and unauthorized probes. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is an external tenant can classify another tenant's warm prefix above chance. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is unauthorized probes must behave like clean misses and expose no foreign hit metadata. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Test the cache-domain invariant

A tiny deterministic function should reject cross-domain reuse before a GPU benchmark makes the bug expensive. For this prefix cache isolation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Encode domain in every block key
• Assert equal text can still miss
• Assert same domain can hit
• Rotate one dependency at a time

The measurement I keep is pairwise key equality across a dependency matrix. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is two requests with different tenant salts receive the same key. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is the key must differ for every changed trust or output-shaping field. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Runnable artifact. Save this bounded check as prefix-cache-tenant-isolation.test.mjs and run node --test prefix-cache-tenant-isolation.test.mjs. Expected output: PASS: tenant salt separates equal prefixes.

import assert from "node:assert/strict";
import test from "node:test";
const key=({tenant,model,tokens})=>[tenant,model,tokens.join(".")].join("|");
test("isolates equal prefixes",()=>{assert.notEqual(key({tenant:"a",model:"m",tokens:[1,2]}),key({tenant:"b",model:"m",tokens:[1,2]}));console.log("PASS: tenant salt separates equal prefixes");});

Design eviction without cross-tenant priority leaks

A shared cache can still reveal demand if one tenant can evict or monopolize another tenant's working set. For this prefix cache isolation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Set domain quotas
• Measure churn by tenant
• Bound admission for long prefixes
• Keep eviction reasons observable

The measurement I keep is hit rate and eviction debt per cache domain. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is one noisy tenant collapses another tenant's hit rate. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is no tenant may consume unbounded blocks or hide another domain's forced misses. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

1. ClassifyClassify

   Name principals and legal reuse boundaries.

2. KeyKey

   Hash every output-shaping dependency.

3. AttackAttack

   Probe timing, eviction, and rotation failures.

4. ReleaseRelease

   Gate hits by tenant and policy telemetry.

Log enough to investigate, not reconstruct prompts

Operational evidence needs hashed identities and block lineage, not raw confidential prompt content. For this prefix cache isolation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Log opaque request and domain IDs
• Record key-version and hit depth
• Redact token content
• Expire diagnostic joins

The measurement I keep is an audit trail that explains each hit without plaintext. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is debug logs become a second prompt store. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is operators can reconstruct authorization decisions but not customer text. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Publish the isolation contract

The release packet should name cache domains, dependencies, salt ownership, rotation triggers, and rollback behavior. For this prefix cache isolation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Document allowed sharing examples
• List all forced-miss fields
• Assign salt rotation ownership
• Keep caching disablement reversible

The measurement I keep is a clean-room replay after restart and rotation. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is an engine upgrade silently changes key semantics. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is requalify the contract after any engine, model, adapter, or policy change. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

The useful version is bounded

Prefix cache isolation is complete when every hit can answer who shared what computational state under which versioned policy. The speedup is valuable only inside that proof.

Keep a forced-miss control and a salt-rotation replay in the release packet. They turn future engine upgrades into testable changes instead of assumptions.

Continue with KV-cache optimization, continuous batching, AI agent operating rules, agent-ready API specs. Those field notes deepen adjacent implementation choices without turning this page into several articles at once.