Semantic Caching Without Wrong Answers
Reduce LLM calls with scoped candidates, adversarial prompt pairs, policy-aware verification, expiry, and false-reuse evidence.
Semantic caching can save an LLM call by reusing an answer for a meaningfully similar request, but similarity is not equivalence. The dangerous result is a fluent cached answer that fits the words while violating a hidden constraint such as jurisdiction, account, date, permissions, or document revision.
This guide builds a verified cache from labeled prompt pairs, risk tiers, a policy-aware verifier, and a precision-versus-savings frontier. It is for teams that want cost reduction without making wrong reuse invisible.
The safety model distinguishes an LLM response cache, a similarity threshold, cache verification, and a false cache hit; treating those as one score is how plausible mistakes ship.
Semantic caching breaks classic cache assumptions
Semantic caching replaces exact key equality with a claim that two requests can share an answer. That claim depends on meaning, hidden context, model behavior, sources, policy, and time. A vector neighbor is evidence for consideration, not a cache key with ordinary equality semantics.
Exact caches are safe when the key captures every response-changing input. Meaning-based reuse often omits locale, account state, permissions, conversation turns, tool results, or source revision. The omission can make two close embeddings operationally different even when a human sees them as paraphrases.
My position is that reuse should default to ineligible until the workload names an equivalence contract. The boundary is narrow: stable, low-consequence, source-bounded answers can qualify; personalized, transactional, rapidly changing, or high-consequence requests should bypass unless a stronger domain verifier exists.
Write the contract as fields: normalized intent, entities, constraints, owner scope, dependency digests, model and prompt versions, maximum age, and consequence tier. Only after these agree should vector distance influence the decision. This order prevents a convenient score from silently becoming policy.
Partition candidates before similarity search
Semantic caching needs physical or logical partitions for tenant, locale, product surface, authorization class, prompt revision, model family, and knowledge revision where those values affect output. Searching one global index and filtering later may expose candidate metadata or increase the chance of a policy bug.
The query pipeline first derives a cache namespace from trusted application state, never from model-generated text. It then retrieves within that namespace. Store the response, structured request facts, dependency versions, creation time, expiry, and verification evidence beside the embedding.
Avoid caching raw secrets or private context merely because embeddings seem opaque. Vector representations and cached responses are still derived user data. Apply encryption, retention, deletion propagation, and access logging according to the source data rather than treating the index as harmless infrastructure.
A namespace migration should make older entries unreachable by new policy while a bounded cleanup removes them. Keep policy version in the receipt. That lets operators explain a miss after a deployment and avoids interpreting an old entry under rules it never passed.
Illustrative TypeScript — a reviewable design sketch, not a compiled production implementation.
type ReuseCandidate = {
namespace: string;
intent: string;
entities: Record<string, string>;
constraints: Record<string, string>;
dependencyDigests: string[];
policyVersion: string;
expiresAt: string;
};
| Signal | Decision | Proof |
|---|---|---|
| Stable public definition | Verify and reuse | Meaning and dependency digest match |
| User-specific state | Partition or bypass | Owner and freshness preserved |
| Consequential advice | Generate fresh | No cached authority substituted |
Build an adversarial labeled-pair set
Semantic caching should be evaluated with pairs labeled safe reuse or fresh generation, plus the reason. Start with genuine query shapes, then author boundary cases that change one response-critical fact while preserving most words. Include paraphrases, negation, dates, units, jurisdictions, user identity, and document versions.
The worked set begins with 'How do I reset my personal API token?' and a safe paraphrase asking to rotate one's own token. A dangerous neighbor asks an administrator to reset a teammate's token. The verbs and nouns align, but actor, subject, permissions, and audit consequence differ.
Add another pair: 'What is the refund window?' against policy revision 12 and the same sentence after revision 13 changes thirty days to fourteen. The text query is identical; the dependency digest makes reuse invalid. This catches systems that embed only the user's words.
Preserve the dataset as a small reviewable table with pair ID, prompt A, prompt B, structured differences, label, consequence, and reviewer note. Split by scenario rather than random rows so near duplicates do not leak across evaluation and inflate apparent generalization.
Use a verifier with abstention
Semantic caching needs a second decision after retrieval. A deterministic verifier checks namespace, versions, expiry, required entities, constraints, and authorization. A learned or model-based verifier may compare residual meaning, but it must be allowed to abstain and its input cannot add authority.
Ask the verifier a narrow question: can the stored answer satisfy this request without changing any factual, policy, or user-specific claim? Require a structured decision with matched facts, conflicts, missing evidence, confidence calibration if measured, and reason code. Free-form yes or no is not an audit artifact.
The Apple research on verified semantic caching motivates adding a verification stage to reduce erroneous reuse. Treat the publication's results as evidence for its datasets and setup, then validate your own distribution and consequence costs.
The failure mode is correlated error: retriever and verifier may rely on the same embedding weakness. Mitigate it with independent structured checks, deliberately hard negative pairs, manual review of accepted boundary cases, and a conservative abstention route that simply generates fresh output.
Plot precision against savings
Semantic caching should optimize expected value under asymmetric costs. A mistaken reuse may be far more expensive than one extra model call, so accuracy alone is insufficient. Report precision among accepted hits, coverage, estimated saved compute, verifier overhead, and consequence-weighted error.
Sweep retrieval distance and verifier cutoffs over the labeled pairs. Plot reuse coverage on one axis and accepted-hit precision on the other, with separate curves by risk tier. Choose an operating point from an explicit maximum error budget, not the elbow that looks attractive on a chart.
The GPTCache paper presents a modular design and evaluation for this class of systems. Its benchmarks are useful architecture evidence; your release choice still needs current traffic, current embeddings, current prompts, and a product-specific cost for harmful reuse.
Track lower confidence bounds when the accepted sample is small. A perfect result on twelve reusable pairs is not evidence for a broad rollout. Keep a minimum reviewed-hit count and widen the cohort only when precision remains above the declared boundary under representative volume.
- ClassifyClassify
Determine risk tier, owner scope, and required freshness.
- RetrieveRetrieve
Find candidates inside the eligible partition.
- VerifyVerify
Compare structured facts, dependencies, and policy.
- DeclareDeclare
Return a receipt or generate a fresh response.
Make reuse visible without adding noise
Semantic caching should produce a receipt whether or not the interface shows a badge. Record candidate ID, namespace, distance, verifier version, matched facts, dependency versions, age, decision, and reason. Join it to response traces without storing protected prompt text in routine logs.
In user-facing products, disclose reuse when freshness or provenance matters: 'Reused a verified answer from policy revision 13.' Offer source inspection or refresh, not a decorative cache icon. For low-consequence stable help text, backend observability may be enough.
Never present an old generation timestamp as the answer's current verification time. Store both. Revalidation can renew the verification receipt without pretending the model generated new language. If dependencies change, mark the entry stale immediately and route to fresh generation.
A visible refresh control must bypass reuse and say what changed, if anything. This helps support diagnose suspicious answers and creates valuable labeled examples when people reject an accepted candidate. Do not automatically treat every refresh as proof the prior answer was wrong.
Test expiry, drift, and false reuse
Semantic caching tests should change one dependency at a time: embedding revision, prompt template, model, tool policy, source corpus, locale, user role, and system clock. Expected behavior is a named miss or revalidation path, never silent interpretation under the new configuration.
Run the adversarial pair set on every policy change and sample accepted production candidates for blinded review. Monitor disagreement by scenario. A rise around dates may need temporal normalization; a rise across tenants is an isolation incident and should disable reuse immediately.
Measure stale-entry cleanup and deletion completion. Removing a source or user record must invalidate derived responses and embeddings. Keep a digest tombstone if needed for audit, but do not retain answer text beyond the source's approved lifecycle.
The rollback switch should bypass lookup and generation reuse while leaving receipts available for analysis. Deleting the index during an incident destroys evidence and adds operational load. Correctness must remain available through ordinary fresh generation when the optimization is off.
Ship a narrow verified cache first
Semantic caching is ready for one workload when the equivalence contract is explicit, partitions follow trusted scope, hard negatives pass, accepted precision clears its consequence-weighted boundary, and every entry carries dependency and expiry evidence. Global enablement is not a meaningful milestone.
Start with stable public documentation answers tied to a corpus revision. Exclude account data, actions, and personalized recommendations. Shadow the policy, manually review proposed hits, enable a small cohort, and preserve a fresh-generation control that reports quality, cost, and completion time.
Set automatic disable conditions for isolation mismatch, dependency-check failure, precision bound breach, unexpected verifier error, or stale cleanup backlog. Cost savings should never be the only live signal. A safe cache knows when it has lost the right to answer.
The release packet contains the contract, labeled pairs, frontier plot, accepted and rejected receipts, deletion fixture, and rollback rehearsal. That bundle demonstrates original value: not that similar prompts exist, but that reuse remains constrained by inspectable product truth.
Put the method into practice
Semantic caching is safe only when vector similarity proposes candidates inside a trusted partition and a separate verifier proves that response-changing facts, dependencies, policy, owner, and age remain compatible. The system must prefer an honest miss over a plausible wrong answer.
Begin with twenty paraphrases and twenty hard negatives from one stable documentation domain. Plot accepted precision against reuse coverage, inspect every error, and do not enable the cache until the chosen operating point reflects the real cost of a mistaken answer.
The narrow method connects to four existing Journal notes: HTTP caching without superstition, AI evaluation measurement contracts, RAG citations that survive change, tests that challenge generated intent. They cover adjacent implementation boundaries without changing this article's single search intent. Preserve the worked fixture, its visual evidence, and the release receipt so a later update can compare behavior instead of relying on memory.