Semantic Cache Invalidation for LLM Products

Semantic cache invalidation is the difference between reusing a verified paraphrase and serving a polished answer whose facts, permissions, or policy have expired. Similarity can locate a candidate, but it cannot establish identity, authority, freshness, or the absence of a new side effect.

This guide builds a layered reuse decision from exact scope, constraint comparison, source validators, policy epochs, invalidation tags, and bounded regeneration. Every hit earns a reason; every miss protects a named product fact.

Semantic cache invalidation begins with answer identity

A semantic cache can find a previous question that resembles the current one, but resemblance does not establish that its answer is reusable. Semantic cache invalidation starts by defining identity across tenant, user permissions, locale, product, model, system prompt, tools, retrieval corpus, policy version, and requested output contract. Vector similarity proposes a candidate only after these hard dimensions match.

Treat identity as a structured key with semantic search nested inside it. A cached answer about an Acme invoice cannot serve Zephyr because the wording matches. A healthcare policy answer cannot cross jurisdiction, and a JSON extraction cannot reuse a prose response. Hash canonical immutable dimensions, retain readable fields for audit, and reject entries whose older key schema cannot express a new boundary.

The HTTP Semantics specification on caching formalizes freshness, validators, and invalidation for web caches. LLM products need additional semantic matching, but the older lesson remains valuable: freshness is metadata tied to representation identity, not a feeling derived from textual similarity. Preserve creation time, validators, and explicit directives beside every generated payload.

Separate candidate similarity from reuse eligibility

Use embeddings or another retrieval method to locate likely paraphrases within a hard identity partition. Then run a deterministic eligibility gate. Check policy and model versions, source snapshot, user scope, tool availability, answer type, safety class, TTL, and invalidation tags. Finally, if the task warrants it, verify entailment between current request constraints and the cached request before returning the stored answer.

The semantic cache key therefore has two layers: exact scope dimensions and a fuzzy intent representation. Store both. A high cosine score can still be a false neighbor when cancel after trial resembles cancel before trial, or when a single product SKU changes the answer. Include extracted entities, dates, negation, and constraints in the eligibility proof rather than asking an embedding to carry all semantics.

Verified semantic caching introduced reuse as a checked decision. This article extends that design into invalidation: the cache must explain not only why a candidate matched but which future events revoke it. A hit receipt should include exact-key fields, similarity, constraint comparison, validators, freshness, and the policy that allowed delivery.

Runnable artifact: The fixture proves cache reuse requires exact scope, source version, and unexpired freshness rather than query similarity alone.

Save this proof as semantic-cache-identity.test.mjs and run node semantic-cache-identity.test.mjs. Expected final line: PASS: cache identity gates.

import assert from "node:assert/strict";
const key=x=>[x.tenant,x.policy,x.model,x.promptVersion,x.sourceVersion].join("|");
const eligible=(stored,live)=>key(stored)===key(live)&&Date.parse(live.now)-Date.parse(stored.created)<stored.ttlMs;
const base={tenant:"a",policy:"p2",model:"m",promptVersion:"7",sourceVersion:"42",created:"2026-08-13T10:00:00Z",ttlMs:60000};
assert.equal(eligible(base,{...base,now:"2026-08-13T10:00:30Z"}),true);
assert.equal(eligible(base,{...base,tenant:"b",now:"2026-08-13T10:00:30Z"}),false);
assert.equal(eligible(base,{...base,sourceVersion:"43",now:"2026-08-13T10:00:30Z"}),false); console.log("PASS: cache identity gates");

Model freshness as clocks and source versions

Time-to-live is only one freshness signal. Some answers expire by wall clock, such as inventory or exchange rates. Others remain valid until a source changes, a policy publishes, a user permission moves, or a model behavior revision invalidates generated wording. Attach multiple validators and declare whether any change revokes the entry or whether a selected subset can be rechecked.

Use freshness validators such as document revision IDs, database update sequences, policy hashes, tool schema versions, and retrieval index manifests. A response grounded in three documents stores all three or a snapshot digest. If the current query resolves to a different source set, miss or regenerate. Do not refresh the timestamp after a superficial read that never verified the claims.

The Amazon Builders' Library article on cache challenges discusses cache keys, expiration, invalidation, and resilience from an official operational perspective. Apply its caution to generated answers: long TTL improves hit rate but expands stale exposure, while an unavailable origin makes cache behavior part of failure policy rather than a transparent optimization.

Attach invalidation tags to product facts

Tag entries with the facts and authorities that determine them: tenant:acme, policy:refund:v7, catalog:sku-42, docs:index-19, prompt:support-12, or tool:orders-v3. Events publish those tags when state changes, and the cache removes or tombstones matching entries. Tags should be bounded, normalized, and created from authoritative IDs rather than free text.

A policy-aware cache invalidates before newly forbidden content can be served. Permission revocation, legal hold, consent withdrawal, safety rule updates, and tenant offboarding are high-priority events. Propagate them synchronously or through a queue with a measured maximum lag, and deny reads when the consumer cannot prove it has processed the relevant policy epoch.

Do not rely on deletion alone in a distributed service. Record a namespace epoch or minimum valid version checked on read, so a replica or browser with an old object cannot resurrect it. The runnable fixture uses exact identity fields and source version with TTL. Production adds epoch comparison and signed invalidation events, then tests lag and replay under partial outages.

Design writes and side effects to bypass answer reuse

Cache only operations whose semantics tolerate reuse. A request that creates, pays, sends, mutates, or reserves must not be satisfied by a cached natural-language success. Tool calls need idempotency and effect receipts at the tool layer. The cache may reuse explanatory content around an operation, but the current authorization and effect state must still be evaluated.

AI agent compensation for failed tools separates confirmed effects from recovery. Semantic cache invalidation should never blur that ledger. Store response class such as informational, computed from immutable inputs, snapshot report, or effectful workflow. Default unknown classes to miss, and make reviewers approve any new cacheable class with concrete repeat-call tests.

Negative results deserve specific rules. No orders found can become stale the moment an order arrives; a rate-limit response should rarely be cached beyond its declared reset; a permission denial might be safe only under the same principal and policy epoch. Do not assign one TTL to successes and errors. Each terminal reason has a different invalidating event and user consequence.

Prevent stampedes without serving obsolete truth

When a popular entry expires, many concurrent requests can regenerate it. Use request coalescing or single-flight per exact identity and semantic cluster, with a bounded wait and independent cancellation. Stale-while-revalidate may be acceptable for low-risk editorial suggestions but not for prices, permissions, medical guidance, or changed policy. Declare the stale-serving classes rather than enabling the feature globally.

A cache stampede control must respect tenancy and authorization. Coalesce only requests whose hard scope matches. If regeneration fails, return the retained entry only when its stale policy allows and show freshness honestly; otherwise fall back to origin, alternate computation, or a temporary unavailable state. Availability cannot silently override a policy invalidation.

Admission control can defer expensive regeneration when the origin is saturated. Couple it to cache state so a miss does not trigger uncontrolled work, but preserve priority for policy and deletion events. Those events protect correctness and should not wait behind ordinary generation traffic. Measure queue age separately for reads, recomputes, and invalidations.

Test false neighbors, stale sources, and policy races

Build paired queries that are lexically close but differ in tenant, negation, date, product, jurisdiction, permission, or output format. Add paraphrases that should reuse under identical hard scope. Change one validator at a time and assert a miss.

Race a read against invalidation, replay duplicate events, disconnect one replica, and restore an old snapshot. Every test should end with a reason-coded hit or miss.

Measure false-hit rate, false-miss rate, hit latency, regeneration cost, invalidation lag, stale exposure time, coalesced waiters, origin failures, and user corrections. The RedisVL semantic cache guide provides an official implementation reference for semantic distance and cache behavior. Aggregate hit rate is not a quality metric: a system can achieve extraordinary reuse by returning yesterday's answer to the wrong person, so slice errors by consequence and keep zero-tolerance gates for cross-tenant and revoked-policy cases.

Use RAG access control before vector search when cached answers include retrieved evidence. Authorization must apply before semantic candidate retrieval, because even scoring a forbidden entry can leak its existence through timing or debug data. Treat cache vectors, metadata, explanations, and observability as protected surfaces, not only the final text.

Publish the complete invalidation contract

The cache receipt lists cacheable operation classes, hard identity dimensions, embedding and threshold, constraint verifier, payload schema, source validators, TTL by result class, invalidation tags, epoch rules, propagation target, replica behavior, stale policy, coalescing boundary, encryption, deletion, metrics, hostile fixtures, and origin-failure fallback. Semantic cache invalidation is a product contract, not a tuning knob hidden inside infrastructure.

Reject release if similarity can cross a hard scope, source changes do not revoke grounded claims, permission epochs lag without read denial, an effectful response is cacheable, stale policy is implicit, invalidation cannot reach every replica, or operators cannot explain a hit. RAG citations that survive document change offers stable source identity that can become a validator rather than a brittle URL.

The durable design is conservative: exact identity narrows the universe, semantics finds a candidate, validators establish freshness, policy confirms reuse, and a reasoned miss invokes bounded generation. That sequence may reduce hit rate compared with similarity alone. It also turns reuse from a clever demo into a system that can say which answer, for whom, from which facts, under which rules, until what event.

Semantic cache invalidation begins with exact scope before similarity proposes any reuse. Semantic cache invalidation must also bind every answer to source and policy versions. Operate semantic cache invalidation as a correctness boundary whose stale-hit cost is measured separately from cache efficiency.