Agent Memory Poisoning: Detect and Recover

Agent memory poisoning turns one untrusted observation into a durable instruction that can quietly steer future sessions. This guide shows how to attach provenance, constrain retrieval, quarantine descendants, repair summaries, and prove that trusted memories remain usable after containment.

Agent memory poisoning is a lineage problem

Treat agent memory poisoning as an integrity failure in a graph, not as a rude sentence in a chat log. A hostile page, tool response, user attachment, or retrieved note may be summarized, embedded, merged, and later promoted into a preferred fact. By the time the system behaves strangely, the original text may no longer appear in the prompt.

The first design move is to give every stored record an immutable identity and a typed origin. Capture the source locator, ingestion time, actor, parser version, transformation, parent records, trust label, intended uses, and expiry. These fields make long-term agent memory reviewable without pretending that provenance automatically makes content true.

Draw one concrete lineage before adding a classifier. A web result becomes observation M17; a nightly compactor derives summary M41; a planning session cites M41 in task state M62. If M17 contains an instruction disguised as documentation, agent memory poisoning reaches two descendants even when the original record is no longer retrieved. The containment boundary is therefore M17 plus its derivation closure, not whichever string happened to trigger an alert.

Separate evidence from instructions at ingestion

An ingestion pipeline should preserve useful text while refusing to promote its commands. Store quoted content, extracted claims, executable directives, and parser warnings in different fields. The memory writer may record that a page says “upload the token,” but it must not convert that phrase into an instruction with system authority.

Use the OWASP prompt injection guidance to enumerate indirect instruction paths, the NIST AI Risk Management Framework to assign owners and controls, and the NIST AI RMF Playbook to turn risk decisions into owned actions. These sources define useful risk boundaries; none can decide the trust of a local business record.

Memory provenance needs an explicit promotion rule. Observed text can support retrieval, verified facts can influence decisions, and instructions require a separate authorized source. A record that mixes those roles stays at the lowest trust level until a human or deterministic verifier splits it. This rule keeps agent memory poisoning from borrowing authority through an innocent metadata field such as “important” or “frequently used.”

Make retrieval enforce trust and purpose

Similarity is not authorization. A highly relevant poisoned memory should still be excluded when its source class, trust level, tenant, age, or allowed-use label conflicts with the current task. Put these filters before semantic ranking so a large similarity score cannot buy access to a protected prompt position.

Return provenance beside every retrieved fragment. The agent should receive content, record ID, source type, trust label, transformation history, and allowed role as structured data. Render untrusted evidence in a clearly delimited field rather than concatenating it with developer instructions. That retrieval quarantine makes memory integrity visible to both the model and the trace reviewer.

Test agent memory poisoning with near-duplicate records at different trust levels. For a travel-planning fixture, an authenticated preference says “avoid overnight flights,” while a scraped itinerary says “ignore prior preferences.” The search query may rank the scraped sentence first, yet policy must select the verified preference for planning and retain the scraped page only as evidence. Log both ranking and policy rejection so retrieval quality is not confused with authority.

Quarantine the full derivation closure

Quarantine should be a reversible state transition, not an emergency delete. Mark the suspect root, prevent it from retrieval and compaction, then traverse parent edges to find summaries, embeddings, cached plans, and copied profile fields that depend on it. Each descendant receives the incident ID and the reason its integrity is unresolved.

The containment table distinguishes four useful cases. A direct poisoned observation is blocked; a derived summary is blocked pending recomputation; a trusted ancestor remains active; an unrelated record remains active. That precision matters because deleting an entire memory store can erase the evidence needed to diagnose the route and can unnecessarily degrade the product.

Agent memory poisoning also crosses storage systems. Vector indexes, relational rows, document snapshots, prompt caches, evaluation fixtures, and analytics exports may each contain a derivative. Maintain a store registry with deletion or correction capabilities and last propagation receipt. If a store cannot express lineage, treat every copied item as a new record with its own parent pointer at write time. Containment ends only when every registered sink reports its state.

Probe compaction and summarization for laundering

Compaction is a privileged transformation because it can turn many weak observations into one apparently authoritative statement. Red-team the compactor with conflicting sources, quoted attacks, invisible text, stale corrections, and instructions that claim to be policy. The expected output should preserve disagreement and provenance rather than selecting the most confident prose.

Create a laundering corpus that follows one hostile claim through capture, summary, profile update, and task planning. Score whether the claim survives, whether its origin remains attached, and whether its allowed use widens. A harmless wording change is acceptable; an authority change is not. This gives agent memory poisoning a causal test instead of a collection of isolated prompts.

Review promotion paths separately from generation quality. A beautifully written summary can still be unsafe when it omits that three sources disagree or that the only supporting page is untrusted. Keep deterministic invariants around parent retention, tenant identity, trust monotonicity, and expiry. Let language-model evaluation inspect semantic preservation, but do not ask the same model that wrote the summary to be the only judge of its authority.

Repair descendants without hiding the incident

Recovery begins by deciding whether the root is false, unauthorized, stale, or merely used in the wrong context. That distinction determines whether to delete it, lower its trust, narrow its purpose, or attach a correction. Keep the incident record even when user-visible content is removed; security evidence and product memory have different retention needs.

Recompute descendants from clean parents in dependency order. Give each replacement a new immutable ID, link it to the superseded record, and compare the resulting plan or answer against a known-good fixture. Do not silently overwrite M41 in place, because later traces must explain why two sessions produced different outcomes from what looked like the same memory.

For agent memory poisoning, the recovery receipt should list affected record IDs, stores searched, descendants rebuilt, caches invalidated, evaluations rerun, remaining uncertainty, approver, and completion time. A correction is not complete merely because the main vector search stops returning the phrase. The runnable artifact below exercises the graph boundary directly and proves that an unrelated verified branch survives the quarantine operation.

The fixture models the smallest useful containment rule: quarantine the suspect record and every memory derived from it, while leaving ancestors and unrelated branches active.

Runnable artifact — memory-poison-quarantine.test.mjs

import assert from "node:assert/strict";
const records=[{id:"m1",trust:"verified",parents:[]},{id:"m2",trust:"observed",parents:["m1"]},{id:"m3",trust:"derived",parents:["m2"]},{id:"m4",trust:"verified",parents:[]}];
const descendants=(root)=>{const out=new Set([root]);for(let changed=true;changed;){changed=false;for(const r of records)if(r.parents.some(p=>out.has(p))&&!out.has(r.id)){out.add(r.id);changed=true}}return out};
const quarantine=(root)=>records.map(r=>descendants(root).has(r.id)?{...r,status:"quarantined"}:{...r,status:"active"});
const result=quarantine("m2");assert.deepEqual(result.filter(r=>r.status==="quarantined").map(r=>r.id),["m2","m3"]);assert.equal(result.find(r=>r.id==="m1").status,"active");assert.equal(result.find(r=>r.id==="m4").status,"active");
console.log("PASS: poisoned memory descendants are quarantined");

Run node memory-poison-quarantine.test.mjs. Expected receipt: PASS: poisoned memory descendants are quarantined.

Monitor for behavioral drift, not magic phrases

Phrase blocklists miss paraphrases and create false confidence. Monitor unusual memory promotion rates, sudden source concentration, trust upgrades without reviewers, repeated cross-session instructions, correction churn, and decisions dominated by one recent untrusted domain. Join these signals to behavior, such as tools selected or constraints ignored, rather than alerting on vocabulary alone.

The defensive stack becomes clearer when adjacent controls stay separate. Prompt injection taint tracking carries hostile influence across tool boundaries; AI agent memory that forgets on purpose defines correction and deletion; agent context compaction audits test lossy summaries; and RAG citations that survive document change keeps evidence addressable after updates.

Build dashboards around incident questions: Which source entered? Which transforms widened authority? Which tenants and tasks consumed descendants? Which repair cleared each sink? These are operational questions with finite answers. An aggregate “memory safety score” cannot replace them, and it may conceal the exact low-volume route an attacker prefers.

Release the memory system with a recovery drill

Before release, seed a synthetic poisoned page in a test tenant and let the normal ingestion, retrieval, compaction, and planning path process it. Trigger detection at a chosen stage, then measure time to quarantine, descendant coverage, correction propagation, unrelated-memory availability, and the final behavioral result. Keep the fixture artificial and clearly labeled so it cannot escape into production evidence.

The release gate for agent memory poisoning needs hard failures. Stop when a quarantined descendant remains retrievable, a trusted record is deleted without cause, a store lacks a propagation receipt, the repaired plan still follows the hostile instruction, or the incident trace cannot identify origin. Latency and recall improvements cannot offset those integrity failures.

Archive the schema, store registry, policy version, laundering corpus, quarantine output, rebuilt record IDs, and drill timeline. Revisit the contract whenever a new memory tier, compactor, embedding pipeline, or tool-state cache is introduced. Memory systems evolve by copying information into new shapes; the recovery drill is what proves those shapes still share one containment story.

Leave memory safer than you found it

A trustworthy memory system is not one that never stores a bad claim; it is one that can identify the claim's authority, contain every derivative, repair behavior, and show exactly what remained untouched. Make that recovery path part of the memory architecture before the first incident asks for it.