AI Agent Fault Injection: A Recovery Lab

AI agent fault injection answers a harder question than whether the happy path can call a tool: what happens when the tool commits an effect and the reply disappears? This lab makes failure timing deterministic, checks idempotency and cancellation, and turns every recovery decision into a replayable receipt.

AI agent fault injection begins with timing

A fault name such as “timeout” is too vague for an agent test. A timeout before a request leaves the process, during remote work, after the effect commits, or while the result is being persisted creates four different recovery obligations. Write the failure point on a sequence diagram before choosing retry behavior.

Start with one workflow that has a visible consequence: reserve inventory, charge a payment, or publish a document. Mark proposal, authorization, tool dispatch, remote commit, response receipt, state persistence, user notification, and terminal status. AI agent fault injection can now target the edge between any two events instead of randomly killing a process and hoping the outcome is informative.

The test oracle must observe both workflow state and external reality. A trace saying “failed” is wrong if the charge exists; a trace saying “complete” is wrong if the reservation vanished. Read the provider through an independent endpoint or fixture ledger. That second observation distinguishes communication failure from effect failure and establishes what recovery must preserve.

Build a deterministic fault switchboard

Wrap every side-effect boundary with a test adapter that accepts a fault plan. The plan names operation, attempt, phase, fault type, delay, and whether the underlying effect happened. Keep production behavior unchanged when no plan is present, and make fault activation impossible outside explicit test tenants or local fixtures.

The AWS guidance on timeouts, retries, and jitter explains why retries amplify load and why timeout placement matters. The OpenTelemetry trace specification supplies causal identifiers, while Jepsen analyses model histories against externally visible outcomes. Together they frame a workflow fault model without claiming that a trace alone proves correctness.

Give the switchboard a seed and a finite schedule: tool A times out after commit on attempt one; state write B rejects on attempt two; cancellation arrives before notification. Determinism is essential. Agent chaos testing that cannot replay a failure produces drama, not evidence, and makes it impossible to distinguish a code fix from a lucky run.

Attack tool results before tool availability

Tools fail while returning successful HTTP statuses too. Inject missing required fields, invalid enums, stale versions, duplicated items, oversized payloads, conflicting identifiers, and plausible but semantically impossible values. Runtime validation should stop these results before they enter planning state or long-term memory.

For each malformed case, record raw response digest, schema version, validation issue, redaction, retry eligibility, user-facing state, and terminal outcome. AI agent fault injection should verify that a parser error does not become a confident model summary. The agent may explain that it could not verify the result, but it must not invent a substitute effect.

Separate transport retry from semantic repair. A truncated response might be safe to retrieve again; a completed payment with an unfamiliar status may require a read-by-idempotency-key rather than another create call. The artifact below demonstrates the important timeout-after-commit branch. Its ledger is deliberately external to the workflow so the second attempt can discover the first effect.

The runnable fixture injects a timeout after the external effect already exists, then proves that retrying with the same idempotency identity returns one effect rather than creating another.

Runnable artifact — agent-fault-injection.test.mjs

import assert from "node:assert/strict";
const ledger=new Map();let calls=0;const tool=({key})=>{calls++;if(calls===1)throw new Error("timeout-after-commit");if(!ledger.has(key))ledger.set(key,{charge:42});return ledger.get(key)};
ledger.set("order-7",{charge:42});const run=async()=>{for(let attempt=1;attempt<=2;attempt++)try{return tool({key:"order-7"})}catch(error){if(attempt===2)throw error}};
assert.deepEqual(await run(),{charge:42});assert.equal(ledger.size,1);assert.equal(calls,2);
console.log("PASS: injected faults preserve one effect");

Run node agent-fault-injection.test.mjs. Expected receipt: PASS: injected faults preserve one effect.

Exercise retries against one effect identity

Attach a stable idempotency identity to the user-approved intent, not to an individual network attempt. Retries carry the same identity and request fingerprint. If the material request changes, open a new intent and ask for authorization again rather than quietly reusing a key whose replay contract no longer matches.

Test before-send failure, timeout-before-commit, timeout-after-commit, duplicate delivery, concurrent duplicate, and late response. For each case, assert external effect count, returned result identity, workflow terminal state, and notification count. Idempotent recovery means these four observations converge; it does not merely mean that the client avoids an exception.

AI agent fault injection also needs a retry budget. Bound attempts, elapsed time, and downstream work; add exponential backoff and jitter where a real shared service is involved. A model should not decide to “try again” forever. The orchestration layer owns retry classification and budget, while the agent receives a structured status it can communicate without overriding operational policy.

Corrupt state at durable boundaries

State failures expose whether the workflow can resume or only restart. Inject a crash after tool commit but before checkpoint, a partial checkpoint, an old schema, a duplicated event, and an out-of-order callback. Then restore from the durable record and compare the resumed history with the external effect ledger.

The failure matrix should name which fields are authoritative. A task status may be reconstructed from events; an approval must retain actor and policy version; a tool effect needs remote identity; a notification can be regenerated only if its deduplication key survives. Tool failure simulation without this ownership map tends to patch whichever row looks empty.

For AI agent fault injection, reject checkpoints that cannot explain their version or migration. A corrupted optional display field may use a fallback, while missing effect identity must stop automated recovery. Keep the raw invalid record for diagnosis and write a new repaired version rather than mutating incident history in place. That discipline turns state repair into a reviewable transition.

Cancel work across every active branch

Cancellation is a causal event, not a UI flag. Inject it while the model is generating, a tool is queued, a request is in flight, a retry timer is sleeping, and a compensating action is running. Each component needs a defined response: abort, finish safely, detach with monitoring, or refuse cancellation because commitment has passed.

Observe what the user can still rely on. If a request cannot be canceled after remote commit, the workflow should stop subsequent work, retrieve the committed outcome, and display it honestly. If compensation begins, show its pending state and verify the result independently. Never label an operation canceled while its consequence remains unknown.

Use AI agent fault injection to race cancel against completion repeatedly with a virtual clock. Assert one terminal state and one visible notification. When both callbacks arrive, a compare-and-set transition or event ordering rule chooses the winner. The losing branch still records what it observed, which prevents a late tool response from silently reviving work that the user already abandoned.

Read the causal trace as a recovery proof

A useful failure trace connects approved intent, attempt IDs, idempotency key, injected phase, remote effect identity, checkpoint versions, retries, cancellation, compensation, and final verification. Redact sensitive payloads but keep stable digests and typed outcomes. The objective is to reconstruct public causality, not private model reasoning.

Adjacent Journal controls deepen individual layers: durable AI agent execution defines resumable state, AI agent compensation for failed tools handles irreversible effects, multi-agent testing with causal traces preserves ownership across actors, and idempotency keys need lifecycle contracts specifies replay semantics.

Review traces by asking whether an uninvolved operator can answer five questions. What did the user authorize? Which effect actually occurred? Why was a retry or compensation allowed? What evidence set the terminal state? What remains uncertain? If those answers require reading model chain-of-thought, the system is missing operational evidence.

AI agent fault injection records branch identifiers beside every span so a reviewer can distinguish one resumed path from a duplicated tool effect after cancellation races.

Gate release on a failure coverage ledger

Create a coverage ledger with workflow edges on one axis and fault classes on the other. Include transport loss, provider rejection, malformed success, timeout after commit, state write failure, duplicate callback, cancellation race, quota exhaustion, and observability loss. Not every cell needs injection, but every cell needs an explicit disposition and owner.

The release drill runs representative cases with fixed seeds and retains traces, external ledger snapshots, terminal-state assertions, and cleanup receipts. AI agent fault injection fails the gate when it duplicates an effect, hides a committed outcome, loses approval evidence, exceeds its retry budget, or cannot reach a verified terminal state. A polished error message cannot compensate for those defects.

Promote new incident shapes into the corpus within the postmortem, then rerun old cases after orchestration changes. The lab becomes a memory of how the system fails, expressed as executable histories rather than folklore. That is the point of forward engineering here: recovery behavior is designed before unreliable networks and tools choose the timing for you.

Turn injected failure into a release receipt

The most valuable agent demo is not a flawless tool call; it is a failed call whose effect, authority, retries, and final state remain intelligible. Keep the fault switchboard close to the workflow so each new capability arrives with its recovery story already executable. Preserve the external effect ledger with every test run so a future reviewer can distinguish a repaired trace from a merely reassuring status label.