Durable AI Agents That Resume Safely
Design checkpoints, effect ledgers, approval pauses, reconciliation, and recovery receipts so agent crashes never repeat side effects.
Durable AI agents must survive a worker crash, a week-long approval pause, and a retrying tool call without repeating an irreversible effect. The answer is to persist decisions around the model: checkpoints for intent, an effect ledger for tools, and recovery receipts that distinguish proposed, attempted, committed, and verified work.
This guide builds that boundary through one failure-injection walkthrough. It is for engineers designing long-running agent workflows whose correctness matters more than reproducing the model's exact prose after every restart.
The design connects durable execution, agent checkpoints, idempotent tools, and workflow replay without pretending that model text itself can be replayed deterministically.
Durable AI agents need a failure model
Durable AI agents operate across unreliable boundaries: model APIs time out, workers restart, queues redeliver, humans answer late, and tools can commit just before a connection fails. List those moments before choosing a workflow engine. Durability means every ambiguous boundary has a recoverable state.
Classify data into durable intent, reconstructable presentation, and external effect. The goal, approved plan, policy version, and effect identifiers must survive. A streaming explanation may be regenerated or replaced by a concise recovery message. A payment or deletion requires independent confirmation.
My position is that exact transcript replay is the wrong promise for most agent systems. The boundary is effect equivalence: after recovery, the workflow must preserve authorized outcomes, avoid duplicate side effects, and expose any uncertainty. Natural-language wording may differ when it is not itself the contractual artifact.
Write a table of failure points with before-state, possible outside state, and permitted recovery. If a row says simply retry, the design is incomplete. A retry is safe only when the receiving tool recognizes the same operation or the workflow proves that no prior commit occurred.
Persist state at semantic boundaries
Durable AI agents should checkpoint after decisions that would be expensive or unsafe to rediscover: normalized user intent, selected plan, policy result, human approval, tool-effect intent, observed result, and terminal outcome. Do not checkpoint every streamed token unless the product truly needs text-level continuation.
Each checkpoint needs a stable workflow ID, monotonically increasing revision, current owner epoch, parent revision, input digests, policy version, and next permitted transitions. Optimistic concurrency prevents two recovered workers from both believing they own the same step after a delayed lease renewal.
Keep large model context outside the core row and reference it by immutable digest with retention policy. Small control records remain quick to lock and inspect, while sensitive context can be encrypted, expired, or deleted according to its own lifecycle. Recovery should degrade honestly if optional context is gone.
A checkpoint is not completion. Store terminal states only after outcome verification or a documented accepted uncertainty. Otherwise dashboards and users will treat a persisted plan as evidence that the operation happened, reproducing the exact ambiguity durability was meant to remove.
Illustrative TypeScript — a reviewable design sketch, not a compiled production implementation.
type DurableStep = {
workflowId: string;
revision: number;
ownerEpoch: number;
state: "planned" | "approved" | "attempted" | "verified" | "blocked";
effectId?: string;
resultDigest?: string;
};
| Signal | Decision | Proof |
|---|---|---|
| No effect attempt | Resume planning | Checkpoint owns current epoch |
| Committed effect | Replay stored result | Idempotency receipt matches |
| Unknown outcome | Reconcile before retry | Independent tool read resolves state |
Walk through a crash after the tool commits
Durable AI agents become concrete with a simple fixture: an approved expense reimbursement. The workflow writes effect ID exp-1042-v1 and the exact amount before calling the payment tool. The tool commits, but the worker crashes before saving the returned transaction ID.
A replacement worker acquires a new ownership epoch and sees an attempted effect with unknown outcome. It does not create a new payment. It queries the tool using the original operation identifier, finds the committed transaction, stores the receipt, verifies amount and recipient, and resumes with a recovery message.
Now run three variations: crash before the call, tool rejects the call, and tool accepts but cannot be queried. The first may attempt safely with the same identifier; the second records a terminal failure or revised plan; the third enters reconciliation and asks for human review instead of gambling with another payment.
The inspectable output is an ordered ledger containing approval identity, policy version, attempt time, operation identifier, tool response or timeout, reconciliation reads, verified outcome, and owner epochs. It proves not just that the agent finished, but why a particular effect occurred once.
Make every consequential tool idempotent
Durable AI agents need tool adapters that accept a caller-generated operation identifier and bind it to a request fingerprint. Repeating the same identifier and fingerprint must return the stored status or result. Reusing the identifier with different arguments must fail visibly rather than silently changing meaning.
When a provider already supports idempotency, preserve its key through every queue and retry layer. When it does not, add a local outbox and reconciliation adapter where feasible. Some effects, such as sending a physical package, may require a domain-specific reservation or manual confirmation instead.
The Temporal documentation explains deterministic history re-execution and durable activity boundaries in that runtime. Use it as an implementation reference, while keeping the product-level effect contract explicit because an orchestration engine cannot make an unsafe third-party endpoint idempotent by itself.
The tradeoff is extra state and adapter work. The mitigation is proportionality: apply the strongest ledger to money, permissions, external messages, deletion, and publishing. Read-only search can tolerate lighter receipts, though it still benefits from request identity and observable completion.
Pause approvals without holding a worker
Durable AI agents should model approval as a durable waiting state, not a suspended process or open database transaction. Persist the proposal, consequence summary, exact tool arguments, approver rule, expiry, and policy version. Then release compute until an authenticated event resumes the workflow.
Bind the response to the proposal revision. If context or arguments change while waiting, invalidate the approval and request a new one. A late approval for revision seven must not authorize revision nine. Show expired, superseded, and already-used states clearly in the review interface.
The OpenAI Agents SDK guide describes durable integrations and resumable runs. Verify the installed SDK and chosen integration directly; the invariant here is independent of vendor: a pause must serialize enough state to review and resume safely.
Recovery messaging should say what survived and what will happen next: 'Your approval was recorded; payment status is being reconciled.' Avoid replaying a stale typing animation or claiming the agent is thinking. The interface is a view over durable state, not proof that the original process remains alive.
- CheckpointCheckpoint
Persist goal, policy, ownership epoch, and pending action.
- AttemptAttempt
Write the effect intent before crossing the tool boundary.
- ReconcileReconcile
Read tool state after a timeout or crash.
- ResumeResume
Continue from a verified result or explicit human decision.
Design the recovery interface around uncertainty
Durable AI agents need visible states for waiting, retry scheduled, reconciling, recovered, completed, failed, cancelled, and needs review. Pair each with last confirmed event, current owner or system action, and the next available user control. A generic spinner hides the exact information recovery requires.
When effect status is unknown, disable any duplicate submit path and offer a bounded refresh or escalation. Explain that the system is checking the provider. If a safe compensating action exists, present it only after the original outcome is known; compensation is a new effect, not an eraser.
Keep the activity timeline editorial. Group noisy heartbeats and automatic retries beneath the semantic step, while exposing technical receipts on demand. The reader should see proposed, approved, attempted, reconciled, and verified in order without scanning every queue delivery.
Accessibility follows the same truth model. Announce state changes once, keep focus on the relevant recovery control, and avoid live-region chatter for background polling. Core history and controls must remain available as semantic HTML when animation or client scripting is unavailable.
Inject failures at every ownership edge
Durable AI agents should be tested by terminating the worker before and after each durable write and external call. Redeliver queue messages, delay approvals, expire leases, return inconsistent provider reads, and allow an old worker to wake after a new owner has advanced the workflow.
Assert invariant outcomes rather than exact prose: at most one committed payment, no unapproved arguments, monotonically increasing revisions, one current owner epoch, and a terminal receipt that matches the provider. Run the same fixture under several valid event schedules.
A stale owner is a named failure mode. Its consequence is concurrent plans or duplicated effects after a lease race. Mitigate it with fencing tokens carried into state writes and, where supported, tool calls. Reject any write whose epoch is lower than the durable workflow record.
Preserve the injected schedule and resulting ledger when a test fails. Random chaos without a replayable event plan produces stories, not evidence. A compact deterministic fixture should run in CI; broader chaos runs can then explore schedules and save any novel counterexample.
Release with a recovery proof
Durable AI agents are ready when every consequential effect has stable identity, fingerprint checking, and reconciliation; every approval binds to a revision; ownership is fenced; and the crash matrix produces the same authorized outside state across valid schedules.
Measure duplicate-effect rate, unknown-outcome age, recovery completion time, expired approvals, and manual reconciliation volume. Zero observed duplicates is necessary but not enough; exercises must deliberately create the dangerous timing window and prove the ledger blocks a second commit.
Roll out one workflow and one effect adapter at a time. Keep a stop switch that prevents new effects while permitting reconciliation and read-only recovery. A broad platform launch before one end-to-end fixture passes merely distributes ambiguous state more efficiently.
The release packet should contain the state model, effect contract, approval specimen, crash schedule, ledger output, UI captures, and operator runbook. Together they let engineering, design, security, and support reason about the same recovery truth.
Put the method into practice
Durable AI agents recover safely when control state is checkpointed at semantic boundaries and external effects are identified, reconciled, and verified independently. The durable system is not the transcript; it is the authorized state transition plus evidence about what happened outside the worker.
Start with the reimbursement crash fixture. Kill the worker after the tool commits but before the result is stored, then prove that a replacement worker discovers the existing payment and reaches a verified outcome without a second effect.
The narrow method connects to four existing Journal notes: multi-agent testing with causal traces, idempotency-key lifecycle contracts, background-job completion receipts, auditable AI agent activity logs. They cover adjacent implementation boundaries without changing this article's single search intent. Preserve the worked fixture, its visual evidence, and the release receipt so a later update can compare behavior instead of relying on memory.