AI Agent Circuit Breakers That Fail Closed

AI agent circuit breakers prevent one failing tool from turning a bounded agent run into a retry storm. This guide builds dependency-scoped closed, open, and half-open states that reject ambiguous calls safely while preserving useful graceful degradation.

AI agent circuit breakers start with one dependency

An agent rarely fails as one indivisible machine. It calls a search index, model, browser, billing service, and write API, each with a different timeout and recovery rule. Put one breaker around the entire run and a sick optional dependency can silence healthy work. Put one around every function without a dependency identity and the state fragments into meaningless counters. AI agent circuit breakers should therefore be keyed by the smallest remote dependency whose health can be observed and whose calls can be stopped together.

Start with a boundary sheet: dependency name, operation class, deadline, retryability, fallback, and owner. A read-only catalog lookup can share a breaker across equivalent calls; a payment mutation should not inherit that state or retry policy. The breaker records outcomes after the operation-level timeout, while the agent run keeps its independent wall-clock and tool-call budget. These two controls answer different questions: whether the remote is healthy enough to call, and whether this run may continue.

The downloadable lab uses an illustrative event trace rather than production telemetry. It creates separate search and billing breakers, runs success and failure events against a monotonic synthetic clock, and emits every transition. This proves the state-machine contract and tool failure isolation for the fixture. It does not estimate a provider outage rate or recommend universal thresholds.

Model closed, open, and half-open states explicitly

The closed state permits calls and counts qualifying failures inside a declared observation window. Once the threshold is reached, the open state rejects new calls locally until a cooldown deadline. Half-open permits only a small probe budget; enough successful probes close the breaker, while any qualifying failure reopens it. AI agent circuit breakers fail closed when missing state, invalid configuration, or an ambiguous probe outcome denies the risky call instead of silently treating it as healthy.

The AWS circuit breaker pattern describes the closed, open, and half-open progression and warns that recovery needs monitoring. Translate that pattern into named events, not a boolean. The receipt should show previous state, input event, next state, reason, cooldown deadline, and whether a remote call was attempted. A local rejection is then distinguishable from a remote failure in traces and user-facing messages.

Keep state transitions deterministic. Use a monotonic time source, define whether the threshold event itself opens the circuit, and state whether probe success resets the prior failure count. The fixture in this article opens on the third qualifying failure, rejects before the cooldown, admits one half-open probe at the deadline, and closes only after two successes. Those are teaching values, not operational defaults; the important result is that every boundary is visible and replayable.

Stop agent retry storms before they amplify

Agent retry storms spend capacity on an already struggling dependency. When nested layers each retry, one user action can multiply into many remote attempts, extend queue time, and make recovery harder. The Amazon Builders’ Library guidance on timeouts, retries, backoff, and jitter explains why retries need limits and why retry behavior belongs at one chosen layer. AI agent circuit breakers add a second boundary: once failure evidence crosses the threshold, new runs stop attempting that dependency during the cooldown.

Draw a retry budget before configuring a breaker. Decide which errors qualify, the maximum attempts for one logical operation, backoff with jitter, and the end-to-end deadline. Authentication failure, invalid arguments, and policy refusal generally should not be retried as availability failures. Rate limits may require a server-provided delay. Unknown outcomes on non-idempotent writes should stop and reconcile rather than replay. The companion guide on compensating failed agent tools covers that business-effect boundary.

The second figure compares two illustrative cascades with identical input events. The retry-only path continues issuing attempts after repeated failures; the breaker path opens and produces local rejections until its probe window. The counts are derived from the committed fixture and labelled synthetic. They demonstrate multiplication and containment under one policy, not the latency or availability of any real service.

Choose failure evidence without hiding recovery

A breaker should react only to evidence that predicts another attempt is unsafe or wasteful. Transport errors, deadline exhaustion, and explicit transient server failures may qualify. Validation errors, permission denials, and user cancellations usually should not. Mixed classes demand separate counters or a policy table, because treating every unsuccessful result as an outage turns product mistakes into false health signals. AI agent circuit breakers are only as trustworthy as this classifier.

Microsoft’s circuit breaker pattern guidance emphasizes that the pattern differs from retry and that half-open behavior tests whether the failure has been corrected. Make the classifier and recovery probe part of the reviewed configuration. A probe should be bounded, low-cost, side-effect-safe, and representative enough to say something about recovery. A synthetic health endpoint that bypasses the failing storage path can close the breaker too early.

Use fault injection for agent systems to exercise timeout, malformed response, overload, credential failure, and late success separately. The lab includes a late result after a local timeout and records it as ambiguous rather than success. That choice prevents a delayed response from erasing failure evidence. It also forces the caller to reconcile any possible side effect before another mutation is admitted.

Make half-open probes scarce and observable

If every waiting agent becomes a probe when cooldown expires, the half-open state recreates the surge that the breaker was meant to stop. Admit a fixed number of leases, attach an expiry, and reject or degrade all other calls until those leases resolve. AI agent circuit breakers should record probe ownership so a crashed worker cannot leave the dependency permanently half-open. A lease that expires is a failed or unknown probe according to the operation’s effect semantics.

Probe success needs a quorum rule. One safe choice is two consecutive successful probes before closing, with any qualifying failure returning directly to open. Another is a capped success ratio over a small set. Whichever rule you choose, freeze it alongside cooldown, threshold, and observation window. Configuration changes should create a new policy version in receipts instead of rewriting the meaning of old transitions.

The lab models two half-open slots but grants them serially for clarity. A request at the cooldown boundary receives a probe lease; a simultaneous request gets graceful degradation. After the first success the state remains half-open, and after the second it closes. Change the event ordering and the receipt hash changes. This same-input replay property makes race assumptions visible, though the single-process fixture is not a proof of distributed locking or clock correctness.

Design graceful degradation as a product state

Opening a circuit is an engineering decision; what the person experiences is a product decision. Optional retrieval might fall back to a cached result labelled with its age. A research action might save a draft and offer retry later. A destructive or financial action might stop completely and present a reconciliation identifier. AI agent circuit breakers need a degradation matrix that distinguishes safe omission, stale read, queued work, human handoff, and hard stop.

Do not present a local breaker rejection as if the remote service just failed. The breaker may be acting on earlier evidence. Tell the user which capability is temporarily unavailable, what work was preserved, whether any side effect is uncertain, and the next safe action. This is where a technically correct breaker becomes understandable rather than mysterious. It also keeps the agent from inventing an answer after a required evidence source is unavailable.

Coordinate sibling tools with structured concurrency. If a required dependency opens, cancel related work whose result can no longer be used, while allowing independent evidence gathering to finish when it still has value. Record which branches were cancelled by policy and which degraded. The result is a bounded partial outcome, not a blanket “agent failed” state that loses the useful work already completed.

Run the dependency-scoped breaker lab

The public Node artifact implements the exact transition function described here. Its configuration validates finite positive thresholds and cooldowns, rejects unknown event types, and defaults an unrecognized persisted state to open. It then replays a synthetic search failure sequence and an independent healthy billing sequence. AI agent circuit breakers pass the fixture only if search opens, early calls are locally rejected, half-open probes are bounded, and billing remains closed.

The receipt contains configuration, normalized input events, per-dependency transitions, remote-attempt counts, local-rejection counts, final states, and a SHA-256 digest. The independent test runs the artifact twice and requires byte-equivalent JSON. It changes one failure to success and requires a different final trace. Boundary cases cover the exact cooldown instant, while hostile cases cover invalid thresholds, unknown events, and corrupted persisted state.

This lab deliberately avoids network calls, wall-clock time, and shared storage. Its evidence is the state machine, not production resilience. A distributed implementation still needs atomic leases, policy distribution, cardinality limits, metrics, and operational review. Run the fixture beside those integration tests so a seemingly small threshold or classifier change produces a reviewable transition diff before deployment.

Runnable artifact — Deterministic dependency-scoped closed, open, and half-open transitions; not a production reliability result.

import assert from "node:assert/strict";
import { createHash } from "node:crypto";
const sha=value=>createHash("sha256").update(JSON.stringify(value)).digest("hex");
const config={failureThreshold:3,cooldownMs:20000,successThreshold:2,probeLimit:1};
function validateConfig(c){for(const key of ["failureThreshold","cooldownMs","successThreshold","probeLimit"])if(!Number.isInteger(c[key])||c[key]<1)throw new Error("invalid-config:"+key)}
function normalize(raw){return ["closed","open","half-open"].includes(raw?.state)?structuredClone(raw):{state:"open",failures:0,probeSuccesses:0,openedAt:0,probes:0,recoveredFromCorruption:true}}
function step(raw,event){const current=normalize(raw);if(!["request","success","failure"].includes(event.type)||!Number.isFinite(event.at))throw new Error("invalid-event");let next={...current},attempted=false,rejected=false,reason="";if(event.type==="request"){if(next.state==="open"&&event.at-next.openedAt<config.cooldownMs){rejected=true;reason="cooldown"}else if(next.state==="open"){next={...next,state:"half-open",probes:1,probeSuccesses:0};attempted=true;reason="probe-admitted"}else if(next.state==="half-open"&&next.probes>=config.probeLimit){rejected=true;reason="probe-limit"}else{attempted=true;reason="call-admitted"}}else if(event.type==="failure"){next.failures+=1;if(next.state==="half-open"||next.failures>=config.failureThreshold)next={...next,state:"open",openedAt:event.at,probes:0,probeSuccesses:0};reason="qualifying-failure"}else{if(next.state==="half-open"){next.probeSuccesses+=1;next.probes=0;if(next.probeSuccesses>=config.successThreshold)next={...next,state:"closed",failures:0,probeSuccesses:0}}else next.failures=0;reason="success"}return{next,attempted,rejected,reason}}
function replay(events,initial={state:"closed",failures:0,probeSuccesses:0,openedAt:0,probes:0}){let state=initial;const transitions=[];for(const event of events){const result=step(state,event);transitions.push({event,from:state.state,to:result.next.state,attempted:result.attempted,rejected:result.rejected,reason:result.reason});state=result.next}return{transitions,final:state,remoteAttempts:transitions.filter(x=>x.attempted).length,localRejections:transitions.filter(x=>x.rejected).length}}
validateConfig(config);
const standard=[{type:"request",at:0},{type:"failure",at:5},{type:"request",at:10},{type:"failure",at:15},{type:"request",at:20},{type:"failure",at:25},{type:"request",at:1000},{type:"request",at:20025},{type:"success",at:20030},{type:"request",at:20031},{type:"success",at:20035}];
const recovered=[...standard];if(process.argv.includes("--recovery"))recovered[5]={type:"success",at:25};
const search=replay(recovered),billing=replay([{type:"request",at:0},{type:"success",at:3}]);
const hostile={invalidConfig:"",unknownEvent:"",corruptState:step({state:"corrupt"},{type:"request",at:1})};try{validateConfig({...config,cooldownMs:0})}catch(error){hostile.invalidConfig=error.message}try{step({}, {type:"mystery",at:0})}catch(error){hostile.unknownEvent=error.message}
assert.match(hostile.invalidConfig,/invalid-config/);assert.match(hostile.unknownEvent,/invalid-event/);assert.equal(hostile.corruptState.next.state,"open");
const core={schema:"ai-agent-circuit-breaker-receipt-v1",fixture:"synthetic dependency events",config,dependencies:{search,billing},hostile,claimBoundary:"Deterministic state transitions and local rejection only; not a distributed lease, outage forecast, or production reliability result."};
console.log(JSON.stringify({...core,receiptHash:sha(core)},null,2));console.log("PASS: dependency-scoped breaker transitions, bounded probes, isolation, hostile controls, and digest verified");

Ship AI agent circuit breakers with receipts

A useful breaker dashboard separates remote attempts from local rejections. It shows state duration, opens by reason, probe results, degradation path, and affected operation class. Alerting on total failures alone can make an open circuit look healthier because it intentionally suppresses calls. AI agent circuit breakers need both demand and admitted-attempt counters to reveal that distinction.

Treat breaker configuration as versioned product policy. Review dependency identity, threshold, observation window, cooldown, probe lease, error classifier, retry budget, fallback, and user message together. Pair every deployment with a controlled exercise and preserve the normalized receipt. Historical receipts explain whether fewer attempts came from recovery, lower demand, or an open circuit.

The strongest success condition is not “the circuit opened.” It is that repeated failure stopped consuming the remote, healthy dependencies continued, uncertain effects were not replayed, and the person received an honest next step. That is the fail-closed promise this article makes. Run the downloadable event trace, mutate one event, and use the resulting diff as the first review artifact for your own breaker policy.