Error Recovery and Retry Strategies in Agent Loops
When an orchestrator hands work to a subagent or a tool, that call can fail outright, hang forever, or come back with a result that looks fine but is wrong or incomplete. This page covers how to detect each of those failure modes inside an agent loop and respond with retries, fallbacks, and circuit breakers, so one bad call doesn't take down the whole run.
Summary
An agent loop that delegates to subagents or tools has a new failure surface beyond "the API call errored": a subagent can stall and never return, or return a result that's technically successful but useless.
Retrying blindly is not a strategy. A retry only helps for transient failures, it makes a broken dependency worse by hammering it, and it can duplicate side effects if the underlying call wasn't idempotent.
This page treats orchestration-level error recovery as three layered decisions: retry the same call a bounded number of times with backoff, fall back to a cheaper or different path when retries are exhausted, and stop calling a dependency altogether once it has clearly failed past a threshold, revisiting it later.
It also covers detecting a stalled subagent, one that never raises an error and never returns, since that failure mode needs an explicit timeout rather than an exception handler.
This is scoped to failures inside the orchestration boundary itself: a subagent invocation, a tool call, a stalled child loop. It is not about the mechanics of retrying a raw Claude API call, which Retry and Exponential Backoff Strategies for Transient Claude API Failures covers separately.
Recipe
Quick-reference recipe card - copy-paste ready.
import asyncio
import random
from claude_agent_sdk import query, AgentOptions, SubagentConfig
async def call_subagent_with_retry(prompt, options, *, max_attempts=3, timeout=45.0):
for attempt in range(1, max_attempts + 1):
try:
return await asyncio.wait_for(
run_subagent(prompt, options), timeout=timeout
)
except (asyncio.TimeoutError, RuntimeError):
if attempt == max_attempts:
raise
delay = (2 ** (attempt - 1)) + random.uniform(0, 0.5)
await asyncio.sleep(delay)
async def run_subagent(prompt, options):
result = None
async for message in query(prompt=prompt, options=options):
if message.get("type") == "subagent_result":
result = message
if result is None:
raise RuntimeError("subagent produced no result")
return resultWhen to reach for this:
- A subagent invocation or tool call can fail transiently and a second attempt would likely succeed.
- A subagent can hang without raising an exception, so you need an explicit timeout to detect a stall.
- Repeated failures against the same dependency should stop generating more load once it's clearly down.
- You want a cheaper or simpler fallback path when the primary approach keeps failing, instead of failing the whole run.
Working Example
import asyncio
import random
import time
from dataclasses import dataclass, field
from claude_agent_sdk import query, AgentOptions, SubagentConfig
class CircuitOpenError(Exception):
"""Raised when a circuit breaker refuses a call because its dependency is unhealthy."""
@dataclass
class CircuitBreaker:
"""Per-dependency circuit breaker with a cooldown-based half-open probe."""
failure_threshold: int = 3
cooldown_seconds: float = 30.0
_failures: int = field(default=0, init=False)
_opened_at: float | None = field(default=None, init=False)
def before_call(self) -> None:
if self._opened_at is None:
return
elapsed = time.monotonic() - self._opened_at
if elapsed < self.cooldown_seconds:
raise CircuitOpenError(
f"circuit open, retry in {self.cooldown_seconds - elapsed:.1f}s"
)
# Cooldown elapsed: let exactly one half-open probe through below.
def record_success(self) -> None:
self._failures = 0
self._opened_at = None
def record_failure(self) -> None:
self._failures += 1
if self._failures >= self.failure_threshold and self._opened_at is None:
self._opened_at = time.monotonic()
async def run_subagent_with_timeout(prompt: str, options: AgentOptions, *, per_message_timeout: float = 20.0):
"""Consume query() messages, raising TimeoutError if the stream stalls between messages."""
stream = query(prompt=prompt, options=options).__aiter__()
result = None
while True:
try:
message = await asyncio.wait_for(stream.__anext__(), timeout=per_message_timeout)
except StopAsyncIteration:
break
if message.get("type") == "subagent_result":
result = message
if result is None:
raise RuntimeError("subagent produced no usable result")
return result
async def call_with_retry(make_call, *, max_attempts: int = 3, base_delay: float = 1.0):
"""Retry an async call with jittered exponential backoff."""
last_error: Exception | None = None
for attempt in range(1, max_attempts + 1):
try:
return await make_call()
except (asyncio.TimeoutError, RuntimeError) as exc:
last_error = exc
if attempt == max_attempts:
break
delay = base_delay * (2 ** (attempt - 1)) + random.uniform(0, 0.5)
await asyncio.sleep(delay)
raise last_error
# One circuit breaker per dependency, keyed by subagent name.
breakers = {
"deep-research": CircuitBreaker(failure_threshold=3, cooldown_seconds=30.0),
"quick-search": CircuitBreaker(failure_threshold=3, cooldown_seconds=15.0),
}
DEEP_RESEARCH = SubagentConfig(
name="deep-research",
description="Runs a thorough, multi-step research pass and cites sources.",
allowed_tools=["web_search", "web_fetch"],
)
QUICK_SEARCH = SubagentConfig(
name="quick-search",
description="Runs a single fast web search and summarizes the top results.",
allowed_tools=["web_search"],
)
async def research_topic(topic: str) -> dict:
"""Try the thorough subagent first; fall back to the cheap one on repeated failure."""
for name, subagent, timeout in [
("deep-research", DEEP_RESEARCH, 60.0),
("quick-search", QUICK_SEARCH, 20.0),
]:
breaker = breakers[name]
try:
breaker.before_call()
except CircuitOpenError:
print(f"[{name}] circuit open, skipping straight to fallback")
continue
options = AgentOptions(allowed_tools=[], subagents=[subagent])
try:
result = await call_with_retry(
lambda: run_subagent_with_timeout(
f"Use the {name} subagent to research: {topic}",
options,
per_message_timeout=timeout,
),
max_attempts=2,
)
breaker.record_success()
result["served_by"] = name
return result
except (asyncio.TimeoutError, RuntimeError, CircuitOpenError) as exc:
breaker.record_failure()
print(f"[{name}] failed after retries: {exc}, trying next path")
continue
raise RuntimeError(f"all research paths failed for topic: {topic}")
asyncio.run(research_topic("recent developments in battery recycling"))What this demonstrates:
- A per-message timeout wrapped around
query()'s async generator to catch a subagent that stalls without ever raising an error. - A generic
call_with_retryhelper applying jittered exponential backoff to any awaitable, kept separate from the subagent-specific stall detection. - A
CircuitBreakerkeyed per subagent name, sodeep-researchfailing doesn't affect the independentquick-searchbreaker. - A fallback ladder that tries the thorough, expensive subagent first and drops to a cheaper one only when the first path's retries and circuit both say no.
- The fallback result tagged with
served_by, so callers can tell a degraded response from the primary one.
Deep Dive
How It Works
- A subagent invocation via
query()can fail three distinct ways: it raises (a tool error, a malformed result), it stalls (no message arrives and no exception is raised), or it completes with a result that's technically valid but empty or wrong. - Retries and stall detection are different mechanisms. Retries respond to a raised exception; stall detection needs an explicit timeout, since a hung subagent produces silence, not an exception, so wrapping the message stream in
asyncio.wait_foris what turns a stall into something retry logic can catch. - A circuit breaker sits above retries: it doesn't decide whether one call succeeds, it decides whether it's worth attempting the call at all, based on that dependency's recent history.
before_call()short-circuits immediately once the failure threshold is crossed, so a known-bad subagent stops absorbing retry budget. - The half-open state is what keeps a circuit from staying open forever: once the cooldown elapses, the next
before_call()passes through, and that single probe's outcome (record_successorrecord_failure) decides whether the circuit closes again or resets its cooldown. - A fallback path is a separate decision from both retries and the circuit breaker: it's what the orchestrator does once it has given up on the primary path for this call, whether that's because retries were exhausted or the circuit was already open.
- Each layer should own a distinct concern: retries handle transient blips, stall detection handles silence, the circuit breaker handles a dependency that's down for longer than one call, and the fallback handles what the orchestrator does once all of that has failed.
Failure Modes at a Glance
| Failure Mode | Symptom | Right Response |
|---|---|---|
| Transient tool error | Exception raised, likely to succeed on a second try | Retry with backoff |
| Stalled subagent | No message, no exception, call just hangs | Timeout via asyncio.wait_for, then treat as a failure |
| Bad/incomplete result | Call returns normally but the result is empty or wrong | Validate the result explicitly; treat validation failure like a call failure |
| Dependency down for a while | Several consecutive failures against the same subagent/tool | Circuit breaker, stop calling it until the cooldown passes |
| Primary path exhausted | Retries and circuit both say no for this call | Fallback to a simpler or different subagent |
Python Notes
# asyncio.wait_for only helps if you apply it to something that can actually
# be interrupted between awaits. Wrapping the whole `async for` loop only
# times out the loop as a whole, not each individual stalled message, so
# call __anext__() directly per message when you need per-message granularity.
stream = query(prompt=prompt, options=options).__aiter__()
message = await asyncio.wait_for(stream.__anext__(), timeout=20.0)
# StopAsyncIteration means the stream ended normally, not that it failed;
# don't let it fall into the same except clause as TimeoutError/RuntimeError.Parameters & Return Values
| Parameter | Type | Description |
|---|---|---|
failure_threshold | int | Consecutive failures before CircuitBreaker opens |
cooldown_seconds | float | How long the circuit stays open before allowing a half-open probe |
max_attempts | int | Total attempts (including the first) for call_with_retry |
base_delay | float | Base seconds for exponential backoff, before jitter |
per_message_timeout | float | Seconds to wait for the next message before treating the subagent as stalled |
Gotchas
- Retrying a non-idempotent tool call. If a tool call has a side effect (sending an email, writing a record), retrying it after a timeout can execute it twice, since the first attempt might have actually succeeded before the response was lost. Fix: only auto-retry calls that are idempotent, or attach an idempotency key the tool can use to dedupe.
- Setting the stall timeout too tight. A
per_message_timeoutshorter than the subagent's normal latency kills legitimately slow-but-working calls, which then look identical to a real stall. Fix: set the timeout from the task's real latency profile, not a guess, and give adeep-research-style subagent more headroom than aquick-search-style one. - Retrying at both the parent and the subagent. If the parent orchestrator retries a failed subagent call, and the subagent's own internal loop also retries its own tool calls, failures compound into far more attempts and cost than intended. Fix: pick one layer to own retries for a given failure, usually the parent orchestrator for delegation failures, and let the subagent's internal loop handle its own tool-level retries separately.
- One circuit breaker for everything. A single shared breaker across unrelated subagents means one flaky dependency trips the circuit for calls that had nothing to do with the failure. Fix: key breakers per dependency, one per subagent name or tool name, as shown in the working example.
- No jitter on backoff delays. Fixed exponential delays without jitter mean many concurrent orchestrated tasks retry in lockstep after a shared outage, creating a new spike exactly when the dependency starts recovering. Fix: add a small random jitter to every backoff delay, as
call_with_retrydoes above. - Falling back silently. If a degraded fallback result looks identical to the primary result to the caller, nobody notices the primary path is failing until it's a bigger problem. Fix: tag fallback results (like
served_byabove) and log or alert on the fallback rate, don't just swallow the failure. - Forgetting to record success on the half-open probe. If
record_success()is never called on the probe that goes through after cooldown, the breaker can never actually close again even once the dependency has recovered. Fix: always route both outcomes of the probe call through the samerecord_success/record_failurepath as a normal call.
Alternatives
| Alternative | Use When | Don't Use When |
|---|---|---|
| Hand-rolled retry loop per call site | A single, throwaway script with one call to protect | The pattern needs to be reused across many subagent/tool call sites |
A retry/circuit-breaker library (e.g. tenacity) | You want battle-tested backoff and breaker primitives instead of maintaining your own | You need timeout semantics specific to a stalled async generator, which most such libraries don't model directly |
| Let the subagent's own internal loop retry its tool calls | The failure is inside the subagent's own tool use, not the parent-child delegation boundary | The subagent itself has stalled or failed to return; only the parent can observe and recover from that |
| Fail fast to a human-in-the-loop checkpoint | The action is high-stakes or has side effects where a blind retry or fallback is unsafe | The failure is a routine transient error where an automated retry is safe and cheap |
FAQs
What's the difference between a subagent "failing" and a subagent "stalling"?
- A failure raises an exception (a tool error, a malformed result) that ordinary exception handling can catch.
- A stall produces neither a result nor an exception, the call just never completes, which is why it needs an explicit timeout rather than a
try/exceptblock.
How many retry attempts should I use?
There's no universal number, but 2-3 total attempts is a reasonable default for orchestration-level retries. More than that usually means the failure isn't transient, and a fallback or circuit breaker is the better response than more retries.
Should every tool call be retried?
No. Only retry calls that are safe to repeat, meaning idempotent calls or ones with a dedupe key. Retrying a call with an un-deduped side effect risks executing that side effect more than once.
How is a fallback different from a retry?
A retry repeats the same call to the same dependency, hoping the failure was transient. A fallback switches to a different path entirely, a simpler subagent, a cheaper tool, once you've decided the primary path isn't going to succeed this time.
Should a circuit breaker be shared across concurrent orchestrated tasks?
Generally yes, per dependency. If ten concurrent tasks are each calling the same subagent, they should share one breaker for that subagent so the circuit reflects the dependency's real health, not each task's private view of it.
What happens during the half-open state?
Once the cooldown elapses, the breaker lets exactly one call through as a probe. If that probe succeeds, the breaker closes and resumes normal calls. If it fails, the breaker reopens and the cooldown starts again.
How do I detect a stall if query() doesn't raise an error for one?
Wrap consumption of the message stream in asyncio.wait_for, applied per message rather than around the whole loop, so a gap longer than your timeout between messages raises TimeoutError you can catch and treat as a failure.
Where should retries live, in the parent orchestrator or inside the subagent?
At whichever boundary actually observed the failure. Delegation-level failures (a subagent call that timed out or errored) belong in the parent orchestrator's retry logic. A subagent's own tool-level failures belong in that subagent's internal loop, not duplicated by the parent.
Does exponential backoff need jitter?
Yes, for anything running with concurrency. Without jitter, many callers retrying the same dependency after a shared failure tend to retry in lockstep, producing a new load spike at each backoff interval instead of spreading retries out.
What should a fallback path actually do?
Something meaningfully cheaper, simpler, or differently-scoped than the primary path, not just a copy of the same call. A fallback that fails the same way the primary did adds latency without adding resilience.
How do I avoid double-counting a failure across retries and the circuit breaker?
Record one breaker failure per call attempt sequence that ultimately failed, not one per individual retry within that sequence. The example above calls record_failure() once, after call_with_retry has exhausted its attempts, not inside the retry loop itself.
Is it safe to validate a subagent's result and treat "bad result" as a failure?
Yes, and it's necessary. A subagent can return normally with an empty or wrong result, which won't raise on its own. Explicitly validating the result and raising if it fails validation is what lets retry, fallback, and circuit-breaker logic treat it the same as any other failure.
Related
- Agentic Orchestration Basics - the orchestration mental model this page's failure handling sits inside
- Guardrails for Multi-Agent Systems: Bounding Cost and Scope - bounding cost and scope, including runaway retries
- Evaluating Agent Quality: A Checklist for Production Readiness - checking recovery behavior before shipping
- Retry and Exponential Backoff Strategies for Transient Claude API Failures - the same backoff and circuit-breaker mechanics applied at the raw API-call level
- Understanding Why Claude Agents Fail in Production - the broader landscape of production agent failure modes
Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the Claude Agent SDK (latest release). Model names, SDK versions, and pricing move quickly - verify current specifics at platform.claude.com/docs before relying on them.