Troubleshooting & Reliability Best Practices
These are the practices that separate a team that treats every Claude API failure as a one-off mystery from one that recognizes patterns, fixes root causes, and gets faster at incident response over time.
How to Use This List
- Treat category A as non-negotiable baseline for any production Claude integration; categories B-E build on it.
- Use this as a periodic audit, not just a one-time setup checklist, revisit it after any incident to see which rule would have caught it earlier.
- Tie enforceable rules (structured logging, error classification) to code review or CI where possible, rather than relying on memory.
- Pair this list with the RCA template when a rule is violated and causes an incident, the RCA's action items often map directly back to one of these practices.
A - Observability Foundations
- Log every Claude API call with a request ID, latency, and outcome. Without a captured request ID on both successes and failures, you often can't get further detail on a past incident, from your own data or from Anthropic support.
- Classify every failure into a failure family (capacity, contract, lifecycle, cost/latency) at the point of capture. Tagging failures when they happen, not after the fact during an RCA, is what makes dashboards and alerts genuinely queryable.
- Track token usage per call, not just per billing period. Per-call
input_tokens/output_tokenslogging is your earliest signal of context growth or a cache miss storm, long before the monthly invoice tells you. - Run an independent health check separate from real traffic. A scheduled low-cost ping confirms whether a problem is upstream (the API) or local (your traffic pattern) when real-user error rates spike.
- Alert on approaching thresholds, not just on hard failures. A warning at 80% of a rate limit or context-window budget gives you time to act before a user-facing failure occurs.
B - Capacity and Retry Discipline
- Read the rate-limit response headers, don't guess which limit was hit.
anthropic-ratelimit-requests-remaining,-input-tokens-remaining, and-output-tokens-remainingtell you exactly which ceiling triggered a 429, which changes the fix. - Distinguish bursty traffic from sustained overload before choosing a fix. A capacity upgrade doesn't fix a recurring burst pattern, and backoff alone doesn't fix genuine sustained growth past your tier.
- Use jittered exponential backoff, never a fixed retry interval. Fixed intervals synchronize retries across clients and can recreate the exact spike that caused the original failure.
- Only retry genuinely transient exceptions.
RateLimitError, connection/timeout errors, and 5xx responses are retryable; a malformed-request 4xx will fail identically every time, retrying it just adds noise. - Cap retry attempts and add a circuit breaker for sustained failure. An unbounded retry loop can turn a transient failure into an indefinitely hanging request; a circuit breaker protects the caller once a dependency is known to be down.
C - Contract Safety (Tool Use)
- Validate every tool-use input against its schema before executing the tool. Nothing structurally guarantees a model's tool call matches your
input_schema; validate even for tools that "always" get simple arguments in practice. - Log the raw tool-use block before attempting to parse or validate it. If validation or repair later fails, the raw log entry is the only evidence you'll have to diagnose what actually happened.
- Return structured
is_errortool results instead of silently swallowing validation failures. A silently caught error looks like a successful turn to the rest of the agent loop, and the real failure surfaces later, disconnected from its cause. - Reject ambiguous mismatches rather than guessing a default. Repair only unambiguous, low-risk cases (case normalization); guessing a value for a missing required field risks executing the tool with meaningfully wrong data.
D - Lifecycle Management (Context and Streaming)
- Count tokens per turn and trim at turn boundaries, not mid-message. Truncating mid-message can orphan a
tool_useblock from itstool_result, which the API will reject outright. - Reserve real headroom below the hard context limit. 15-25% of the window, reserved for the model's own response and any pending tool results, prevents a turn from failing right at the edge.
- Persist full conversation history outside the trimmed working context. Trimming for the API's sake should never mean losing data for your own logs or a later human review.
- Track partial output as it streams and decide up front whether a reconnect resumes or restarts. The API does not resume a dropped stream from where it left off; a reconnect is always a new request, so decide the behavior before the first disconnect happens in production, not during the incident.
E - Cost and Cache Stability
- Keep cached prefixes (system prompts, tool definitions) fully static, with volatile data moved to the user message. A single dynamic field inside a cached block, a timestamp, a request ID, guarantees a miss on every call.
- Build tool definitions in a fixed, explicit order. Assembling tool lists from an unordered
dictorsetcan silently reorder the serialized prefix between deploys, breaking the cache hash without any code change to the tools themselves. - Monitor cache hit rate as a first-class metric, not an afterthought. A cache miss produces no error, so without dedicated monitoring a miss storm is invisible until someone checks the bill or the latency dashboard.
- Diff the actual serialized prefix, not just the source template, when hit rate drops. Interpolated values can shift the final bytes sent even when the template itself looks unchanged in a diff.
F - Incident Response
- Write an RCA for any incident that paged someone, breached an SLA, or caused a noticeable cost spike. A fast resolution with no RCA often means the same root cause resurfaces later without anyone recognizing the pattern.
- Separate the triggering event from the root cause in every RCA. The trigger (a traffic spike, a deploy) is often unavoidable; the root cause (missing backoff, no concurrency cap) is usually what's actually fixable.
- Assign an owner and a target date to every action item. An action item without both quietly becomes a wish instead of a commitment.
- Store completed RCAs searchable by failure family. A recurring capacity or contract failure pattern across multiple incidents is a strong signal to invest in the underlying fix, not just patch each occurrence.
When You Are Done
Passing every item in categories A-C is the baseline for a production Claude integration that fails gracefully under normal conditions. Categories D-F matter most for agentic, multi-turn, or high-volume workloads, where lifecycle and cost failures compound in ways a simple chatbot integration rarely sees.
FAQs
Which category should a small team tackle first?
Category A, observability foundations. Structured logging, failure classification, and token tracking cost little to set up and make every other category, retries, validation, cache monitoring, dramatically easier to reason about once you actually have incidents to diagnose.
Is retrying every failed Claude API call a good default?
No. Only retry genuinely transient exceptions (rate limits, connection/timeout errors, 5xx). Retrying a malformed-request 4xx or a schema validation failure will fail identically every time and just adds latency and noise.
Why does cache hit rate need its own monitoring separate from error rate?
Because a cache miss never produces an error, the request still succeeds normally, it's just billed and timed at full cost. Without a dedicated hit-rate metric, a miss storm is invisible until a cost or latency dashboard eventually reveals it.
What's the single most common mistake across these categories?
Treating a failure as resolved once the symptom stops, without capturing enough detail (a request ID, a header breakdown, a serialized prefix diff) to actually confirm the root cause. That gap is what turns a one-time incident into a recurring one.
Should every tool call be validated, even simple ones?
Yes. Nothing structurally guarantees a model's tool call matches your schema, it's a strong prompt-level convention, not an enforced contract. A tool that "always" gets simple arguments in testing can still receive a malformed call in production.
How often should this list be revisited?
At minimum, after every incident, checking which rule would have caught it earlier or faster. Many teams also do a lighter periodic pass (quarterly is common) independent of incidents, since new tools or workflows can introduce gaps this list didn't originally cover.
Does context trimming risk losing data permanently?
Not if you follow the persistence practice in category D: keep the full conversation history in your own storage and only trim the working window sent to the API. Trimming should only ever affect what the model sees, never what you retain.
Why does tool definition ordering matter for caching?
The cache hashes the exact serialized prefix, including tool definitions. If tool lists are built from an unordered collection, the serialized order can shift between deploys or process restarts, changing the hash and breaking the cache even though nothing about the tools themselves changed.
What makes an RCA action item actually get done, versus becoming a wish?
An explicit owner, a target date, and a concrete verification step (a test that now passes, a metric that recovered and stayed recovered). Action items missing any of these three tend to quietly stall.
Is a circuit breaker necessary for every Claude API integration, or just high-volume ones?
It matters most for high-volume or agentic workloads where a sustained outage would otherwise mean every in-flight request keeps retrying into a known-bad dependency. Low-volume integrations can often get by with capped retries and good alerting alone.
Related
- Understanding Why Claude Agents Fail in Production - the failure taxonomy these practices are organized around.
- Troubleshooting & Reliability Basics - the hands-on starting point category A builds on.
- Retry and Exponential Backoff Strategies for Transient Claude API Failures - the full pattern behind category B's retry rules.
- Handling Malformed Tool-Use JSON and Schema Validation Failures - the full pattern behind category C.
- Root-Cause-Analysis Template for Agent Failures - the template category F's incident-response rules point to.
Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official
anthropicPython SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.