Troubleshooting & Reliability Best Practices

These are the practices that separate a team that treats every Claude API failure as a one-off mystery from one that recognizes patterns, fixes root causes, and gets faster at incident response over time.

How to Use This List

• Treat category A as non-negotiable baseline for any production Claude integration; categories B-E build on it.
• Use this as a periodic audit, not just a one-time setup checklist, revisit it after any incident to see which rule would have caught it earlier.
• Tie enforceable rules (structured logging, error classification) to code review or CI where possible, rather than relying on memory.
• Pair this list with the RCA template when a rule is violated and causes an incident, the RCA's action items often map directly back to one of these practices.

A - Observability Foundations

• Log every Claude API call with a request ID, latency, and outcome. Without a captured request ID on both successes and failures, you often can't get further detail on a past incident, from your own data or from Anthropic support.
• Classify every failure into a failure family (capacity, contract, lifecycle, cost/latency) at the point of capture. Tagging failures when they happen, not after the fact during an RCA, is what makes dashboards and alerts genuinely queryable.
• Track token usage per call, not just per billing period. Per-call input_tokens/output_tokens logging is your earliest signal of context growth or a cache miss storm, long before the monthly invoice tells you.
• Run an independent health check separate from real traffic. A scheduled low-cost ping confirms whether a problem is upstream (the API) or local (your traffic pattern) when real-user error rates spike.
• Alert on approaching thresholds, not just on hard failures. A warning at 80% of a rate limit or context-window budget gives you time to act before a user-facing failure occurs.

B - Capacity and Retry Discipline

• Read the rate-limit response headers, don't guess which limit was hit. anthropic-ratelimit-requests-remaining, -input-tokens-remaining, and -output-tokens-remaining tell you exactly which ceiling triggered a 429, which changes the fix.
• Distinguish bursty traffic from sustained overload before choosing a fix. A capacity upgrade doesn't fix a recurring burst pattern, and backoff alone doesn't fix genuine sustained growth past your tier.
• Use jittered exponential backoff, never a fixed retry interval. Fixed intervals synchronize retries across clients and can recreate the exact spike that caused the original failure.
• Only retry genuinely transient exceptions. RateLimitError, connection/timeout errors, and 5xx responses are retryable; a malformed-request 4xx will fail identically every time, retrying it just adds noise.
• Cap retry attempts and add a circuit breaker for sustained failure. An unbounded retry loop can turn a transient failure into an indefinitely hanging request; a circuit breaker protects the caller once a dependency is known to be down.

C - Contract Safety (Tool Use)

• Validate every tool-use input against its schema before executing the tool. Nothing structurally guarantees a model's tool call matches your input_schema; validate even for tools that "always" get simple arguments in practice.
• Log the raw tool-use block before attempting to parse or validate it. If validation or repair later fails, the raw log entry is the only evidence you'll have to diagnose what actually happened.
• Return structured is_error tool results instead of silently swallowing validation failures. A silently caught error looks like a successful turn to the rest of the agent loop, and the real failure surfaces later, disconnected from its cause.
• Reject ambiguous mismatches rather than guessing a default. Repair only unambiguous, low-risk cases (case normalization); guessing a value for a missing required field risks executing the tool with meaningfully wrong data.

D - Lifecycle Management (Context and Streaming)

• Count tokens per turn and trim at turn boundaries, not mid-message. Truncating mid-message can orphan a tool_use block from its tool_result, which the API will reject outright.
• Reserve real headroom below the hard context limit. 15-25% of the window, reserved for the model's own response and any pending tool results, prevents a turn from failing right at the edge.
• Persist full conversation history outside the trimmed working context. Trimming for the API's sake should never mean losing data for your own logs or a later human review.
• Track partial output as it streams and decide up front whether a reconnect resumes or restarts. The API does not resume a dropped stream from where it left off; a reconnect is always a new request, so decide the behavior before the first disconnect happens in production, not during the incident.

E - Cost and Cache Stability

• Keep cached prefixes (system prompts, tool definitions) fully static, with volatile data moved to the user message. A single dynamic field inside a cached block, a timestamp, a request ID, guarantees a miss on every call.
• Build tool definitions in a fixed, explicit order. Assembling tool lists from an unordered dict or set can silently reorder the serialized prefix between deploys, breaking the cache hash without any code change to the tools themselves.
• Monitor cache hit rate as a first-class metric, not an afterthought. A cache miss produces no error, so without dedicated monitoring a miss storm is invisible until someone checks the bill or the latency dashboard.
• Diff the actual serialized prefix, not just the source template, when hit rate drops. Interpolated values can shift the final bytes sent even when the template itself looks unchanged in a diff.

F - Incident Response

• Write an RCA for any incident that paged someone, breached an SLA, or caused a noticeable cost spike. A fast resolution with no RCA often means the same root cause resurfaces later without anyone recognizing the pattern.
• Separate the triggering event from the root cause in every RCA. The trigger (a traffic spike, a deploy) is often unavoidable; the root cause (missing backoff, no concurrency cap) is usually what's actually fixable.
• Assign an owner and a target date to every action item. An action item without both quietly becomes a wish instead of a commitment.
• Store completed RCAs searchable by failure family. A recurring capacity or contract failure pattern across multiple incidents is a strong signal to invest in the underlying fix, not just patch each occurrence.

When You Are Done

Passing every item in categories A-C is the baseline for a production Claude integration that fails gracefully under normal conditions. Categories D-F matter most for agentic, multi-turn, or high-volume workloads, where lifecycle and cost failures compound in ways a simple chatbot integration rarely sees.

FAQs

Related

• Understanding Why Claude Agents Fail in Production - the failure taxonomy these practices are organized around.
• Troubleshooting & Reliability Basics - the hands-on starting point category A builds on.
• Retry and Exponential Backoff Strategies for Transient Claude API Failures - the full pattern behind category B's retry rules.
• Handling Malformed Tool-Use JSON and Schema Validation Failures - the full pattern behind category C.
• Root-Cause-Analysis Template for Agent Failures - the template category F's incident-response rules point to.

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official anthropic Python SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.