Observability Best Practices

Instrumenting a Claude application well means a cost spike, a latency regression, or a cache regression surfaces on a dashboard or a page within the hour, not on next month's invoice.

This checklist consolidates the practices covered across this section into rules you can check your own instrumentation against.

How to Use This Checklist

• Treat section A as the non-negotiable floor for any Claude integration beyond a throwaway script.
• Sections B and C become worth the effort once you have multi-step agent loops or production traffic to watch.
• Revisit this list after your first real incident and note which rule would have caught it sooner.
• Use the checkboxes as a literal audit of your current instrumentation, not just a reading exercise.

A - Logging Fundamentals

• Log every Claude call in a consistent structured schema. Capture prompt, response, model, token counts, latency, and a request ID for every call, not just the ones that fail.
• Use logging.Logger, not print. Structured output through a real logging framework integrates with log level filtering and most log aggregation agents out of the box.
• Log errors in the same schema as successes. A shared shape with a status field lets one query cover both outcomes instead of maintaining separate error logs.
• Generate the request ID before the call, not after. This guarantees you have an identifier even when the call raises an exception.
• Centralize log construction in one function or adapter. Building the log dict inline at every call site is how schemas silently drift apart across a codebase.
• Capture cache-related token fields defensively. Access cache_read_input_tokens and similar fields with getattr(..., None) since they're only populated once prompt caching applies.

B - Tracing and Cost Visibility

• Emit one root span per agent run, with a child span per model call and tool invocation. This is what preserves the shape of a multi-step run, which flat logs alone cannot show.
• Follow the gen_ai.* attribute naming convention on spans. Consistent naming keeps traces queryable the same way regardless of which service or model provider produced them.
• Set span status explicitly on error, not just record_exception. A span without an explicit error status can still render as successful in some trace viewers.
• Keep full prompt/response text out of span attributes. Spans are for short, structured metadata; put full text in structured logs instead, joined by request ID.
• Compute and ship per-call cost, not just token counts. Token counts alone don't answer "what did this cost"; multiply by current per-model pricing and treat that pricing table as configuration.
• Put Claude cost and latency metrics in the same dashboard as your other infrastructure metrics. A separate LLM-only dashboard gets checked far less often than the one your team already watches daily.
• Tag every metric with at least model and feature/endpoint. Untagged cost metrics can't tell you which model or which part of your product is driving spend.

C - Alerting and Incident Response

• Alert on spend relative to a rolling baseline, not a fixed dollar amount alone. A fixed threshold that fits today's traffic becomes stale as usage grows.
• Require a minimum absolute floor alongside any ratio-based alert. A ratio-only check can page on statistically large but practically meaningless swings when the baseline is small.
• Alert on cache-miss rate independently of spend. It's a leading indicator that often moves before the cost graph visibly bends.
• Run alert checks on a schedule, not inline in the request path. Alert evaluation logic shouldn't add latency or a new failure mode to code already on the critical path for users.
• Use your paging tool's deduplication, not custom dedupe logic. Re-implementing escalation and dedupe in the check itself usually produces worse behavior than the paging tool's built-in handling.
• Check deploy correlation before assuming a provider-side issue. Most latency or error spikes trace back to a recent deploy, prompt edit, or config change faster than they trace back to an Anthropic-side incident.
• Write down what you checked after every incident, including dead ends. A short incident note saves the next investigation from re-walking the same path.

D - Team Consistency

• Document your span naming and attribute schema in an ADR before a second service adopts tracing. Without a documented decision, each service invents its own naming, and cross-service trace queries become impossible.
• Document your sampling policy explicitly, including the error-span exception. "Sample appropriately" is not precise enough for every service to implement the same way.
• Supersede ADRs instead of editing them in place when the schema changes. Editing in place destroys the historical record of why the original decision was made.
• Give existing services a concrete migration deadline when the schema changes. An ADR with no rollout plan leaves already-instrumented services permanently out of sync with new ones.

Applying the Checklist in Order

• Section A first, always. Structured logging costs almost nothing to add and is the foundation every other practice on this list builds on.
• Section B once you have a multi-step agent loop or production traffic worth watching. Tracing and dashboards add real engineering effort, so they earn their place once logging alone stops answering your questions.
• Section C once B is in place. Alerting on metrics that don't exist yet isn't possible; get the signals flowing before you set thresholds on them.
• Section D as soon as a second team or service starts instrumenting. Waiting until three services have already diverged turns this from a proactive decision into a cleanup project.

FAQs

Related

• How Observability Works for LLM Applications - the mental model this checklist is built on.
• Observability Basics - the minimal starting point for section A.
• Instrumenting Agent Loops with OpenTelemetry Tracing - the tracing practices in section B.
• Integrating Claude Usage and Cost Dashboards with Datadog - the dashboard practices in section B.
• Alerting on Spend Spikes and Rising Cache-Miss Rates - the alerting practices in section C.
• Correlating Latency and Errors with Code Deploys - the deploy-correlation checklist referenced in section C.
• ADR Template: Defining Your OTel Span Schema for Tool Calls - the ADR practices in section D.

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official anthropic Python SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.