Observability Best Practices

Instrumenting a Claude application well means a cost spike, a latency regression, or a cache regression surfaces on a dashboard or a page within the hour, not on next month's invoice.

This checklist consolidates the practices covered across this section into rules you can check your own instrumentation against.

How to Use This Checklist

Treat section A as the non-negotiable floor for any Claude integration beyond a throwaway script.
Sections B and C become worth the effort once you have multi-step agent loops or production traffic to watch.
Revisit this list after your first real incident and note which rule would have caught it sooner.
Use the checkboxes as a literal audit of your current instrumentation, not just a reading exercise.

A - Logging Fundamentals

Log every Claude call in a consistent structured schema. Capture prompt, response, model, token counts, latency, and a request ID for every call, not just the ones that fail.
Use logging.Logger, not print. Structured output through a real logging framework integrates with log level filtering and most log aggregation agents out of the box.
Log errors in the same schema as successes. A shared shape with a status field lets one query cover both outcomes instead of maintaining separate error logs.
Generate the request ID before the call, not after. This guarantees you have an identifier even when the call raises an exception.
Centralize log construction in one function or adapter. Building the log dict inline at every call site is how schemas silently drift apart across a codebase.
Capture cache-related token fields defensively. Access cache_read_input_tokens and similar fields with getattr(..., None) since they're only populated once prompt caching applies.

B - Tracing and Cost Visibility

Emit one root span per agent run, with a child span per model call and tool invocation. This is what preserves the shape of a multi-step run, which flat logs alone cannot show.
Follow the gen_ai.* attribute naming convention on spans. Consistent naming keeps traces queryable the same way regardless of which service or model provider produced them.
Set span status explicitly on error, not just record_exception. A span without an explicit error status can still render as successful in some trace viewers.
Keep full prompt/response text out of span attributes. Spans are for short, structured metadata; put full text in structured logs instead, joined by request ID.
Compute and ship per-call cost, not just token counts. Token counts alone don't answer "what did this cost"; multiply by current per-model pricing and treat that pricing table as configuration.
Put Claude cost and latency metrics in the same dashboard as your other infrastructure metrics. A separate LLM-only dashboard gets checked far less often than the one your team already watches daily.
Tag every metric with at least model and feature/endpoint. Untagged cost metrics can't tell you which model or which part of your product is driving spend.

C - Alerting and Incident Response

Alert on spend relative to a rolling baseline, not a fixed dollar amount alone. A fixed threshold that fits today's traffic becomes stale as usage grows.
Require a minimum absolute floor alongside any ratio-based alert. A ratio-only check can page on statistically large but practically meaningless swings when the baseline is small.
Alert on cache-miss rate independently of spend. It's a leading indicator that often moves before the cost graph visibly bends.
Run alert checks on a schedule, not inline in the request path. Alert evaluation logic shouldn't add latency or a new failure mode to code already on the critical path for users.
Use your paging tool's deduplication, not custom dedupe logic. Re-implementing escalation and dedupe in the check itself usually produces worse behavior than the paging tool's built-in handling.
Check deploy correlation before assuming a provider-side issue. Most latency or error spikes trace back to a recent deploy, prompt edit, or config change faster than they trace back to an Anthropic-side incident.
Write down what you checked after every incident, including dead ends. A short incident note saves the next investigation from re-walking the same path.

D - Team Consistency

Document your span naming and attribute schema in an ADR before a second service adopts tracing. Without a documented decision, each service invents its own naming, and cross-service trace queries become impossible.
Document your sampling policy explicitly, including the error-span exception. "Sample appropriately" is not precise enough for every service to implement the same way.
Supersede ADRs instead of editing them in place when the schema changes. Editing in place destroys the historical record of why the original decision was made.
Give existing services a concrete migration deadline when the schema changes. An ADR with no rollout plan leaves already-instrumented services permanently out of sync with new ones.

Applying the Checklist in Order

Section A first, always. Structured logging costs almost nothing to add and is the foundation every other practice on this list builds on.
Section B once you have a multi-step agent loop or production traffic worth watching. Tracing and dashboards add real engineering effort, so they earn their place once logging alone stops answering your questions.
Section C once B is in place. Alerting on metrics that don't exist yet isn't possible; get the signals flowing before you set thresholds on them.
Section D as soon as a second team or service starts instrumenting. Waiting until three services have already diverged turns this from a proactive decision into a cleanup project.

FAQs

Which practice on this list matters most if I can only do one thing?

Structured logging (section A): a consistent schema capturing prompt, response, tokens, model, latency, and request ID for every call. It costs little to add, and every other practice on this list (tracing, dashboards, alerting) depends on this data existing in a queryable form.

Do I need OpenTelemetry tracing for a simple, single-call integration?

Not necessarily. Tracing earns its cost once you have a multi-step agent loop where call ordering and nesting matter. A single-call integration can rely on structured logging alone until that changes.

Why is cost treated as an observability concern instead of a finance concern on this list?

A runaway agent loop is a reliability bug that happens to surface first as a spend anomaly. Treating cost as a first-class engineering signal, alongside latency and error rate, catches that bug the same day instead of at the end of a billing cycle.

What's the risk of skipping the ADR practice in section D?

Each service that adopts tracing independently tends to invent its own span names and attribute keys. Once that happens, cross-service trace queries and shared dashboards stop working, and fixing it later requires a migration instead of a one-time decision.

Should alerting checks run as part of my normal request-handling code?

No. Run alert evaluation on a schedule, separate from the request path, whether as a cron job or a monitor evaluated by your metrics backend. Putting alert logic inline adds latency and a new failure mode to code already serving users.

Why does the checklist recommend a rolling baseline instead of a fixed spend threshold?

A fixed threshold that fits your traffic today becomes either too sensitive or too late to catch a real spike as usage grows. A rolling baseline (a median of recent windows) adapts automatically, keeping the alert meaningful without manual retuning.

What should I do differently once a second service starts instrumenting Claude calls?

Write the ADR documenting span names, attributes, and sampling policy before that second service ships its instrumentation, not after. This is the point where inconsistency starts to compound, and it's much cheaper to prevent than to migrate away from later.

Is 100% span sampling ever the right choice?

At low traffic volume, yes, since the cost and noise are negligible. As traffic grows, most teams shift to a lower sampling rate for routine successful spans while keeping error and cost-outlier spans at full sampling, which the ADR should document explicitly.

How do I know if my logging schema is missing something important?

If you can't answer "what did this specific call cost, how long did it take, and what did it return" from your logs alone, without re-running the request, the schema is incomplete. The fields in section A (prompt, response, model, tokens, latency, request ID) are the minimum that makes that question answerable.

What's the fastest way to check whether a metric spike is caused by my own deploy?

Compare the anomaly's start timestamp against your deploy log for the same window, and check Anthropic's status page to rule out a provider-side cause. This is the first step of the deploy-correlation checklist linked below, and it resolves most incidents faster than starting from scratch.

Should cost metrics live on a separate LLM-specific dashboard or with my other infrastructure metrics?

With your other infrastructure metrics. A dedicated LLM-only dashboard tends to get checked rarely, while spend sitting next to latency and error-rate graphs your team already watches daily gets noticed the day something goes wrong.

How Observability Works for LLM Applications - the mental model this checklist is built on.
Observability Basics - the minimal starting point for section A.
Instrumenting Agent Loops with OpenTelemetry Tracing - the tracing practices in section B.
Integrating Claude Usage and Cost Dashboards with Datadog - the dashboard practices in section B.
Alerting on Spend Spikes and Rising Cache-Miss Rates - the alerting practices in section C.
Correlating Latency and Errors with Code Deploys - the deploy-correlation checklist referenced in section C.
ADR Template: Defining Your OTel Span Schema for Tool Calls - the ADR practices in section D.

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official anthropic Python SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.

Observability Best Practices

Instrumenting a Claude application well means a cost spike, a latency regression, or a cache regression surfaces on a dashboard or a page within the hour, not on next month's invoice.

This checklist consolidates the practices covered across this section into rules you can check your own instrumentation against.

How to Use This Checklist

Treat section A as the non-negotiable floor for any Claude integration beyond a throwaway script.
Sections B and C become worth the effort once you have multi-step agent loops or production traffic to watch.
Revisit this list after your first real incident and note which rule would have caught it sooner.
Use the checkboxes as a literal audit of your current instrumentation, not just a reading exercise.

A - Logging Fundamentals

Log every Claude call in a consistent structured schema. Capture prompt, response, model, token counts, latency, and a request ID for every call, not just the ones that fail.
Use logging.Logger, not print. Structured output through a real logging framework integrates with log level filtering and most log aggregation agents out of the box.
Log errors in the same schema as successes. A shared shape with a status field lets one query cover both outcomes instead of maintaining separate error logs.
Generate the request ID before the call, not after. This guarantees you have an identifier even when the call raises an exception.
Centralize log construction in one function or adapter. Building the log dict inline at every call site is how schemas silently drift apart across a codebase.
Capture cache-related token fields defensively. Access cache_read_input_tokens and similar fields with getattr(..., None) since they're only populated once prompt caching applies.

B - Tracing and Cost Visibility

Emit one root span per agent run, with a child span per model call and tool invocation. This is what preserves the shape of a multi-step run, which flat logs alone cannot show.
Follow the gen_ai.* attribute naming convention on spans. Consistent naming keeps traces queryable the same way regardless of which service or model provider produced them.
Set span status explicitly on error, not just record_exception. A span without an explicit error status can still render as successful in some trace viewers.
Keep full prompt/response text out of span attributes. Spans are for short, structured metadata; put full text in structured logs instead, joined by request ID.
Compute and ship per-call cost, not just token counts. Token counts alone don't answer "what did this cost"; multiply by current per-model pricing and treat that pricing table as configuration.
Put Claude cost and latency metrics in the same dashboard as your other infrastructure metrics. A separate LLM-only dashboard gets checked far less often than the one your team already watches daily.
Tag every metric with at least model and feature/endpoint. Untagged cost metrics can't tell you which model or which part of your product is driving spend.

C - Alerting and Incident Response

Alert on spend relative to a rolling baseline, not a fixed dollar amount alone. A fixed threshold that fits today's traffic becomes stale as usage grows.
Require a minimum absolute floor alongside any ratio-based alert. A ratio-only check can page on statistically large but practically meaningless swings when the baseline is small.
Alert on cache-miss rate independently of spend. It's a leading indicator that often moves before the cost graph visibly bends.
Run alert checks on a schedule, not inline in the request path. Alert evaluation logic shouldn't add latency or a new failure mode to code already on the critical path for users.
Use your paging tool's deduplication, not custom dedupe logic. Re-implementing escalation and dedupe in the check itself usually produces worse behavior than the paging tool's built-in handling.
Check deploy correlation before assuming a provider-side issue. Most latency or error spikes trace back to a recent deploy, prompt edit, or config change faster than they trace back to an Anthropic-side incident.
Write down what you checked after every incident, including dead ends. A short incident note saves the next investigation from re-walking the same path.

D - Team Consistency

Document your span naming and attribute schema in an ADR before a second service adopts tracing. Without a documented decision, each service invents its own naming, and cross-service trace queries become impossible.
Document your sampling policy explicitly, including the error-span exception. "Sample appropriately" is not precise enough for every service to implement the same way.
Supersede ADRs instead of editing them in place when the schema changes. Editing in place destroys the historical record of why the original decision was made.
Give existing services a concrete migration deadline when the schema changes. An ADR with no rollout plan leaves already-instrumented services permanently out of sync with new ones.

Applying the Checklist in Order

Section A first, always. Structured logging costs almost nothing to add and is the foundation every other practice on this list builds on.
Section B once you have a multi-step agent loop or production traffic worth watching. Tracing and dashboards add real engineering effort, so they earn their place once logging alone stops answering your questions.
Section C once B is in place. Alerting on metrics that don't exist yet isn't possible; get the signals flowing before you set thresholds on them.
Section D as soon as a second team or service starts instrumenting. Waiting until three services have already diverged turns this from a proactive decision into a cleanup project.

FAQs

Which practice on this list matters most if I can only do one thing?

Do I need OpenTelemetry tracing for a simple, single-call integration?

Why is cost treated as an observability concern instead of a finance concern on this list?

What's the risk of skipping the ADR practice in section D?

Should alerting checks run as part of my normal request-handling code?

Why does the checklist recommend a rolling baseline instead of a fixed spend threshold?

What should I do differently once a second service starts instrumenting Claude calls?

Is 100% span sampling ever the right choice?

How do I know if my logging schema is missing something important?

What's the fastest way to check whether a metric spike is caused by my own deploy?

Should cost metrics live on a separate LLM-specific dashboard or with my other infrastructure metrics?

How Observability Works for LLM Applications - the mental model this checklist is built on.
Observability Basics - the minimal starting point for section A.
Instrumenting Agent Loops with OpenTelemetry Tracing - the tracing practices in section B.
Integrating Claude Usage and Cost Dashboards with Datadog - the dashboard practices in section B.
Alerting on Spend Spikes and Rising Cache-Miss Rates - the alerting practices in section C.
Correlating Latency and Errors with Code Deploys - the deploy-correlation checklist referenced in section C.
ADR Template: Defining Your OTel Span Schema for Tool Calls - the ADR practices in section D.

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official anthropic Python SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.

Observability Best Practices

Search Documentation

Observability Best Practices