How Observability Works for LLM Applications

Q: What should I put in a structured log entry for a Claude API call?

At minimum: the prompt, the response text, input_tokens, output_tokens, the model name, latency, and a request ID. Add cache token fields (cache_read_input_tokens, cache_creation_input_tokens) once you rely on prompt caching, since those numbers directly affect cost interpretation.

Observability for an LLM application answers a simple question under pressure: what is my Claude-powered system actually doing right now, and why did it just get slow, wrong, or expensive.

That question sounds like classic infrastructure observability, and it mostly is.

The difference is what you are watching move through the system: prompts, tool calls, tokens, and dollars, instead of just HTTP requests and database queries.

This page builds the mental model that the rest of the observability section assumes: tracing agent and model-call spans, logging prompts and responses in a consistent schema, and alerting on cost or reliability anomalies before they become incidents.

Summary

Observability for LLM applications means tracing every model call and tool invocation as a span, logging the prompt/response/token counts for each call in a structured schema, and alerting on cost or reliability anomalies, the same three pillars used for any distributed system, applied to Claude calls.
Insight: An agent loop can call a model five times, invoke three tools, and fail on the fourth step, and without tracing you only see "the request failed," not where or why.
Key Concepts: span, trace, structured log, token usage, cost anomaly, cache-miss rate.
When to Use: Any Claude integration beyond a single throwaway script benefits from this, but it becomes non-negotiable once you have multi-step agent loops, tool use, or production spend you are accountable for.
Limitations/Trade-offs: Observability adds latency overhead, log volume, and engineering time; over-instrumenting a low-stakes prototype wastes effort better spent shipping.
Related Topics: OpenTelemetry tracing, structured logging schemas, cost dashboards, spend alerting, deploy correlation.

Foundations

Three ideas carry almost all the weight in this space: traces, structured logs, and metrics.

A trace is the shape of one end-to-end operation, made of spans.

In an LLM agent, a trace typically covers one user request, and each span inside it covers one model call or one tool invocation, nested to reflect the actual call order.

A structured log is different from a trace even though people often conflate them.

A structured log entry captures the content of a single Claude API call: the prompt, the response, the token counts, the model used, the latency, and a request ID, all in a consistent, machine-parseable schema.

Think of the trace as answering "what happened and in what order," and the structured log as answering "what exactly did we send and get back."

You need both, because a trace without log content tells you a tool call was slow but not why, and a log without a trace tells you what one call contained but not how it fit into the larger agent run.

A simple mental analogy: if your agent loop is a phone call between departments, the trace is the call log showing who called whom and for how long, and the structured log is the transcript of what was actually said on each call.

Metrics are the third pillar: aggregated numbers over time, like tokens per minute, dollars per day, or p95 latency, usually derived from the traces and logs rather than captured separately.

Mechanics & Interactions

Here is how these pieces interact during a real agent run.

A request comes in, and your instrumentation opens a root span for it.

Each time the agent calls Claude, it opens a child span, records latency and outcome, and closes it; each time it invokes a tool, same thing.

Trace: handle_user_request
  Span: model_call (claude-sonnet-5)      120ms
  Span: tool_call (search_docs)            85ms
  Span: model_call (claude-sonnet-5)      340ms
  Span: tool_call (write_file)             40ms
  Span: model_call (claude-sonnet-5)       95ms

That shape alone, visible in an OpenTelemetry trace viewer, tells you the agent made three model calls and two tool calls, and that the second model call is where most of the latency lived.

Independently, each of those three model calls should also produce a structured log line: the exact prompt sent, the response text, input_tokens, output_tokens, and any cache_read_input_tokens, tagged with the same request ID as the trace.

The request ID is the join key: it lets you pivot from "this trace span was slow" to "here is the exact prompt and response that made it slow" without guessing.

This is where LLM observability diverges from typical web service observability: token counts and model identity are as important as latency, because they drive cost directly.

A span that took 200ms and cost 40,000 input tokens is a very different problem than a span that took 200ms and cost 200 tokens, even though a latency-only dashboard would render them identically.

Metrics then roll these up: total spend per hour, cache-hit rate across calls, error rate per tool, latency percentiles per model.

The common pitfall in reasoning about this is treating "observability" as synonymous with "logging," and stopping there.

Logging alone gives you content without shape; you can see what one call did but not how a chain of six calls degraded over the course of an agent run, and you lose the ability to alert on trends like "cache-miss rate crept up 15% since the last deploy."

Advanced Considerations & Applications

At scale, three things separate a team that can debug an incident in minutes from one that spends an afternoon guessing.

First, consistent span and attribute naming across services, so that a trace from your ingestion pipeline and a trace from your agent runtime use the same vocabulary for "model," "tool," and "request ID."

Second, correlating LLM-specific signals (spend, cache-miss rate, refusal rate) with the same deploy markers you already use for regular service regressions, so a spike is attributable to "the deploy at 14:02" rather than treated as unexplained noise.

Third, exporting Claude usage and cost data into the same dashboards your team already watches for infrastructure, rather than maintaining a separate, easily-ignored LLM-only dashboard.

Approach	Strength	Weakness	Best Fit
Print statements / ad hoc logging	Fast to add, zero setup	No structure, unqueryable, no cost visibility	Local scripts, quick experiments
Structured logs only	Queryable, captures prompt/response/token detail	No visibility into multi-step call ordering or latency chains	Single-call integrations, simple request/response APIs
OTel tracing + structured logs	Full shape and content visibility, joinable via request ID	More instrumentation effort, added overhead	Multi-step agent loops, tool-using agents, production systems
Tracing + logs + third-party cost dashboard (e.g. Datadog)	Cost and reliability sit next to existing infra metrics, supports alerting	Requires an export pipeline and dashboard maintenance	Teams already running production observability tooling

A related failure mode worth naming: sampling every span at 100% in a high-volume agent product gets expensive and noisy fast.

Most teams settle on full sampling for errors and cost outliers, with a lower sampling rate for routine successful spans, a decision worth writing down explicitly (see the ADR template referenced below) so it stays consistent as the team grows.

Common Misconceptions

"Logging the response is enough." - Without token counts and a request ID tying the log to a trace span, you can see what came back but not why it was slow or expensive relative to the rest of the run.
"Observability is only for catching outages." - A large share of the value is proactive: catching a spend spike or a rising cache-miss rate before it becomes an incident, not just diagnosing one after the fact.
"Tracing and logging are the same thing." - Tracing shows the shape and ordering of a multi-step run; structured logging shows the content of one call; you need both to fully debug an agent.
"This only matters at large scale." - Even a single-agent side project benefits from basic prompt/response/token logging, since Claude API cost and refusal patterns are easy to misjudge without any record.
"Cost tracking is a finance concern, not an engineering one." - A runaway agent loop is a reliability bug that happens to show up first as a spend anomaly, so the same team that owns latency should own cost alerting.

FAQs

Is LLM observability really different from regular application observability?

Mostly the same three pillars (traces, logs, metrics), applied to a new kind of call.
The real differences are what you track: token counts, model identity, and cost per call, none of which exist in typical web service telemetry.
Latency still matters, but a fast, expensive call and a slow, cheap call need to be distinguishable, which plain latency dashboards do not do.

Do I need OpenTelemetry specifically, or can I use my own tracing format?

OTel is not mandatory, but it is the de facto standard for span-based tracing and integrates with most existing dashboards and third-party ingestion tools (like Datadog). Rolling your own trace format works for a single service but adds friction the moment you want to correlate traces across services or export to a shared dashboard.

What is the minimum viable observability setup for a small Claude integration?

Log the prompt, response, model, token counts, latency, and a request ID for every call, in a consistent schema.
That alone, without tracing or dashboards, catches most early cost and quality problems.
Add tracing once you have multi-step agent loops where call ordering matters.

Why does token usage matter as much as latency?

Token usage drives cost directly, and unlike latency, it does not always correlate with wall-clock time. A call with a large cached prompt can be fast but still consume significant input tokens, so tracking both independently is necessary to catch cost regressions that a latency-only view would miss.

What is a "cache-miss rate" and why would I alert on it?

It is the proportion of calls that fail to hit Anthropic's prompt caching, meaning the full prompt gets reprocessed at full cost instead of the cheaper cached rate. A rising cache-miss rate usually signals a code change that broke cache eligibility (for example, an unstable prompt prefix), and it silently increases spend until someone notices the dashboard or gets an alert.

How does a trace actually help me debug a slow agent run, concretely?

It shows you the ordered list of spans (model calls and tool calls) for one request, with duration per span.
You can see immediately whether the slowness is one dominant span or evenly spread across several.
From there you jump to the structured log for that specific span's request ID to see the actual prompt and response content.

Should every span be logged and traced at 100% sampling?

Not usually at meaningful production volume. A common pattern is full sampling for errors and cost/latency outliers, with a lower sampling rate for routine successful spans, documented explicitly so the whole team applies it consistently.

What is the difference between a "span" and a "trace"?

A trace is the entire end-to-end operation (typically one user request); a span is one unit of work inside it (one model call or one tool invocation). A trace is made of one or more nested spans.

Why tie Claude cost metrics into the same dashboard as my other infrastructure metrics?

Engineers already watch that dashboard daily; a separate LLM-only dashboard gets ignored.
Cost and latency anomalies in the model layer often correlate with the same deploys that affect the rest of the system.
It makes cost a first-class reliability signal instead of a monthly billing surprise.

What should I put in a structured log entry for a Claude API call?

At minimum: the prompt, the response text, input_tokens, output_tokens, the model name, latency, and a request ID. Add cache token fields (cache_read_input_tokens, cache_creation_input_tokens) once you rely on prompt caching, since those numbers directly affect cost interpretation.

How do deploys fit into observability for LLM apps?

Deploy timestamps become a reference line on your latency, error, and cost graphs. When a metric spikes right after a deploy marker, you can attribute the regression to a specific code change instead of investigating from scratch, which is much faster than treating every spike as a mystery.

Is it worth instrumenting a small prototype before it has real users?

Full tracing and dashboards are probably overkill, but basic structured logging (prompt, response, tokens, request ID) is cheap to add early and saves significant debugging time as soon as the prototype starts misbehaving or costing more than expected.

Observability Basics - the minimal starting point: log a prompt, response, and token count for every call.
Instrumenting Agent Loops with OpenTelemetry Tracing - how to emit the spans described in this page's mental model.
Structured Logging of Prompts, Responses, and Token Counts - the log schema that pairs with tracing.
Integrating Claude Usage and Cost Dashboards with Datadog - putting Claude metrics next to your other infrastructure dashboards.
Alerting on Spend Spikes and Rising Cache-Miss Rates - turning these signals into proactive alerts.
Correlating Latency and Errors with Code Deploys - a checklist for the deploy-correlation workflow.
ADR Template: Defining Your OTel Span Schema for Tool Calls - documenting span naming and sampling decisions.
Observability Best Practices - a consolidated checklist across this section.

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official anthropic Python SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.

That question sounds like classic infrastructure observability, and it mostly is.

The difference is what you are watching move through the system: prompts, tool calls, tokens, and dollars, instead of just HTTP requests and database queries.

Summary

Observability for LLM applications means tracing every model call and tool invocation as a span, logging the prompt/response/token counts for each call in a structured schema, and alerting on cost or reliability anomalies, the same three pillars used for any distributed system, applied to Claude calls.
Insight: An agent loop can call a model five times, invoke three tools, and fail on the fourth step, and without tracing you only see "the request failed," not where or why.
Key Concepts: span, trace, structured log, token usage, cost anomaly, cache-miss rate.
When to Use: Any Claude integration beyond a single throwaway script benefits from this, but it becomes non-negotiable once you have multi-step agent loops, tool use, or production spend you are accountable for.
Limitations/Trade-offs: Observability adds latency overhead, log volume, and engineering time; over-instrumenting a low-stakes prototype wastes effort better spent shipping.
Related Topics: OpenTelemetry tracing, structured logging schemas, cost dashboards, spend alerting, deploy correlation.

Foundations

Three ideas carry almost all the weight in this space: traces, structured logs, and metrics.

A trace is the shape of one end-to-end operation, made of spans.

In an LLM agent, a trace typically covers one user request, and each span inside it covers one model call or one tool invocation, nested to reflect the actual call order.

A structured log is different from a trace even though people often conflate them.

Think of the trace as answering "what happened and in what order," and the structured log as answering "what exactly did we send and get back."

Metrics are the third pillar: aggregated numbers over time, like tokens per minute, dollars per day, or p95 latency, usually derived from the traces and logs rather than captured separately.

Mechanics & Interactions

Here is how these pieces interact during a real agent run.

A request comes in, and your instrumentation opens a root span for it.

Each time the agent calls Claude, it opens a child span, records latency and outcome, and closes it; each time it invokes a tool, same thing.

Trace: handle_user_request
  Span: model_call (claude-sonnet-5)      120ms
  Span: tool_call (search_docs)            85ms
  Span: model_call (claude-sonnet-5)      340ms
  Span: tool_call (write_file)             40ms
  Span: model_call (claude-sonnet-5)       95ms

That shape alone, visible in an OpenTelemetry trace viewer, tells you the agent made three model calls and two tool calls, and that the second model call is where most of the latency lived.

The request ID is the join key: it lets you pivot from "this trace span was slow" to "here is the exact prompt and response that made it slow" without guessing.

This is where LLM observability diverges from typical web service observability: token counts and model identity are as important as latency, because they drive cost directly.

A span that took 200ms and cost 40,000 input tokens is a very different problem than a span that took 200ms and cost 200 tokens, even though a latency-only dashboard would render them identically.

Metrics then roll these up: total spend per hour, cache-hit rate across calls, error rate per tool, latency percentiles per model.

The common pitfall in reasoning about this is treating "observability" as synonymous with "logging," and stopping there.

Advanced Considerations & Applications

At scale, three things separate a team that can debug an incident in minutes from one that spends an afternoon guessing.

Third, exporting Claude usage and cost data into the same dashboards your team already watches for infrastructure, rather than maintaining a separate, easily-ignored LLM-only dashboard.

Approach	Strength	Weakness	Best Fit
Print statements / ad hoc logging	Fast to add, zero setup	No structure, unqueryable, no cost visibility	Local scripts, quick experiments
Structured logs only	Queryable, captures prompt/response/token detail	No visibility into multi-step call ordering or latency chains	Single-call integrations, simple request/response APIs
OTel tracing + structured logs	Full shape and content visibility, joinable via request ID	More instrumentation effort, added overhead	Multi-step agent loops, tool-using agents, production systems
Tracing + logs + third-party cost dashboard (e.g. Datadog)	Cost and reliability sit next to existing infra metrics, supports alerting	Requires an export pipeline and dashboard maintenance	Teams already running production observability tooling

A related failure mode worth naming: sampling every span at 100% in a high-volume agent product gets expensive and noisy fast.

Common Misconceptions

"Logging the response is enough." - Without token counts and a request ID tying the log to a trace span, you can see what came back but not why it was slow or expensive relative to the rest of the run.
"Observability is only for catching outages." - A large share of the value is proactive: catching a spend spike or a rising cache-miss rate before it becomes an incident, not just diagnosing one after the fact.
"Tracing and logging are the same thing." - Tracing shows the shape and ordering of a multi-step run; structured logging shows the content of one call; you need both to fully debug an agent.
"This only matters at large scale." - Even a single-agent side project benefits from basic prompt/response/token logging, since Claude API cost and refusal patterns are easy to misjudge without any record.
"Cost tracking is a finance concern, not an engineering one." - A runaway agent loop is a reliability bug that happens to show up first as a spend anomaly, so the same team that owns latency should own cost alerting.

FAQs

Is LLM observability really different from regular application observability?

Mostly the same three pillars (traces, logs, metrics), applied to a new kind of call.
The real differences are what you track: token counts, model identity, and cost per call, none of which exist in typical web service telemetry.
Latency still matters, but a fast, expensive call and a slow, cheap call need to be distinguishable, which plain latency dashboards do not do.

Do I need OpenTelemetry specifically, or can I use my own tracing format?

What is the minimum viable observability setup for a small Claude integration?

Log the prompt, response, model, token counts, latency, and a request ID for every call, in a consistent schema.
That alone, without tracing or dashboards, catches most early cost and quality problems.
Add tracing once you have multi-step agent loops where call ordering matters.

Why does token usage matter as much as latency?

What is a "cache-miss rate" and why would I alert on it?

How does a trace actually help me debug a slow agent run, concretely?

It shows you the ordered list of spans (model calls and tool calls) for one request, with duration per span.
You can see immediately whether the slowness is one dominant span or evenly spread across several.
From there you jump to the structured log for that specific span's request ID to see the actual prompt and response content.

Should every span be logged and traced at 100% sampling?

What is the difference between a "span" and a "trace"?

A trace is the entire end-to-end operation (typically one user request); a span is one unit of work inside it (one model call or one tool invocation). A trace is made of one or more nested spans.

Why tie Claude cost metrics into the same dashboard as my other infrastructure metrics?

Engineers already watch that dashboard daily; a separate LLM-only dashboard gets ignored.
Cost and latency anomalies in the model layer often correlate with the same deploys that affect the rest of the system.
It makes cost a first-class reliability signal instead of a monthly billing surprise.

What should I put in a structured log entry for a Claude API call?

How do deploys fit into observability for LLM apps?

Is it worth instrumenting a small prototype before it has real users?

Observability Basics - the minimal starting point: log a prompt, response, and token count for every call.
Instrumenting Agent Loops with OpenTelemetry Tracing - how to emit the spans described in this page's mental model.
Structured Logging of Prompts, Responses, and Token Counts - the log schema that pairs with tracing.
Integrating Claude Usage and Cost Dashboards with Datadog - putting Claude metrics next to your other infrastructure dashboards.
Alerting on Spend Spikes and Rising Cache-Miss Rates - turning these signals into proactive alerts.
Correlating Latency and Errors with Code Deploys - a checklist for the deploy-correlation workflow.
ADR Template: Defining Your OTel Span Schema for Tool Calls - documenting span naming and sampling decisions.
Observability Best Practices - a consolidated checklist across this section.

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official anthropic Python SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.

How Observability Works for LLM Applications

Search Documentation

How Observability Works for LLM Applications