Instrumenting Agent Loops with OpenTelemetry Tracing

An agent loop that calls Claude multiple times and invokes tools in between is hard to debug from logs alone, because logs don't preserve the shape of the run.

OpenTelemetry (OTel) tracing fixes that by wrapping each model call and tool invocation in a span, nested under one root span per agent run, so you can see exactly where time went and where things broke.

Summary

OpenTelemetry is a vendor-neutral standard for emitting traces, made of spans that represent units of work, with a parent-child relationship that mirrors your call structure.

For a Claude agent loop, the natural mapping is one root span per user request, with a child span for each model call and each tool invocation.

Each span carries attributes: model name, token counts, stop reason, tool name, and duration, which is enough to answer "why was this run slow" without opening a log viewer.

Most OTel setups export spans to a backend (Jaeger, Datadog, Honeycomb, or a local console exporter for development) via the OTel SDK's exporter and processor pipeline.

This page builds a working tracer for a Claude agent loop from scratch, using the console exporter for local development.

Recipe

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
 
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(ConsoleSpanExporter())
)
tracer = trace.get_tracer("claude.agent")
 
with tracer.start_as_current_span("agent_run") as run_span:
    with tracer.start_as_current_span("model_call") as call_span:
        call_span.set_attribute("gen_ai.request.model", "claude-sonnet-5")
        # ... make the Claude API call here ...

When to reach for this:

• Your app runs a multi-step agent loop (model call, tool call, model call, ...) and you need to see where latency or failures concentrate.
• You already export traces to a backend (Jaeger, Datadog, Honeycomb) for other services and want Claude calls in the same view.
• You are debugging a specific slow or failing run and need the call-order shape, not just aggregate metrics.
• You are about to write an ADR for your team's span schema and need a working reference implementation first.

Working Example

import time
 
import anthropic
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.trace import Status, StatusCode
 
# --- One-time tracer setup ---
resource = Resource.create({"service.name": "claude-agent-service"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
 
tracer = trace.get_tracer("claude.agent")
client = anthropic.Anthropic()
 
 
def traced_model_call(messages: list[dict], model: str = "claude-sonnet-5"):
    """Make one Claude call wrapped in a child span."""
    with tracer.start_as_current_span("model_call") as span:
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        try:
            response = client.messages.create(
                model=model,
                max_tokens=500,
                messages=messages,
            )
        except anthropic.APIStatusError as exc:
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            span.set_attribute("error.type", exc.__class__.__name__)
            raise
 
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
        span.set_attribute("gen_ai.response.stop_reason", response.stop_reason)
        return response
 
 
def traced_tool_call(tool_name: str, tool_input: dict) -> str:
    """Run a tool wrapped in a child span."""
    with tracer.start_as_current_span("tool_call") as span:
        span.set_attribute("gen_ai.tool.name", tool_name)
        start = time.monotonic()
 
        if tool_name == "get_weather":
            result = f"Weather for {tool_input.get('city', 'unknown')}: sunny, 72F"
        else:
            span.set_status(Status(StatusCode.ERROR, "unknown tool"))
            raise ValueError(f"Unknown tool: {tool_name}")
 
        span.set_attribute("gen_ai.tool.duration_ms", round((time.monotonic() - start) * 1000, 1))
        return result
 
 
def run_agent_loop(user_prompt: str, max_steps: int = 4) -> str:
    """One root span per run, containing one child span per step."""
    with tracer.start_as_current_span("agent_run") as run_span:
        run_span.set_attribute("gen_ai.agent.max_steps", max_steps)
        messages = [{"role": "user", "content": user_prompt}]
 
        tools = [{
            "name": "get_weather",
            "description": "Get current weather for a city",
            "input_schema": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        }]
 
        for step in range(max_steps):
            response = traced_model_call(messages)
 
            if response.stop_reason != "tool_use":
                run_span.set_attribute("gen_ai.agent.steps_taken", step + 1)
                return response.content[0].text
 
            messages.append({"role": "assistant", "content": response.content})
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    result_text = traced_tool_call(block.name, block.input)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": result_text,
                    })
            messages.append({"role": "user", "content": tool_results})
 
        run_span.set_attribute("gen_ai.agent.steps_taken", max_steps)
        return "Max steps reached without a final answer."
 
 
if __name__ == "__main__":
    answer = run_agent_loop("What is the weather in Denver right now?")
    print(answer)

What this demonstrates:

• One root agent_run span per user request, with model_call and tool_call spans nested inside it as children.
• Span attributes (gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.tool.name) that let you query traces by model, tool, or token cost later.
• Recording failures on the span itself via set_status(Status(StatusCode.ERROR, ...)) instead of only logging them elsewhere.
• A tools and tool_use handling flow that mirrors how a real Claude agent loop invokes and resolves tool calls.
• A ConsoleSpanExporter for local development, swapped for a real backend exporter in production.

Deep Dive

How It Works

• TracerProvider is the OTel SDK's factory for tracers; you configure it once at process startup with a Resource (service metadata) and one or more span processors.
• BatchSpanProcessor batches finished spans and hands them to an exporter asynchronously, which keeps tracing overhead off your request's critical path.
• tracer.start_as_current_span(...) opens a span and makes it the "current" span in a context variable, so any span started inside that with block is automatically nested as its child. This is what produces the parent-child tree without manually passing span objects around.
• Ending a span (leaving the with block) records its duration automatically; you never set duration manually.
• Attributes are just key-value pairs attached to a span; there's no schema enforcement at the SDK level, which is exactly why teams write an ADR to keep names consistent (see Related).

Attribute Naming Convention

These follow the emerging gen_ai.* semantic convention used across OTel-instrumented LLM tooling, which keeps your spans queryable the same way regardless of which model provider a given span came from.

Sampling Considerations

Tracing every span at 100% is fine at low volume but gets expensive and noisy once an agent product has real traffic.

A common pattern is a ParentBased sampler at the root span combined with a higher fixed rate for error spans, so you keep full visibility into failures while sampling down routine successful runs.

from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
 
# Sample 20% of root spans; every child span inherits the parent's decision.
provider = TracerProvider(
    resource=resource,
    sampler=ParentBased(root=TraceIdRatioBased(0.2)),
)

Python Notes

• start_as_current_span is also usable as a decorator (@tracer.start_as_current_span("name")) if you prefer wrapping whole functions instead of with blocks.
• Always call set_status on error, not just record_exception; a span with an unset status can still render as "successful" in some trace viewers even if you logged an exception on it.
• Use provider.shutdown() (or rely on the SDK's atexit hook) in short-lived scripts so buffered spans in the BatchSpanProcessor actually flush before the process exits.

Gotchas

• Forgetting to flush before process exit. In a short script or Lambda-style handler, buffered spans in a BatchSpanProcessor can be lost if the process exits before the next batch flush. Fix: call trace.get_tracer_provider().force_flush() before returning, or use a SimpleSpanProcessor for short-lived processes.
• Recording exceptions without setting span status. Calling span.record_exception(exc) alone does not mark the span as failed. Fix: always pair it with span.set_status(Status(StatusCode.ERROR, str(exc))).
• Logging the full prompt text as a span attribute. Span attributes are meant for short, structured metadata, and large text blobs bloat your trace storage and can leak sensitive data into a trace backend with different retention rules than your logs. Fix: keep prompt/response text in your structured logs, and put only token counts, model name, and short identifiers on spans.
• Losing the parent-child relationship across threads or async tasks. start_as_current_span relies on context propagation; spawning a new thread or asyncio.create_task without propagating context can produce orphaned spans. Fix: use opentelemetry.context.attach/detach to carry context across thread or task boundaries, or use the OTel async instrumentation helpers.
• 100% sampling in production. Tracing every span at full volume once traffic grows adds real cost and noise in the trace backend. Fix: apply a ParentBased sampler with a lower root rate, keeping 100% for error and cost-outlier spans.
• Inconsistent span and attribute names across services. If one service calls it model_call and another calls it llm_request, cross-service traces become unqueryable as a set. Fix: agree on naming in an ADR before multiple teams start instrumenting independently.

Alternatives

FAQs

with tracer.start_as_current_span("tool_call") as span:
    try:
        result = run_tool(name, args)
    except Exception as exc:
        span.record_exception(exc)
        span.set_status(Status(StatusCode.ERROR, str(exc)))
        raise

Related

• How Observability Works for LLM Applications - the mental model tracing fits into.
• Observability Basics - the flat, pre-tracing logging pattern this article builds on.
• Structured Logging of Prompts, Responses, and Token Counts - the log schema that pairs with spans via request ID.
• ADR Template: Defining Your OTel Span Schema for Tool Calls - documenting the naming and sampling decisions made here.
• Correlating Latency and Errors with Code Deploys - using trace data to attribute a regression to a deploy.
• Observability Best Practices - a consolidated checklist across this section.

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official anthropic Python SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.