Correlating Latency and Errors with Code Deploys
When Claude call latency, error rate, or refusal rate spikes, the fastest path to a root cause is checking whether it lines up with your last deploy, not starting from scratch.
This checklist walks through confirming that correlation, ruling out the wrong culprit, and closing the loop so the next incident is faster to diagnose.
How to Use This Checklist
- Work through the items in order; each step either confirms or rules out deploy correlation before you move to the next.
- Keep a scratch note of what you checked and what you found, since that note becomes your incident writeup.
- Not every item applies to every incident; skip steps that don't fit your setup (for example, if you have no deploy markers yet, item 3 becomes your first fix, not a skippable check).
- Revisit this checklist after each real incident and add anything you had to improvise, so it stays current with how your team actually deploys.
Immediate Triage (1-5)
- Note the exact time the anomaly started: Pull the first timestamp where latency, error rate, or refusal rate crossed your normal range from your dashboard or alert, not the time you noticed it.
- Pull your deploy log for the same window: List every deploy (code, config, prompt, or infra) in the hour before and after the anomaly's start time.
- Check for a deploy marker on the same dashboard: If your latency/error graphs already show deploy markers as vertical lines, visually confirm whether the spike starts right after one.
- Confirm the anomaly is Claude-specific, not infrastructure-wide: Check whether other services (database, unrelated APIs) show the same spike at the same time; a shared cause points away from your Claude call path specifically.
- Check Anthropic's status page for the same window: Rule out a provider-side incident before spending more time on your own deploy history.
Narrowing the Cause (6-13)
- Diff the deploy against the previous one: Pull the actual code diff for the most likely deploy, not just its commit message, since the message may undersell what changed.
- Check for prompt or system-prompt changes specifically: A prompt edit is a common, easy-to-miss cause of latency change (longer prompts, different formatting) or new refusal patterns.
- Check for a model version or model name change: Confirm the deploy didn't switch models (for example from Claude Haiku 4.5 to Claude Sonnet 5) as part of a routing or config change, since that alone explains both latency and behavior shifts.
- Check for a
max_tokensor temperature/parameter change: A raisedmax_tokensvalue or parameter tweak can measurably shift latency and, less obviously, response character. - Check for new or modified tool definitions: An added tool, or a changed tool schema, can alter how often the model chooses
tool_use, changing the number of round trips per request. - Check for retry or timeout logic changes: A shortened timeout or a new retry-on-failure path can look like a latency spike in aggregate metrics even if per-call latency didn't change.
- Check for a change in traffic pattern coinciding with the deploy: A marketing push, a new feature launch, or a bot/scraper hitting your endpoint at the same time as a deploy can masquerade as a deploy-caused regression.
- Check cache-miss rate specifically if spend or latency both moved: A deploy that changed prompt structure (even reordering a few lines) can break prompt caching eligibility, raising both latency and cost simultaneously.
Confirming and Closing the Loop (14-18)
- Cross-reference the trace spans from the affected window: Pull a handful of traces from just after the deploy and compare span-level latency (model call vs tool call) to a baseline trace from before the deploy, to localize where the extra time is going.
- Cross-reference structured logs for the same requests: Check whether prompt length, token counts, or stop reasons shifted for calls right after the deploy versus before.
- Test a rollback in staging, not production, first if the cause is still unclear: Reverting the suspected deploy in a staging environment and reproducing the same traffic pattern confirms attribution without a production rollback's risk.
- If confirmed, roll back or hotfix, then verify the metric returns to baseline: Don't close the incident until the dashboard shows latency/error rate back in its normal range, not just that you shipped a fix.
- Write down what you found, including the false leads: A short incident note listing what you checked and ruled out saves the next person from re-walking every dead end.
FAQs
What's the very first thing to check when latency spikes unexpectedly?
- The exact timestamp the anomaly started.
- Your deploy log for that same window.
- Anthropic's status page, to rule out a provider-side cause before digging further into your own changes.
How do I tell if a latency spike is caused by my deploy versus an Anthropic-side issue?
Check Anthropic's status page for the same time window first. If it shows no incident, and other unrelated services in your stack aren't affected at the same time, the correlation points toward your own recent deploy rather than a provider-side cause.
Why check for prompt changes specifically, not just code changes?
Prompt edits are easy to bundle into an otherwise unrelated deploy and easy to overlook when scanning a diff for "real" code changes. A longer or restructured prompt can shift latency and, separately, can change refusal or output behavior in ways a pure code diff wouldn't hint at.
What if the deploy log shows several deploys in the same window?
Diff each candidate deploy individually rather than assuming the most recent one is responsible. Cross-reference trace spans and structured logs from just after each deploy's timestamp to narrow down which one actually correlates with the metric shift.
Why would a deploy raise both latency and spend at the same time?
A common cause is a change that breaks prompt caching eligibility, for example reordering or editing the start of a prompt. That forces full-price, full-latency processing on every call instead of the cheaper, faster cached path, which shows up as a joint spike in both cache-miss rate metrics and latency.
Should I roll back immediately, or investigate first?
If the cause is already clear and confirmed, roll back or hotfix right away and verify the metric returns to baseline. If it's still unclear, test the rollback in staging first to confirm attribution before rolling back production, so you don't lose the chance to diagnose the real cause under a reverted deploy.
How do trace spans help narrow down where the extra latency is coming from?
Comparing span-level latency (model call spans versus tool call spans) between a trace from just after the deploy and a baseline trace from before it shows you exactly which part of the request got slower, rather than just knowing the whole request did.
What if the traffic pattern changed at the same time as the deploy?
Check for a coinciding traffic change (a marketing push, a feature launch, unexpected bot traffic) before concluding the deploy itself is the cause. A traffic spike can produce a latency spike on its own, independent of any code change.
Do I need deploy markers on my dashboard for this checklist to work?
They make step 3 nearly instant, but they're not required; without them, you fall back to manually cross-referencing your deploy log's timestamps against the metric's anomaly start time. Adding deploy markers to your latency/error dashboards is a worthwhile follow-up after your first incident without them.
Why write an incident note even after the issue is resolved?
A short note listing what you checked, including the dead ends, saves the next person (possibly you) from re-walking the same investigation from scratch during a future incident. It's also the raw material for tightening this checklist itself if you hit a step it was missing.
What role do structured logs play beyond confirming the deploy correlation?
Comparing prompt length, token counts, and stop reasons in logs from before and after the deploy can reveal the specific mechanism, for example a longer prompt (raising token counts and latency together) or a new tool causing more tool_use stop reasons and extra round trips.
Is checking a model version change really necessary if I didn't intend to change models?
Yes, because model routing or config changes sometimes ship as an unintended side effect of an unrelated deploy, especially if the model name is set via a config file or environment variable that got touched incidentally. Confirming the actual model used in affected requests rules this out quickly via your structured logs.
Related
- How Observability Works for LLM Applications - the mental model behind why deploy correlation matters.
- Instrumenting Agent Loops with OpenTelemetry Tracing - the trace spans referenced in steps 14-15.
- Integrating Claude Usage and Cost Dashboards with Datadog - where deploy markers typically live alongside latency graphs.
- Alerting on Spend Spikes and Rising Cache-Miss Rates - the alert that often triggers this checklist in the first place.
- ADR Template: Defining Your OTel Span Schema for Tool Calls - documenting the span schema that step 14 depends on.
- Observability Best Practices - a consolidated checklist across this section.
Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official
anthropicPython SDK (latest 0.x release). Model names, pricing, and SDK versions move quickly - verify current specifics at platform.claude.com/docs before relying on them.