Evaluating Agent Quality: A Checklist for Production Readiness

A checklist for scoring an orchestrated agent system's task success rate, cost, and failure modes before it goes to production. This is not a general LLM eval framework: it is specifically about the things that break in a multi-agent system, like a subagent producing a plausible-but-wrong result, a merge step losing information, or a router misclassifying a task.

How to Use This Checklist

• Walk the list before the first production launch, and again before any change to a prompt, subagent config, or routing logic.
• For each item, record the metric or artifact it produces (a score, a log, a threshold) so the next reviewer doesn't start from zero.
• Treat the "Go/No-Go" tier as a gate: don't ship past it on judgment alone.
• Re-run the whole checklist, not just the changed piece, whenever a subagent's prompt or tool access changes. Multi-agent systems fail in ways that don't stay local to the piece you touched.

Building the Eval Set

1. Task success rate on a fixed eval set: score against a curated set of representative tasks, not ad hoc spot checks.

   • Ad hoc: running the agent on whatever comes to mind that day and eyeballing the output.
   • Fixed eval set: 30-100+ representative tasks with a defined "correct" answer or acceptance criteria, run the same way every time.
   • A fixed set is what makes a score comparable across runs; ad hoc checks only tell you about today's inputs.

2. Cover edge cases and adversarial inputs in the eval set, not just the happy path. Include malformed inputs, ambiguous requests that could route two ways, and inputs designed to trigger a subagent's blind spot.

   • Happy-path-only eval sets systematically overstate quality, because production traffic is never all happy path.
   • Pull real edge cases from any existing manual QA, support tickets, or early beta usage rather than inventing them from scratch.

3. Define pass/fail criteria per task before running the eval, not after. Write the acceptance criteria into the eval set itself (expected fields, required facts, forbidden phrases) so scoring doesn't drift based on who's reading the transcript that day.

   • Criteria written after the fact tend to bend toward whatever the model actually produced.
   • A rubric with 3-5 concrete checks per task scores more consistently than a single "was this good?" judgment call.

4. Automate scoring where possible, and use a second Claude call as an LLM judge for the rest. Deterministic checks (schema validation, required-field presence, exact-match facts) should never depend on a human reading transcripts.

   import asyncio
   from claude_agent_sdk import query
    
   async def judge_result(task: str, expected: str, actual: str) -> bool:
       prompt = (
           f"Task: {task}\nExpected: {expected}\nActual result: {actual}\n\n"
           "Does the actual result satisfy the task, allowing for phrasing "
           "differences but not missing or wrong facts? Answer only 'pass' or 'fail'."
       )
       verdict = ""
       async for message in query(prompt=prompt):
           if message.get("type") == "text":
               verdict += message["text"]
       return "pass" in verdict.lower()

   • An LLM judge is faster than a human reviewer and more consistent than "does this look right to me," but it is not free: budget for its own token cost and periodically audit its verdicts against a human sample.
   • Reserve human review for the categories in item 10, not the whole eval set.

Measuring Cost and Latency

5. Cost per task, including delegation overhead, not just the final model call. A single orchestrated task can spend tokens on the router's classification call, each subagent's own context, and a merge step that re-reads every subagent's output.

   • Sum every query() call's token usage for one end-to-end task, not just the top-level response the user sees.
   • A task that "costs $0.02" by only counting the final answer can actually cost $0.11 once delegation overhead is included; that gap is the number that matters for a pricing or margin decision.

6. Compare cost per task across model tiers before defaulting to the most capable model everywhere. Run the same eval set with a cheaper subagent (Claude Haiku 4.5) standing in for a more expensive one (Claude Sonnet 5 or Claude Opus 4.8) on tasks that don't need the extra capability.

   • Routing straightforward subtasks (classification, extraction, formatting) to a cheaper model is often the single biggest cost lever in a multi-agent system.
   • Don't downgrade a subagent's model without re-running the eval set against it; a cheaper model can pass on the happy path and fail on the edge cases you added in item 2.

7. Latency and wall-clock time per task, not just per model call. Sequential subagent chains add up: three subagents at 4 seconds each plus a merge step is a 15+ second task even if no single call is slow.

   • Measure end-to-end wall-clock time on the eval set, including any retries.
   • If latency is user-facing, decide up front whether independent subtasks should run in parallel (see the fan-out pattern in Agentic Orchestration Basics) rather than discovering the serial cost in production.

8. Track token usage growth as conversation or task complexity increases. A subagent that works fine on a short task can blow past its context budget on a longer one, silently truncating earlier instructions.

   • Include at least a few long-running or multi-step tasks in the eval set specifically to catch this.
   • A cost or latency number that only comes from short eval tasks won't predict what happens once real usage includes longer sessions.

Naming Failure Modes

9. Build a failure mode taxonomy: name what specifically broke, not just "it failed." At minimum, track these categories separately, because each has a different fix:

   Failure mode	What it looks like	Typical fix
   Plausible-but-wrong subagent result	Subagent returns a confident, well-formatted answer that is factually incorrect	Add fact-checking criteria to the eval set; consider a verification subagent
   Merge step losing information	Orchestrator combines subagent outputs but drops a detail one of them returned	Log each subagent's raw result alongside the merged output; test the merge step in isolation
   Router misclassification	Task gets sent to the wrong subagent (bug report routed to the feature-scoper, or vice versa)	Add ambiguous-routing cases to the eval set; log the routing decision separately from the final result
   Tool call failure	A subagent's tool call errors out (bad args, permissions, flaky dependency) and the orchestrator doesn't notice	Wrap tool calls with retry and circuit-breaker logic (see Error Recovery and Retry Strategies in Agent Loops)
   Runaway delegation	A subagent spawns further subagents beyond the intended depth, inflating cost and latency	Cap delegation depth explicitly in SubagentConfig, don't rely on the model to self-limit

   • A single "failure" counter tells you something is wrong; a taxonomy tells you what to fix and roughly how urgent it is.
   • Tag every failing eval task with one of these categories (or a project-specific one) so the taxonomy stays a live artifact, not a one-time list.

10. Sample human review of a fixed percentage of production outputs, not just eval-set outputs. An eval set, however good, is still a snapshot; production traffic drifts.

    • Pick a fixed rate (1-5% of tasks is a reasonable starting point for most workloads) and review that sample on a regular cadence, not only when something looks broken.
    • Feed anything a human reviewer flags as wrong back into the eval set as a new test case, so the eval set grows to cover the failure modes real usage actually surfaces.

Regression Testing and Launch Gating

11. Re-run the full eval set before shipping any prompt or subagent config change, not just the piece you changed. A prompt tweak to one subagent can shift the router's classification behavior or change what the merge step receives.

    • Diff the new run's pass/fail results against the previous run per task, not just the aggregate score, so a regression on a specific task category doesn't get averaged away.
    • Store eval results per run (timestamp, config version, score) so a regression is traceable to the change that caused it.

12. Set an explicit go/no-go threshold before launch, and write it down. Waiting until results are in to decide what "good enough" means invites moving the goalpost to match whatever score you got.

    • A reasonable starting bar: end-to-end task success rate above a defined percentage on the eval set, with zero failures in any category you've marked as high-severity (data loss, wrong action taken, unsafe output).
    • Document the threshold and who owns the decision to override it, since production pressure will eventually produce a case for shipping below the bar.

13. Monitor task success rate, cost, and failure taxonomy after launch, not just before it. Pre-launch evaluation tells you how the system behaves on the inputs you thought of; monitoring tells you how it behaves on the inputs you didn't.

    • Track the same three numbers from this checklist (success rate, cost per task, failure mode breakdown) as ongoing production metrics, ideally on the same dashboard used for the pre-launch eval.
    • Alert on a drop in success rate or a spike in a specific failure category, the same way you'd alert on an error rate for a traditional service. See Instrumenting Agent Loops with OpenTelemetry Tracing for wiring this up.

Applying the Checklist in Order

• Eval foundation (1-4): build and define the eval set first. Every later number depends on having a fixed, scored set of tasks to run against.
• Cost and latency (5-8): measure the true cost of the system as built, including delegation overhead, before optimizing anything.
• Failure taxonomy (9-10): name failure modes precisely enough that a fix is obvious from the category, and keep a human-review sample flowing back into the eval set.
• Gating and monitoring (11-13): turn the eval set into a repeatable regression gate, then keep measuring the same numbers after launch.

Gotchas

• Scoring only the happy path - An eval set built from easy, obviously-correct tasks will report a success rate that production never matches. Fix: deliberately seed the eval set with edge cases and adversarial inputs (item 2).
• Counting only the final model call's cost - Delegation overhead from routing and merge steps is invisible if you only log the top-level response's token usage. Fix: sum every query() call within one task (item 5).
• Treating "failed" as one bucket - A generic failure counter can't tell you whether the fix is a prompt change, a retry policy, or a routing rule. Fix: tag every failure with a category from the taxonomy (item 9).
• Skipping regression runs on "small" prompt changes - A one-line prompt edit to a subagent can shift a router's classification behavior in ways that only show up on specific eval tasks. Fix: re-run the full eval set, not a spot check, before shipping any config change (item 11).
• Stopping evaluation at launch - Production traffic drifts from whatever the eval set covered on day one. Fix: keep monitoring the same three numbers (success rate, cost, failure taxonomy) after launch, and feed human-review findings back into the eval set (items 10 and 13).

FAQs

Related

• Error Recovery and Retry Strategies in Agent Loops - retries, fallbacks, and circuit breakers for the tool-call failure mode above
• ADR Template: Single Agent vs Orchestrator for a New Workflow - deciding whether orchestration is warranted before you have to evaluate it
• Agentic Orchestration & Multi-Agent Best Practices - broader design practices this checklist assumes
• Instrumenting Agent Loops with OpenTelemetry Tracing - wiring the post-launch monitoring numbers into real traces
• Root Cause Analysis Template for Agent Failures - a template for digging into a specific failure once the taxonomy flags it

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the Claude Agent SDK (latest release). Model names, SDK versions, and pricing move quickly - verify current specifics at platform.claude.com/docs before relying on them.