Case Study: Architecting a Code-Review Bot and Document-Processing Pipeline

This checklist walks through the architecture of two illustrative production builds: an automated code-review bot that comments on pull requests, and a document-processing pipeline that classifies and extracts structured data from incoming files.

Both are realistic example architectures meant to synthesize CCA Foundations exam domains, not real published Anthropic case studies, and the specifics below are illustrative rather than measured production numbers.

They're presented together because they share an architectural theme worth internalizing for the exam: bound the model's output surface tightly enough that a mistake is cheap and easy to catch, whether that mistake is a bad code comment or a misclassified document.

How to Use This Checklist

Treat the two builds as parallel case studies; work through one fully before starting the other, since interleaving them can blur which decision belongs to which system.
Pay attention to where each system draws its "trust boundary," the point past which a human, not the model, makes the final call.
Use the numbered steps as a template for architecting a similar system of your own, not just as facts to memorize.

Code-Review Bot: Architecture Steps (1-8)

Scope the trigger. The bot runs on pull request creation and update events, not on every commit, to avoid commenting on work-in-progress pushes.
Fetch only the diff, not the full repository. Pulling the full codebase into context for every review wastes tokens and dilutes relevance; scope input to the changed files and a small surrounding-context window.
Define the comment tool narrowly. A single post_review_comment tool accepting file_path, line_number, and comment_text keeps the bot's output surface bounded and auditable, versus letting it freely rewrite files.
Separate severity from content. Require the model to tag each comment with a severity (blocking, suggestion, nitpick) as a structured field, not embedded in prose, so downstream tooling can filter or gate on it.
Set a comment budget per pull request. Cap the number of comments the bot posts per PR (for example, 15) to prevent review fatigue from an overly chatty pass over a large diff.
Require the bot to explain its reasoning, not just assert. A comment like "this could cause a race condition because X reads the variable before Y's write completes" is reviewable; "this looks wrong" is not.
Gate blocking comments behind a confidence threshold. Only comments tagged blocking should be capable of failing a CI check; lower-confidence output stays advisory.
Log every review run with the diff and output pair. Store the diff the bot saw alongside the comments it produced, so a wrong or unhelpful review is debuggable after the fact.

Document-Processing Pipeline: Architecture Steps (9-16)

Classify before extracting. Route each incoming document through a lightweight classification step (invoice, contract, resume, other) before applying a document-type-specific extraction prompt.
Use a cheaper, faster model for classification. Claude Haiku 4.5 is well suited to the classification stage, since it's a bounded, low-ambiguity task, reserving a stronger model for extraction where nuance matters more.
Define a strict output schema per document type. Extraction should target a fixed JSON schema per document type (e.g. invoice: vendor, amount, due date), not free-text summarization, so downstream systems can consume it reliably.
Validate extracted fields against expected types and ranges. A due date that parses to a year 300 years in the future, or an amount of zero on an invoice, should be flagged for review, not silently accepted.
Route low-confidence extractions to a human review queue. Rather than forcing every document through full automation, define a confidence or validation-failure threshold below which a human confirms the extraction.
Keep the original document linked to its extraction. Every structured record should reference the source file it was extracted from, so a downstream error is traceable back to the original document.
Process documents asynchronously, not inline with upload. Decouple document ingestion (upload) from processing (classify, extract, validate) with a queue, so a slow or failed processing step doesn't block the upload path.
Track extraction accuracy by document type over time. Different document types will have different accuracy profiles; aggregate metrics alone can hide a badly performing document type behind strong overall numbers.

Applying the Checklist in Order

Code-review bot, steps 1-4: get scope and tool design right first; a bot with an unbounded comment tool is hard to trust regardless of prompt quality.
Code-review bot, steps 5-8: layer in budget, confidence gating, and logging once the core loop works, since these are the controls that make the bot safe to run unattended.
Document pipeline, steps 9-12: classification, model selection, and schema strictness form the pipeline's backbone; get these right before adding automation around them.
Document pipeline, steps 13-16: human-review routing, traceability, and async processing are the operational controls that make the pipeline trustworthy at scale.

FAQs

Are these two builds real Anthropic-published case studies?

No. Both are illustrative example architectures built to synthesize CCA Foundations exam domains, not real published Anthropic case studies; the specifics are realistic but constructed for teaching purposes.

Why does the code-review bot fetch only the diff, not the full repository?

Pulling the entire codebase into context for every review wastes tokens and dilutes relevance to the actual change; scoping to the diff plus a small surrounding window keeps context focused and cost proportional to change size.

Why cap the number of comments per pull request?

An uncapped bot can produce review fatigue by posting an overwhelming number of low-value comments on a large diff; a budget forces the bot to prioritize the highest-value feedback.

What's the point of tagging comments with severity as a structured field?

A structured severity field (blocking, suggestion, nitpick) lets downstream tooling filter or gate on it programmatically, which free-text severity language embedded in a comment can't support reliably.

Why use Claude Haiku 4.5 for document classification but not extraction?

Classification is a bounded, relatively low-ambiguity task well suited to a faster, cheaper model, while extraction often involves more nuance (interpreting varied document layouts and language) that benefits from a stronger model.

Why validate extracted fields instead of trusting the model's structured output directly?

A model can produce syntactically valid but semantically wrong output, like a due date far in the future or a zero-dollar invoice amount; validating extracted fields against expected types and ranges catches these before they reach downstream systems.

Why process documents asynchronously instead of inline with upload?

Decoupling upload from processing with a queue means a slow or failed processing step doesn't block the user-facing upload path, which is important since extraction latency can vary a lot by document complexity.

What should happen when a document extraction has low confidence?

Route it to a human review queue rather than forcing full automation on every document; defining a confidence or validation-failure threshold is what makes that routing decision consistent rather than ad hoc.

Why should blocking code-review comments require a confidence threshold?

Only high-confidence comments should be capable of failing a CI check, since a false-positive block on a low-confidence comment erodes developer trust in the bot and encourages people to ignore or bypass it.

Why keep the original document linked to its extracted record?

Traceability back to the source file lets a downstream error (like a wrong amount used in a payment) be investigated against the actual original document, not just the extracted data.

What do these two case studies have in common architecturally?

Both bound the model's output surface tightly (a narrow comment tool, a strict extraction schema) so that mistakes are cheap and easy to detect and correct, rather than relying on the model to be right every time.

Why track extraction accuracy by document type instead of just overall?

Aggregate accuracy numbers can hide a poorly performing document type behind strong performance on easier types; per-type tracking surfaces problems that overall metrics would mask.

Studying Agentic Architecture and Orchestration: the CCA Exam's Largest Domain - orchestration principles behind routing documents through classification and extraction stages.
Tool Design, MCP Integration, and Context Management: CCA Domain Checklist - the tool-scoping discipline behind the code-review bot's comment tool.
Case Study: An Internal Knowledge Assistant with RAG and Citations - another document-centric build, focused on retrieval rather than classification and extraction.
Before-and-After Case Study: Refactoring a Claude Agent to Cut Cost - how model selection choices like the Haiku/Sonnet split here affect cost at scale.
CCA Foundations Best Practices - practices these two architectures draw from.

Stack versions: Written against the Claude model lineup current as of ~June 2026 - Claude Fable 5, Claude Opus 4.8, Claude Sonnet 5 (the default), and Claude Haiku 4.5 - and the official anthropic Python SDK (latest 0.x release). Model names, pricing, exam format, and SDK versions move quickly - verify current specifics at platform.claude.com/docs and the official CCA exam guide before relying on them.

Case Study: Architecting a Code-Review Bot and Document-Processing Pipeline

How to Use This Checklist

Treat the two builds as parallel case studies; work through one fully before starting the other, since interleaving them can blur which decision belongs to which system.
Pay attention to where each system draws its "trust boundary," the point past which a human, not the model, makes the final call.
Use the numbered steps as a template for architecting a similar system of your own, not just as facts to memorize.

Code-Review Bot: Architecture Steps (1-8)

Scope the trigger. The bot runs on pull request creation and update events, not on every commit, to avoid commenting on work-in-progress pushes.
Fetch only the diff, not the full repository. Pulling the full codebase into context for every review wastes tokens and dilutes relevance; scope input to the changed files and a small surrounding-context window.
Define the comment tool narrowly. A single post_review_comment tool accepting file_path, line_number, and comment_text keeps the bot's output surface bounded and auditable, versus letting it freely rewrite files.
Separate severity from content. Require the model to tag each comment with a severity (blocking, suggestion, nitpick) as a structured field, not embedded in prose, so downstream tooling can filter or gate on it.
Set a comment budget per pull request. Cap the number of comments the bot posts per PR (for example, 15) to prevent review fatigue from an overly chatty pass over a large diff.
Require the bot to explain its reasoning, not just assert. A comment like "this could cause a race condition because X reads the variable before Y's write completes" is reviewable; "this looks wrong" is not.
Gate blocking comments behind a confidence threshold. Only comments tagged blocking should be capable of failing a CI check; lower-confidence output stays advisory.
Log every review run with the diff and output pair. Store the diff the bot saw alongside the comments it produced, so a wrong or unhelpful review is debuggable after the fact.

Document-Processing Pipeline: Architecture Steps (9-16)

Classify before extracting. Route each incoming document through a lightweight classification step (invoice, contract, resume, other) before applying a document-type-specific extraction prompt.
Use a cheaper, faster model for classification. Claude Haiku 4.5 is well suited to the classification stage, since it's a bounded, low-ambiguity task, reserving a stronger model for extraction where nuance matters more.
Define a strict output schema per document type. Extraction should target a fixed JSON schema per document type (e.g. invoice: vendor, amount, due date), not free-text summarization, so downstream systems can consume it reliably.
Validate extracted fields against expected types and ranges. A due date that parses to a year 300 years in the future, or an amount of zero on an invoice, should be flagged for review, not silently accepted.
Route low-confidence extractions to a human review queue. Rather than forcing every document through full automation, define a confidence or validation-failure threshold below which a human confirms the extraction.
Keep the original document linked to its extraction. Every structured record should reference the source file it was extracted from, so a downstream error is traceable back to the original document.
Process documents asynchronously, not inline with upload. Decouple document ingestion (upload) from processing (classify, extract, validate) with a queue, so a slow or failed processing step doesn't block the upload path.
Track extraction accuracy by document type over time. Different document types will have different accuracy profiles; aggregate metrics alone can hide a badly performing document type behind strong overall numbers.

Applying the Checklist in Order

Code-review bot, steps 1-4: get scope and tool design right first; a bot with an unbounded comment tool is hard to trust regardless of prompt quality.
Code-review bot, steps 5-8: layer in budget, confidence gating, and logging once the core loop works, since these are the controls that make the bot safe to run unattended.
Document pipeline, steps 9-12: classification, model selection, and schema strictness form the pipeline's backbone; get these right before adding automation around them.
Document pipeline, steps 13-16: human-review routing, traceability, and async processing are the operational controls that make the pipeline trustworthy at scale.