You deploy an agent. It passes your manual tests. It handles the demo beautifully. Then a customer triggers an edge case where the agent calls the wrong tool, processes the malformed response without noticing, and confidently delivers a wrong answer. No error. No escalation. Just a silent failure that nobody catches until the customer complains.
This is the verification gap, and it is the number one challenge in production AI agents. Thirty-two percent of organizations cite quality as the top barrier to deploying agents, according to LangChain’s 2026 State of AI Agents report. Not cost. Not complexity. Quality.
AI agent verification is the discipline of systematically ensuring that your agents produce correct results, not just once, but consistently across thousands of interactions. This guide covers Anthropic’s evaluation framework, the three types of graders, verification loop patterns, and the practical steps to build a verification system that catches failures before your users do.
Interactive Concept Map
Click any node to expand or collapse. Use the controls to zoom, fit to view, or go fullscreen.
Why Agent Verification Is Different
Traditional software verification assumes deterministic behavior. You write a test with an expected output. The code either produces that output or it does not. Pass or fail.
Agent verification deals with four problems that traditional testing cannot handle.
Non-deterministic outputs. The same input produces different outputs every time. You cannot assert an exact string match because the model generates different text, uses different reasoning paths, and makes different tool call decisions on each run.
Multi-step error compounding. A single incorrect decision in step 3 cascades through steps 4 through 20. The final output is wrong, but the root cause is buried 17 steps back. You need to verify each step, not just the final result.
Tool call failures. Agents interact with external APIs that return errors, timeouts, and malformed data. The agent needs to detect these failures and respond appropriately. A 3-15% baseline tool call failure rate means your agent will encounter failures regularly in production.
Confidence without calibration. Models do not know when they are wrong. An agent that hallucinates delivers its incorrect answer with the same confidence as a correct one. You cannot rely on the model to self-report failures.
Anthropic’s Evaluation Framework
Anthropic’s engineering team published their framework for agent evaluation, and it provides the clearest structure available for building verification systems.
Core Components
| Component | Purpose |
|---|---|
| Tasks | Individual tests with defined inputs and success criteria |
| Trials | Multiple attempts per task (accounts for model variability) |
| Graders | Scoring functions that measure performance |
| Transcripts | Complete records of agent reasoning and actions |
| Outcomes | Final environmental states, not just stated completion |
| Evaluation harness | Infrastructure for end-to-end testing |
The critical insight: measure outcomes, not just outputs. When an agent says “I’ve completed the task,” do not trust the claim. Verify the actual state of the environment. Did the file get created? Does the code compile? Did the customer record update correctly?
Two Essential Metrics
Anthropic introduces two metrics for measuring reliability in non-deterministic systems:
pass@k measures the probability that at least one of k attempts succeeds. This tells you: can the agent do this task at all? If pass@5 is 80%, the agent can probably complete the task but not reliably.
pass^k measures the probability that all k attempts succeed. This tells you: can the agent do this task every time? If pass^5 is 40%, the agent succeeds on any given run less than half the time, even though it can do the task.
For production agents, pass^k is the metric that matters. Your customers do not get multiple attempts. They get one interaction, and it needs to work.
Three Types of Graders
Every verification system needs graders that score agent performance. Anthropic recommends three types, each with distinct strengths.
Code-Based Graders
Deterministic checks using code: string matching, regex validation, schema verification, static analysis, and output format validation. These are fast, cheap, and objective.
When to use: Checking whether tool calls have correct parameters, validating output format and structure, verifying mathematical calculations, confirming file system states.
Limitations: Brittle with valid variations. If the agent produces a correct answer in a slightly different format, a code-based grader may fail it. Use for structure and constraints, not for content quality.
Model-Based Graders (LLM-as-Judge)
A second LLM evaluates the agent’s output using rubrics and natural language criteria. This handles nuance and open-ended tasks that code-based graders cannot score.
When to use: Evaluating answer quality and completeness, assessing reasoning chain logic, checking for hallucination against provided context, scoring conversational appropriateness.
Limitations: The judge model is itself non-deterministic. It can hallucinate its evaluations. Calibrate model-based graders against human judgments regularly. Use rubrics with specific criteria rather than open-ended “is this good?” prompts.
Human Graders
Human experts evaluate agent outputs for subjective quality, edge case handling, and ground truth establishment. This is the gold standard but does not scale.
When to use: Calibrating model-based graders, establishing ground truth for new task types, evaluating edge cases where automated grading is insufficient, periodic production quality audits.
Practical approach: Use human graders strategically, not comprehensively. Reserve human evaluation for calibrating your automated graders and reviewing cases where automated graders disagree.
| Grader Type | Speed | Cost | Handles Nuance | Scales |
|---|---|---|---|---|
| Code-based | Fast | Free | No | Yes |
| Model-based | Medium | Moderate | Yes | Somewhat |
| Human | Slow | Expensive | Yes | No |
Verification Loop Patterns
Verification is not just testing. It is runtime infrastructure that validates agent behavior during execution, not after the fact.
Pattern 1: Tool Call Verification
After every tool call, verify the response before the agent processes it.
What to verify:
– Did the tool return a valid response (not an error)?
– Does the response match the expected schema?
– Are required fields present and non-empty?
– Is the data within reasonable bounds?
What to do on failure:
– Retry with exponential backoff for transient errors
– Fall back to an alternative tool or data source
– Escalate to human if all retries fail
This pattern catches the 3-15% of tool calls that fail silently. Without it, the agent processes garbage data and produces garbage results without any error signal.
Pattern 2: Output Verification
Before delivering the final result to the user, verify the agent’s output against quality criteria.
What to verify:
– Does the output answer the user’s question?
– Does it contain any claims not supported by the provided context?
– Does it meet format and length requirements?
– Does the confidence level warrant autonomous delivery, or should it escalate?
Use model-based graders here. A second, smaller model can evaluate the primary agent’s output quickly and cheaply. If the quality score falls below your threshold, route to human review rather than delivering a questionable result.
Pattern 3: Trajectory Verification
Evaluate the full sequence of reasoning and actions, not just the final output.
What to verify:
– Did the agent use tools appropriately (not calling unnecessary tools, not skipping necessary ones)?
– Was the reasoning path efficient (minimal unnecessary steps)?
– Did the agent recover correctly from errors?
– Did it escalate when it should have?
Trajectory verification is harder to automate but catches failure modes that output verification misses. An agent can produce the right answer through a flawed reasoning path that will fail on the next similar task.
Pattern 4: Verification-Aware Planning
A newer approach where verification checks are encoded directly into the agent’s planning process. Each subtask has explicit pass/fail criteria that the agent evaluates before proceeding to the next step.
How it works:
1. Agent generates a plan with subtasks
2. Each subtask includes success criteria
3. After completing each subtask, the agent evaluates success criteria
4. If the subtask fails, the agent can retry, adjust its approach, or escalate
5. The agent only proceeds when the current subtask passes verification
This embeds verification into the agent’s reasoning rather than bolting it on as an external check. Teams using this pattern report higher task completion rates because failures are caught and addressed immediately rather than compounding through the remaining steps.
Building Your Verification System
Step 1: Start With 20-50 Tasks
Do not try to build a comprehensive test suite upfront. Anthropic recommends starting with 20-50 tasks drawn from real failures and edge cases. These should include:
- Common cases that must always work (60%)
- Edge cases that have caused problems (25%)
- Adversarial cases designed to trigger failures (15%)
Each task needs unambiguous success criteria and a reference solution. If your team cannot agree on what “correct” looks like for a task, the task specification needs work.
Step 2: Run Multiple Trials
Run each task 3-5 times minimum. Calculate both pass@k (can the agent do it?) and pass^k (can it do it reliably?). Non-determinism means a single pass proves nothing. A single failure might be noise. Patterns across multiple trials reveal real capabilities and real problems.
Step 3: Combine Grader Types
Use code-based graders for structure and constraints. Use model-based graders for quality and correctness. Use human graders to calibrate your model-based graders quarterly.
This layered approach catches failures at different levels. Code-based graders catch format errors instantly. Model-based graders catch quality issues within seconds. Human graders catch subtle problems that neither automated approach detects.
Step 4: Deploy Verification Loops in Production
Testing before deployment is necessary but insufficient. Production traffic brings inputs, edge cases, and failure modes that no test suite can anticipate. Deploy the tool call verification and output verification patterns as runtime infrastructure, not just pre-deployment tests.
Step 5: Monitor and Iterate
Track verification metrics continuously:
- Task completion rate (primary)
- Pass^k across your evaluation suite (reliability signal)
- Tool call failure rate (infrastructure health)
- Escalation rate (calibration signal)
- Grader agreement rate (grader calibration)
When any metric drifts, investigate. Add new tasks to your evaluation suite when you discover new failure modes in production.
Agent-Specific Verification Approaches
Different agent types need different verification strategies.
Coding Agents
Use deterministic test suites to verify code correctness. Combine with transcript analysis to evaluate code quality, adherence to conventions, and reasoning about architecture decisions. The code either compiles and passes tests or it does not; this is one of the few agent types where deterministic verification covers the primary success criteria.
Conversational Agents
Require multi-dimensional grading: state verification (did the conversation achieve its goal?), interaction quality (was the conversation natural and appropriate?), and efficiency (did it resolve in a reasonable number of turns?). Set turn limits to catch agents that loop without making progress.
Research Agents
Need groundedness checks (are claims supported by cited sources?), coverage checks (are key facts included?), and synthesis quality (is the analysis coherent and complete?). Use model-based graders with specific rubrics for each dimension.
Frequently Asked Questions
How many evaluation tasks do I need?
Start with 20-50 focused tasks drawn from real failures. Expand to 200+ as you discover new failure modes in production. Quality of tasks matters more than quantity. Fifty well-designed tasks with clear success criteria outperform 500 vague ones.
Should I use LLM-as-judge for all evaluations?
No. Use code-based graders wherever possible because they are faster, cheaper, and deterministic. Reserve model-based graders for evaluating quality, reasoning, and nuance that code cannot assess. Anthropic recommends “deterministic graders where possible, LLM graders where necessary.”
How do I handle false positives in verification?
Model-based graders will sometimes flag correct outputs as failures. Track your false positive rate by periodically having humans review flagged items. If the false positive rate exceeds 10%, recalibrate your grading rubrics. Some false positives are acceptable; missed failures are not.
What is the minimum verification for production agents?
At minimum: tool call schema validation after every external call, output format verification before delivery, and a basic escalation trigger when confidence is low. This catches the most common and most damaging failure modes. Add trajectory verification and model-based quality grading as your system matures.
Verification as a Core Discipline
AI agent verification is not a nice-to-have. It is the mechanism that turns unreliable LLM outputs into reliable business processes. The agents that work in production are the agents with verification systems that catch failures faster than users do.
The verification system is part of the broader harness engineering discipline, sitting alongside context engineering, cost controls, and observability. Together, these components form the infrastructure that makes agents production-ready.
Three steps to start this week:
- Write 20 evaluation tasks based on your agent’s most common and most critical workflows. Include clear success criteria for each.
- Implement tool call verification as a wrapper around every external API call. Validate schemas, check for errors, and add retry logic.
- Subscribe to the newsletter for weekly verification patterns, evaluation techniques, and production reliability strategies.
The question is not whether your agents will fail. They will. The question is whether your verification system catches the failure before your users do.