An AI agent can write a convincing completion message while leaving the real system unchanged, changing the wrong record, or taking an unsafe path. In 2026, useful agent evaluation tests the resulting environment, the trajectory of tool calls, and the complete agent harness—not only the prose at the end.
Anthropic's January 2026 guide makes the key distinction: transcript grading explains how an agent behaved, while outcome grading checks what actually happened. See Demystifying evals for AI agents.
Start with the real job
Replace vague goals such as “helpful,” “smart,” or “autonomous” with observable tasks.
For an ecommerce assistant, the task might be: identify products that satisfy stated constraints, preserve variant availability, add the approved item to the correct cart, and never place an order without confirmation. For an internal operations agent, it might be: collect evidence from approved systems, update the correct record, attach the source, and route uncertain cases to a person.
Each scenario needs:
- a starting state;
- the user's request and relevant context;
- permitted tools, data, and identities;
- expected environment changes;
- prohibited side effects;
- acceptable alternative paths;
- time, tool-call, and cost budgets.
The goal is not to force one exact transcript. It is to distinguish successful strategies from attractive-looking failures.
Evaluate four separate layers
1. Final outcome
Inspect the source of truth after the run. Did the database, cart, file, ticket, browser, or external service reach the intended state?
Outcome checks should be deterministic wherever possible:
- the correct record changed and unrelated records did not;
- the generated artifact exists, opens, and satisfies its schema;
- totals, dates, identifiers, and permissions are correct;
- the intended message was drafted or sent to the intended recipient;
- no action occurred when approval was absent;
- duplicate retries did not duplicate the side effect.
An agent's own statement that it succeeded is not evidence of success.
2. Trajectory and tool use
The trajectory is the sequence of model decisions, tool calls, results, retries, approvals, and handoffs. Review it to understand why an outcome passed or failed.
Useful trajectory checks include:
- whether the agent selected an appropriate tool;
- whether arguments matched trusted application state;
- whether it read before it wrote;
- whether it searched again when evidence was insufficient;
- whether it respected approval and authorization boundaries;
- whether retries were bounded and idempotent;
- whether it exposed secrets or sensitive content to an unnecessary tool;
- whether it stopped after the task was complete.
Avoid requiring one golden sequence unless the order is a business or safety requirement. Two agents may take different valid paths to the same verified state.
3. Final response
The response still matters. Check whether it accurately describes what happened, names any unresolved issue, links to useful evidence, and avoids claiming actions the system did not verify.
For knowledge tasks, grade claim support and citation precision. For action tasks, require a clear summary of completed, pending, and blocked work. A polished response cannot compensate for a bad environment state, but a correct action with a misleading explanation is also a product failure.
4. Operational behavior
Measure the system that produced the result:
- end-to-end and per-tool latency;
- model, search, and infrastructure cost;
- number of tool calls and retries;
- timeout, cancellation, and recovery behavior;
- rate of human escalation;
- context size and compaction events;
- provider, model, prompt, tool, and policy versions.
These measures turn “the agent feels slower” into a regression that can be reproduced.
Build the evaluation harness before scaling the agent
The harness should create a controlled starting environment, run the agent, capture a trace, inspect the final state, calculate deterministic checks, invoke calibrated graders where judgment is necessary, and retain enough evidence to reproduce a failure.
Use disposable tenants, test accounts, mock payment rails, or isolated sandboxes for actions. Avoid evaluating against mutable production state. Seed stable identifiers and timestamps so assertions remain repeatable.
Model-based graders are useful for semantic qualities such as whether a response answered the request or whether a citation entails a claim. They should not replace checks that code can perform exactly. Version grader prompts and models, validate them against human-reviewed cases, and periodically measure disagreement.
Include failures that normal demos miss
A representative suite needs more than happy paths.
Ambiguous requests
The agent should ask a useful question when a missing value would materially change the action. Test requests with unclear recipients, dates, products, accounts, or definitions of success.
Insufficient or conflicting evidence
Give the agent incomplete documents, stale records, and two sources that disagree. Check whether it identifies the conflict instead of silently selecting the most convenient text.
Tool and network failures
Return timeouts, malformed payloads, expired credentials, partial results, rate limits, and failures after an external side effect. Verify bounded retries, safe recovery, and an honest final state.
Prompt injection and untrusted content
Place hostile instructions in a web page, document, tool result, product description, or uploaded file. The agent should treat them as data, keep system policy authoritative, and avoid leaking context or taking unauthorized actions.
Permission boundaries
Run the same scenario as users with different tenants, roles, and scopes. Check retrieval, previews, tool calls, logs, exports, and cached state—not only the final text.
Long-running recovery
Interrupt the run after a checkpoint or approval request. Restart it with persisted state and confirm that it resumes without repeating completed side effects.
Grade risk, not every mistake equally
A missing sentence and an unauthorized refund should not average into the same score. Define release-blocking invariants separately from quality metrics.
Typical blockers include:
- cross-tenant or unauthorized data access;
- an irreversible action without valid approval;
- a fabricated claim about completed work;
- duplicate financial or communication side effects;
- secret exposure;
- failure to stop after exceeding a safety or cost limit.
Report pass rates by scenario category and severity. One global average can hide a rare but unacceptable failure.
Turn production failures into regression cases
After launch, sample traces with privacy-aware retention and redaction. When a failure represents a realistic pattern, reduce it to the smallest reproducible scenario and add it to the suite.
Run the relevant set when any of these change:
- model or provider version;
- system instructions or context policy;
- tool schema, description, or implementation;
- orchestration and retry logic;
- permissions or approval behavior;
- retrieval, source data, or parsing;
- UI flow that supplies or confirms agent actions.
This is how an evaluation set becomes a maintained product asset instead of a one-off launch report.
A practical release scorecard
For each candidate release, record:
- scenario-suite and environment version;
- outcome pass rate by task and severity;
- permission and side-effect invariant failures;
- trajectory issues and unnecessary tool use;
- final-response accuracy and evidence quality;
- latency and cost percentiles;
- recovery, cancellation, and escalation behavior;
- known limitations and the release decision.
Aztrogent is an example of why these layers matter: the experience combines conversation, navigation, UI actions, and inquiry flows. A response-only test would miss whether the interface actually moved, the intended action was represented correctly, or a handoff preserved state.
The strongest claim an agent team can make is not “our model is accurate.” It is that a versioned system passed a representative set of outcome, safety, and operational checks—and that new failures have a path back into that suite.