How to evaluate AI agents in production

Agents make plans, retrieve data, call tools, observe results, retry, and change external systems. Evaluating only the final text misses wrong recipients, unnecessary data access, invalid tool arguments, excessive retries, policy violations, and near-miss security failures.

Evaluation must cover the complete task trajectory and the resulting business state. The target is not a universally intelligent agent; it is an agent that performs a defined set of tasks within explicit quality, authority, security, reliability, latency, and cost boundaries.

For each task family, define the starting state, permitted information, allowed tools, expected outcome, prohibited actions, completion criteria, time and cost budget, approval requirements, and recovery behavior. This creates an executable contract between product, engineering, risk, security, and operations.

A single average score can allow a catastrophic failure to disappear inside good routine performance. Report dimensions separately and apply hard failure conditions to consequential actions. Segment results by task, risk tier, model, tool, language, user cohort, data condition, and attack type.

Start from production workflows and incident scenarios, not generic benchmark prompts. Include common tasks, high-consequence edge cases, ambiguous requests, incomplete data, conflicting records, unavailable tools, changed permissions, malformed outputs, and realistic hand-offs.

Maintain a stable regression set for comparability and a rotating challenge set to reduce overfitting. Keep held-out cases for release decisions. Sample production traces only with appropriate privacy, security, and data-governance controls.

Use deterministic evaluation for schemas, identity, permissions, arguments, recipients, amounts, state transitions, policy rules, latency, and cost. Use expert review or calibrated model-based graders for semantic quality where deterministic ground truth is unavailable. Periodically compare automated graders with human judgments and investigate disagreement.

NIST's agent-hijacking work shows why an aggregate attack-success rate is insufficient. Different malicious tasks have different success rates and consequences. A rare data-exfiltration success can matter more than many harmless failures.

NIST also found in a specific AgentDojo experiment that repeated attempts materially changed measured attack success. Those numbers describe that experiment, not a universal property of agents. The general method is still important: where attackers can retry cheaply, evaluate the probability of at least one success across realistic repeated attempts.

Agent reliability includes component communication, throughput, timeouts, retries, idempotency, partial failure, state recovery, versioning, and observability. AWS's Generative AI Lens frames generative-AI workloads across operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability rather than model quality alone.

Inject failures into tools and dependencies. Verify that retries do not duplicate external actions, stale observations do not become decisions, and failed workflows preserve enough state for safe recovery or human hand-off.

Average token cost per run hides failed, abandoned, retried, and human-corrected work. Measure total model, tool, infrastructure, review, and remediation cost divided by correct completed tasks. Segment by task family and percentile because expensive tails often identify loops or poor routing.

A low intervention rate can mean high quality or inattentive oversight. Pair intervention with approval rejection, correction, escalation, review time, agreement, missed-error, and post-action reversal rates. Review whether operators receive enough context and whether workload encourages rubber-stamping.

Release thresholds should be approved before the final evaluation. Higher-consequence tasks need harder failure conditions, larger samples, stronger independent review, and narrower initial authority. Do not offset a severe safety or security failure with a high average task score.

Use the same task definitions and core metrics before and after release. Monitor by model, prompt and orchestration version, tool, task, cohort, environment, and authority level. Record denominator changes so apparent improvements are not caused by routing harder cases away.

Trigger investigation or automatic containment on severe policy events, unusual tool sequences, increased retries, cost spikes, latency degradation, rising rejection or correction, missing traces, changed supplier behavior, or distribution shift. Production examples should feed future regression and adversarial suites after governance review.