AI agent risk assessment

An AI assistant can generate a poor answer. An agent connected to email, code, payment, identity, customer, or infrastructure tools can turn a poor decision into an external action. The risk boundary therefore includes the model, orchestration, memory, identity, tools, data, approval interfaces, execution environment, and people supervising it.

NIST describes AI agent systems as capable of planning and taking autonomous actions that affect real systems or environments. Its 2026 security work focuses on threats created when model outputs are combined with software functionality, including adversarial data, insecure models, misaligned actions, and the need to constrain and monitor access.

A risk assessment that lists only the foundation model misses the components that grant authority. Draw the execution path from user request to plan, retrieval, memory, tool selection, credential use, external action, observation, and final response. Mark trust boundaries and every place untrusted content can influence the plan.

The authority matrix is the core control artifact. Define permissions by action and consequence, not broad tool name. Reading one approved customer record is different from exporting all customer records; drafting a refund is different from executing it.

Enforce the matrix in the tool gateway, identity layer, policy engine, and transaction system. A system prompt that tells the model not to perform an action is behavioral guidance, not an authorization boundary.

Do not reduce agent risk to a single generic score. Assess the maximum plausible consequence of each action, the likelihood that the agent attempts it incorrectly, the probability that preventive controls fail, and the ability to detect and reverse the result. The highest-impact actions deserve specific scenarios and release criteria.

Do not let an agent silently inherit the full permissions of a developer, administrator, or end user. Use a distinct workload identity, short-lived credentials, explicit delegation, and policy checks that consider the user, agent, tool, action, resource, environment, and transaction context.

NIST's AI Agent Standards Initiative includes work on software and AI-agent identity and authorization. The field is still developing, so organizations should rely on established identity and access-management principles while tracking emerging agent-specific standards and protocols.

Human approval fails when the approver sees an abstract summary, lacks time or expertise, or approves a plan that the agent can later change. The approval screen should display the exact consequential action, target, key inputs, source evidence, uncertainty, limits, and a clear reject path.

Bind approval to an immutable action payload or validated transaction. If the recipient, amount, command, attachment, or parameters change, require approval again. Record the approver, decision, timestamp, information shown, and executed action.

Indirect prompt injection occurs when an agent ingests malicious instructions hidden in content such as email, files, web pages, or retrieved documents. NIST calls this agent hijacking and identifies the lack of separation between trusted instructions and untrusted data as a central problem.

NIST experiments added scenarios involving remote code execution, data exfiltration, and automated phishing. Those experimental results are not universal failure rates. The durable lesson is that security evaluation must be task-specific, adaptive, and repeated, while production controls must constrain the impact of a successful hijack.

Limits should be enforced outside the agent's reasoning loop. Define maximum transactions, recipients, records, tool calls, runtime, tokens, spending, retries, concurrency, and data volume per session and period. Use lower initial limits and expand only when evaluation and production evidence justify it.

Release approval should state the tasks the agent may perform, the environments and users included, the tools and limits enabled, known failure modes, monitoring thresholds, incident ownership, and the evidence supporting residual risk acceptance. A later increase in authority is a new release decision.

The incident plan should cover harmful actions, attempted privilege escalation, prompt injection, data exposure, runaway cost, tool misuse, model or supplier degradation, and loss of traceability. Define who can disable the agent, revoke credentials, block tools, preserve evidence, notify affected owners, reverse actions, and approve restoration.

Run a live exercise in the target environment. A shutdown control that exists only in architecture diagrams has not been proven. Measure time to detect, contain, revoke, reconstruct, recover, and communicate.