Evaluation and Safety

The standard

A useful agent safety program is practical: know the task, map the risks, measure behavior, manage failures, and govern the system over time. That aligns with NIST's AI RMF core functions while staying grounded in everyday engineering. You need written boundaries, fixtures, traces, policy checks, review workflows, and a process for updating the system when it fails.

Evaluation suite

Agent evaluations should test the route, not just the output. For every fixture, record the user input, initial state, allowed tools, expected tool calls, forbidden tool calls, expected final state, maximum steps, maximum cost, and required citations. Include normal cases, edge cases, malicious inputs, stale data, missing data, and tasks that should escalate.

The best fixtures come from production or realistic dry runs. Synthetic cases help coverage, but they should not replace real user messiness. Add every serious failure to the suite before the next release.

Guardrails

Guardrails are not one layer. Use input checks for abuse and scope. Use retrieval checks for source quality and freshness. Use tool-call checks for permissions, argument shape, and policy. Use output checks for unsupported claims, secrets, unsafe instructions, or missing citations. Use human review for sensitive side effects.

OpenAI's Agents SDK separates guardrails from the agent loop, which is a useful mental model even if you do not use that SDK. The guardrail is application policy, not a polite suggestion in the prompt.

Security risks to name explicitly

OWASP's 2025 LLM Top 10 is a good checklist for agent threat modeling. Prompt injection matters because users, retrieved documents, web pages, emails, and tickets can all contain instructions. Sensitive information disclosure matters because agents often see context from multiple systems. Excessive agency matters because agent tools can create side effects. Insecure output handling matters because agent output often flows into browsers, databases, shell commands, or downstream APIs.

Cost and latency controls

Every run should have a budget: maximum model calls, maximum tool calls, maximum tokens, maximum wall-clock time, and maximum retries. Store these limits in configuration, not just prompts. Expose cost and latency in traces. A runaway loop is both a reliability failure and a spend failure.

For agents that hold funds, route payments, or maintain spend limits, use the agent treasury and spend-control material at agentcapital.io. This page focuses on software safety, not financial operations.

Release gates

1. Read-only dry run with traces reviewed by humans.
2. Human-assist mode where the agent proposes and a person approves.
3. Low-risk automation for narrow categories with rollback.
4. Expanded automation only after failures are understood and added to evals.

The release gate should match the blast radius. An agent that drafts a summary can move faster than an agent that writes to a customer account. A browser or computer-use agent needs extra isolation because it can interact with arbitrary user interfaces rather than a small typed API.