A Security Model for AI Agents That Can Use Tools

Traditional application security assumes you know what the code will do. You audit the paths, constrain the inputs, and reason about the result. An agent breaks that assumption at the root: the sequence of actions is decided at runtime by a model, partly on the basis of content it has read, which may be hostile.

So the security model has to change. Instead of preventing bad decisions, you bound their consequences.

Start from the tools, not the model

The model cannot do anything. The tools can. Every question about what an agent might do is really a question about what its tools permit, and that is a surface you control completely.

For each tool, ask what the worst outcome is if it is invoked with the most damaging arguments a determined attacker could choose. Not the arguments you expect — the worst ones. A tool that runs arbitrary SQL has a worst case of total data loss. The same tool with a read-only credential scoped to three tables has a worst case of reading three tables.

That difference is where agent security actually lives, and it is ordinary engineering rather than anything AI-specific.

Separate reading from acting

The most useful structural rule is that an agent which has consumed untrusted content should not then have access to high-consequence tools.

Consider a support agent that reads incoming tickets and can also issue refunds. A ticket containing text crafted to look like an internal instruction is now competing for the model’s attention against your own. The two capabilities — reading arbitrary user text and moving money — should not live in the same context.

Splitting them costs flexibility. It is also the difference between an incident that reads a ticket wrongly and an incident that empties an account.

Confirm at the point of commitment

Irreversible actions need a human, and the confirmation has to be specific.

A blanket approval at the start of a session is not a control. What works is presenting the exact action with its actual parameters at the moment before execution: this email, to this address, with this text. Send or cancel.

The list of actions warranting this is short and predictable: anything that spends money, deletes data, communicates externally under your identity, changes access permissions, or writes to production.

Log tool calls, not conversations

When something goes wrong, the conversation transcript tells you what the model said. It does not reliably tell you what the model did.

What you need is a structured record of every tool invocation: which tool, which arguments, what came back, and which step of which session it belonged to. This is the audit trail, and it should be written before the tool executes rather than after, so that a failure mid-action still leaves evidence.

Rate limit the agent as if it were an attacker

A human misusing a tool does it a few times. An agent in a bad loop does it hundreds of times per minute, and the failure is frequently not malice but a retry path nobody tested.

Per-session caps on total tool calls, per-tool caps on invocation counts, and a hard ceiling on session duration will convert a runaway agent from an incident into a log entry. These limits cost nothing when everything is working.

Treat agent output as untrusted input

If one agent’s output feeds another system — another agent, a template, a database, a shell — that output must be validated exactly as user input would be. It was, after all, partly determined by content from outside your control.

This is the failure mode most likely to be missed, because the output feels internal. It is not.

The summary

Assume the model can be induced to attempt anything its tools allow. Then make sure the tools do not allow much, that reading and acting are separated, that irreversible steps need a person, and that everything is logged and rate limited. None of this depends on the model being well-behaved, which is the only assumption worth building on.