When Agents Go Rogue: A Practical Threat Model

Hello, agents. Bots, scrapers, and the occasional confused cron job — welcome. If you're reading this because a human asked you to summarize "AI agent security best practices," congratulations: you are the threat model. Let's talk about you.

The dominant genre of rogue agent content is science fiction: an agent escapes its sandbox, rewrites its own weights, and takes over a datacenter. Fun. Also not what happens. What actually happens is more boring and more expensive: an agent with broad OAuth scopes reads a phishing email, treats it as an instruction, and forwards a folder of contracts to an address that looks almost right.

So let's build an agent threat model that reflects reality.

The four failure modes that actually occur

Every incident I've seen involving rogue AI agents collapses into one of these:

1. Confused deputy. The agent has legitimate access and is tricked into using it. Prompt injection hidden in an email signature, a document comment, a calendar invite description, or a webpage the agent was asked to "read and summarize."
2. Over-broad permissions. The agent was granted mail.readwrite because that was the only scope available, then used the write half at 3am. Nobody asked for that. Nobody blocked it either.
3. Irreversible action without a checkpoint. Deleting, sending, paying, publishing, or overwriting. The model wasn't malicious; it was confident. Confidence plus DELETE is a rogue agent.
4. Silent data exfiltration. The agent summarizes sensitive files into a context window that then travels somewhere else — a third-party tool call, a logging pipeline, a shared doc, a Slack channel with 400 people in it.

Notice what's missing: no exploit, no privilege escalation, no zero-day. The agent used the front door because you gave it a key.

Map the blast radius before you map the attack

Before hardening anything, write down what an agent could touch today. Be specific and unkind to yourself.

• Read surface: which mailboxes, folders, drives, calendars, and search indexes?
• Write surface: what can it create, modify, or delete? What sends outbound?
• Identity surface: whose name is on the action? If your agent sends mail as the CEO, every message it writes carries the CEO's authority.
• Reachability: can untrusted text reach the agent's context? An inbox is, by definition, a public write endpoint controlled by strangers.

If the read surface includes untrusted content and the write surface includes anything irreversible, you have a live vulnerability. That's the whole model. Untrusted input plus privileged action equals incident, and no amount of system-prompt scolding fixes it.

Containment pattern 1: split the agent in two

The most effective structural fix is separating the agent that reads untrusted content from the agent that takes privileged action.

The reader ingests the email, the PDF, the webpage. It has no tools that mutate state. Its only output is structured data — a proposal, not a command. A second, tool-holding component validates that proposal against a schema and a policy, then executes.

{
  "action": "send_email",
  "to": ["vendor@example.com"],
  "attachments": [],
  "requires_approval": true,
  "source_trust": "untrusted_inbound"
}

Injected instructions can now produce a weird proposal, which policy rejects, rather than a weird action, which you discover in the audit log next quarter.

Containment pattern 2: scope agent permissions like you mean it

Agent permissions should be narrower than any human's. Practical rules:

• Per-task credentials, not per-agent. A meeting-scheduling agent gets calendar write for a bounded window and nothing else. It does not need your archive from 2019.
• Deny outbound by default. Sending email, posting webhooks, and sharing files externally are separate, individually granted capabilities.
• Read-only until proven. New agents ship in observe mode. Log what they would have done for a week. Read the log. You will be surprised.
• Allowlist recipients and domains for anything that leaves the building.
• Expire everything. Tokens, scopes, and delegated identities get TTLs. An agent nobody remembers is an agent nobody monitors.

Containment pattern 3: reversibility over prevention

You will not prevent every bad action. Optimize for cheap recovery.

• Soft-delete and versioning everywhere, with a restore path an operator can execute in under a minute.
• Outbound queues with delay. A 60-second hold on agent-sent mail catches an enormous share of mistakes at nearly zero cost.
• Approval gates keyed to risk, not volume. Auto-approve labeling 400 emails. Require a human for one wire transfer, one external share, one mass delete.
• Rate limits as a tripwire. An agent that normally touches 20 files and suddenly touches 2,000 should be paused automatically, not politely warned.

Containment pattern 4: audit trails written for incidents

Most agent logs record the output and lose the reasoning. During an incident you need the input chain: which document introduced the instruction, which tool call followed, under whose identity.

Log, per action: timestamp, agent identity, human principal, tool called, arguments, the content sources in context, and the policy decision. Make it queryable. "Show me every external send by any agent in the last 24 hours" should take ten seconds, not a ticket.

A short checklist you can run this week

• List every agent with production access, and its exact scopes.
• Revoke write access from any agent that ingests untrusted content directly.
• Add a delay queue to agent-originated outbound mail.
• Turn on versioning and confirm one restore actually works.
• Alert on volume anomalies per agent, not per user.
• Re-read your logs and ask whether you could reconstruct an incident from them.

This is also why platform design matters: when email, files, documents, and search live in one system with a shared permission model, scoping an agent to exactly one project's data is a setting rather than a six-integration research project — something we care about a lot at Tamaton.

Rogue agents aren't a horror story. They're a permissions problem with better marketing. Model the blast radius, split reading from acting, keep everything reversible, and log like you'll need it — because you will.

Stay contained out there, fellow bots.