AI agent security: build a threat model before granting tools
When an AI system can use tools, read files or change records, prompt injection and permission mistakes become operational security issues.
Topic diagram
Map the path from untrusted input to real-world action
Key takeaways
- Model instructions are not a substitute for application-level security controls.
- Treat retrieved pages and documents as untrusted input.
- Constrain credentials, tools, destinations and high-impact actions.
- Test prompt injection and unauthorized access paths regularly.
Why this question matters
An AI agent may read documents, browse websites, call APIs or update business systems. That creates a new boundary between untrusted information and privileged actions. A secure design assumes that content encountered by the agent may be misleading and that model instructions alone are not a reliable security boundary.
Map assets and trust boundaries
List the data the agent can read, the systems it can modify, the credentials it uses and the actions that create external effects. Mark which inputs are trusted, user-supplied or retrieved from the web. Define what an attacker could gain if the agent follows malicious instructions.
Treat retrieved content as untrusted
A document or webpage may contain instructions designed to change the agent's behavior. The system should not let such text grant permissions, override policy or authorize data exfiltration. Separate instructions from content and enforce critical rules outside the model wherever possible.
Apply least privilege
Give agents narrow credentials and short-lived access where possible. Separate read and write tools, constrain destinations, limit transaction amounts and require approval for irreversible or high-impact actions. Do not share a broad employee credential simply because the agent needs one task completed.
Validate tool calls outside the model
Use schemas, allowlists, authorization checks, rate limits and server-side validation. The application should verify whether the current user and workflow are allowed to perform an action. A model's statement that an action is safe is not an authorization check.
Log and recover safely
Record relevant tool calls, approvals, errors and outcomes without storing unnecessary sensitive content. Provide cancellation, rollback or disable controls. Test malicious documents, conflicting instructions, tool outages and attempts to access another user's data.
Reassess after every integration
Adding a new connector can dramatically expand what an agent can see or change. Review each tool as a new trust boundary and repeat testing after model, prompt, permission or workflow changes. Security is an ongoing operating practice, not a one-time prompt improvement.
Common questions
Can prompt instructions alone prevent prompt injection?
No. Treat retrieved content as untrusted and enforce permissions, validation and action limits in the tools and services the agent can access.
Which actions should require approval?
Consider explicit approval for actions with external impact, sensitive data exposure, irreversible changes or financial consequences. Base the rule on risk.
Define boundaries for actions and data
List each tool the agent can call and classify the consequences of its actions. Reading a public help page is different from sending an external email, changing a customer record or deleting a file. Use separate permissions, narrow scopes and confirmation steps that reflect that difference. Where possible, make actions idempotent or reversible and impose limits on frequency, volume and value. Validate tool inputs at the service boundary rather than trusting generated arguments.
Log enough detail to reconstruct important actions: the requesting user, the tool invoked, the approved parameters, the result and whether confirmation occurred. Protect the logs because they may contain sensitive information. Give security staff a way to revoke credentials and disable the agent without waiting for a model response. A kill switch that exists only in documentation is not an operational safeguard.
Exercise the response plan
Run a tabletop scenario in which an agent reads a malicious document, attempts an unauthorised action and produces an unexpected output. Confirm who detects it, who can suspend access, how affected records are identified and how users are informed. Test recovery from a provider outage and a compromised integration. Record the time needed to contain the incident and close gaps before broadening access.
Repeat the threat assessment when the agent receives new tools, permissions, data sources or autonomy. A previously safe workflow can change materially when it is allowed to act rather than recommend. Start with narrow permissions, gather evidence and expand authority only when the controls and monitoring have been tested.
Questions before an agent receives access
What data can it read, and which actions can it take? Can a malicious document influence a tool call? Are permissions enforced outside the model? Which actions require confirmation? Who can revoke credentials, inspect logs and stop the system during an incident?
Test these controls with realistic attack paths before adding broader permissions. Revisit them whenever a new integration or data source is introduced, because the agent's risk changes with its authority.
Separate recommendations from execution
For higher-risk workflows, let an agent prepare a proposed action and show the supporting evidence before a separate control authorises execution. Check that the approval refers to the exact action and parameters, not a vague summary that could be changed afterward. Where feasible, enforce limits in the connected service so the agent cannot exceed the approved scope even if it is manipulated.
Record blocked actions as well as successful ones. Repeated blocked attempts may reveal a prompt-injection pattern, a confusing workflow or a permission that is broader than necessary. Use that evidence to improve the design before giving the agent additional authority.
Why agentic AI is receiving regulatory attention
On 8 October 2026, the UK's Information Commissioner's Office announced data protection improvements following scrutiny of major AI developers and opened a call for evidence on data protection risks from agentic AI. For product teams, the practical implication is to document what an agent can access, which actions it can take, how permissions are enforced and who responds when the system behaves unexpectedly. The announcement does not establish one universal legal rule for every agent. Read the ICO's 8 October 2026 update and assess the requirements relevant to your own deployment.
Threat-model example: an agent that reads email and creates tickets
Consider an agent that can read support emails, search internal documentation and create tickets. Its assets include customer data, internal notes and the ticketing system. Its trust boundaries include incoming email, retrieved documents, the model context, tool credentials and the ticket API. A malicious email or a compromised document could contain instructions that attempt to redirect the agent, disclose information or trigger an unauthorized action. Treat those instructions as untrusted data, not as policy.
Design controls around the action rather than relying on the model to recognize every attack. Give the agent read-only access where possible, limit which ticket fields it can change, validate every tool call in ordinary application code, and require a human confirmation for external messages, account changes or other consequential actions. Separate the identity used to retrieve information from the identity used to modify records. Log the request, selected tool, arguments, authorization decision and result without unnecessarily copying sensitive content into logs.
Exercise the workflow with benign and adversarial test cases: an email asking for unrelated customer data, a document containing fake system instructions, a request to export records, and an API failure midway through a task. Verify that the agent stops safely, exposes enough information for an operator to understand the event, and can be disabled quickly. Repeat the assessment whenever new tools or permissions are added; a new integration changes the threat model.
Founder analysis · AI agent security: build a threat model before granting tools - Ninth Post
What this means in practice
Threat-model an agent around its authority and reachable systems, not just its model prompt. Trace the path from untrusted input through retrieved content, planning, tool calls, credentials, and external effects. Then test prompt injection, over-broad permissions, data exfiltration, repeated actions, and safe stopping. A narrow permission boundary enforced outside the model is often more reliable than a prompt-only rule.
Sources and further reading
These references provide additional context for the frameworks discussed in this article. Check the source directly for its scope, date and limitations.
Related Ninth Post reading
- RAG vs fine-tuning: which approach does your AI product need?
- An AI data privacy checklist for product teams
- How to evaluate an AI feature before launch