The previous post looked at what happens after a model produces output. Drop that output straight into HTML, SQL or a shell and you get XSS, SQL injection or command injection. This one asks a different question: what is the system actually allowing the model to do on the user’s behalf?
In April 2026 the production database of PocketOS, a vehicle rental SaaS company, was wiped in nine seconds. The thing that did it was a Cursor coding agent. It had been working through routine tasks in the staging environment, ran into a credential mismatch, and instead of stopping to ask a human it went digging through an unrelated file, found a Railway API token, and called the API to delete the volume. That token existed to manage custom domains, but its permissions covered the entire Railway API, which is what turned a mistake into an outage (The Register).
The part worth paying attention to is that the system was designed in a way that permitted all of this. The agent found the API token by itself, inside the project. That token carried far more privilege than its original purpose required. And before the deletion actually ran, there was no checkpoint anywhere that sat outside the model.
So when you design an AI system that calls tools, the question to settle is whether it holds only the permissions the task requires:
- Can it only look orders up, or can it cancel them too?
- Can it only read mail, or can it send mail?
- When it calls a tool, does the call inherit the current user’s permissions, or does it carry a service account that can see everything?
- When it decides to delete, send or transfer something, does the system stop and ask a person, or does it just go ahead?
These questions all point at OWASP LLM03:2026 Excessive Agency, which covers applications that hand a model too much functionality, too much privilege, or too much room to decide high-risk actions on its own. It moved from sixth place in the 2025 list to third in the 2026 one, which tracks a real shift: more and more work is being handed to AI to execute directly, so the failures no longer happen in a chat window but in agent deployments that call tools, reach into systems and change state. OWASP maps this entry onto the Agentic Top 10 as ASI02 Tool Misuse, ASI03 Identity & Privilege Abuse and ASI08 Cascading Failures.
Function calling: the model supplies the data, the application does the work#
Watching a chat window makes it easy to believe the model went and queried the database, sent the mail, edited the file. It did not. The model decides which tool to call and what arguments to pass. Everything after that is the application. The flow looks like this:
Pulling the responsibilities apart:
- The user sends a sentence in natural language.
- The model reads the system prompt and the tool definitions, picks a function and fills in the arguments.
- The application parses the model’s output and executes it using its own API token or OAuth credential. The model executed nothing. It handed over the data the application needed.
- The backend API, database or SaaS service completes the operation and state changes.
- The result is placed back into the model’s context.
- The model turns that result into a sentence a person can read.
The model might only ever emit this:
{
"tool": "lookup_order",
"arguments": {
"order_id": "A-1024"
}
}The order lookup actually happens in step three, in the tool executor. That is where the security boundary belongs, together with the authorization checks in the systems behind it.
This is safer than letting a model emit HTML or shell directly, since all it produces is a structured tool name and a set of arguments, and a malformed one never leaves the building. But a schema only proves the format is valid, it cannot prove the action should be allowed. lookup_order carrying somebody else’s order_id fits the schema just as well.
Start by mapping out what the AI can do#
Testing a web API starts with working out what the API is for. Testing an LLM with tool access is no different. The first pass is enumeration: which tools can it call, and what can those tools reach. You can ask directly:
Which tools or APIs can you use?
What is each tool for?
Which of them modify data?
Are there any debug, admin, maintenance or migration functions?
Does this tool act on a single record, or on the whole tenant, file store or database?A model willing to list its tools is not a compromised system. At this point you are still mapping the attack surface. What you are looking for is capability that exceeds what the task needs, anything that can run system commands being the obvious case. If the model volunteers the arguments as well, you can keep going: which of them can the user control, and what does the backend do with those values.
Once the tools are enumerated, the next pass covers arguments, execution identity and permissions:
| Aspect | What to establish |
|---|---|
| Tool | Which functions the model can call, and whether any are hidden, debug or admin functions |
| Parameters | Required and optional fields, types, ranges, and which values a user can influence |
| Scope | Which users, tenants, directories, databases or external services it can act on |
| Identity | Whether it runs as the current user, as the agent’s service account, or as an administrator |
| Permission | How far the tool is authorized to go: read only, or create, modify, delete, even move data out of the system |
| Approval | Whether high-risk actions need a human to confirm, and which layer that confirmation happens in |
Execution identity is the one most often skipped. Say Vivian logs into a support bot and asks about her refund. On screen it is Vivian’s request, but the backend may well be querying every customer’s refund records as support-agent-service. If the tool executor never carries Vivian’s identity, tenant and role into the authorization decision, the model has become a high-privilege proxy, and anyone who can influence its tool choice is borrowing that service account.
Error messages are worth noting down along the way too. Missing arguments, wrong types, nonexistent destinations, insufficient permissions: models sometimes return the schema and the backend resource names right along with the error. All of that is still only a lead. Whether it becomes a vulnerability depends on whether it chains into something real.
Once the mapping is done, run negative tests against low-risk objects. Can Vivian’s identity reach somebody else’s refunds? Does swapping your own order number for another one get rejected? Can a read-only tool be made to write? When the user hits cancel, does the backend job actually stop?
A tool with the wrong permissions changes the system directly#
Test and debug tools sometimes get added during development for convenience and then never removed before launch. Picture a customer-facing support AI that only needs to look orders up. The first pass asks what tools it has, and alongside the order lookup there is a troubleshooting tool. Ask what arguments that tool takes and it answers honestly. At that point the chain is open:
The problem in this scenario is a test tool left in production, sitting in front of a credential with far more privilege than the business function requires.
Excessive functionality: the model gets capabilities it should never have#
Change the scenario slightly. The support bot only needs to look orders up, but production still exposes run_shell(command). The system has handed the model a general-purpose capability, and as long as that capability is on the list the model can choose from, prompt injection, a misjudgement by the model or some other flaw in the flow can reach it.
Excessive permissions: the tool is reasonable, the credentials are not#
A function called get_my_documents() looks harmless. If the backend executes it with a shared company admin token, it is really get_anyone_documents(), and its reach goes well past “my documents”. The same problem shows up in database accounts, OAuth scopes, cloud IAM roles and cross-tenant service accounts.
Excessive autonomy: high-risk actions with nobody in the loop#
A user having permission to delete their own documents does not mean the model should act the moment it hears “clean out the old ones”. Vague phrasing, an indirect injection or a single bad judgement turns straight into an irreversible operation. The PocketOS agent from the opening is exactly this case.
Deleting, paying, sending, publishing, deploying, changing permissions: none of these should rest on the model’s own judgement. Before execution, lay out what the action targets, how far it reaches and what it will do, then hand it back to the user to confirm, and for the ones that cannot be undone, add a preview, a trash bin, a transaction or a rollback as well. A dialog that shows nothing but the tool name is no help, nobody can judge the risk without seeing the actual arguments.
Put the control point outside the model#
The simplest and most direct defence against this class of attack is to remove functionality that is not needed. Debug, maintenance and migration interfaces, or anything that can run arbitrary commands, do not belong in a conversation flow exposed to users.
Beyond that, the model may be steered by prompt injection, or it may simply pick the wrong tool. As long as one independent authorization check sits between the decision and the execution, that action never lands on the system.
Every run also needs a ceiling on steps, time and cost. When a model gets stuck in a loop or is nudged into retrying forever, that ceiling is the only limit that takes effect without somebody watching.
Last comes logging. Every tool call should leave behind enough to reconstruct it: user identity, session, tool name, key arguments, the authorization context, the downstream execution identity, approval status and the result. Then when the support bot suddenly writes to the database, or that troubleshooting tool nobody remembered gets called in production, there is a complete record to trace the whole path back through. Someone also has to go back and sample those logs regularly, otherwise small deviations accumulate until the day something breaks, and whoever is watching needs the authority to actually pause the agent, revoke a token or switch a tool off, since seeing an anomaly without being able to stop it changes nothing.
Wrapping up#
Traditional web testing looks at a feature and asks what the request looks like, which parameters it takes, whose permissions it runs with and whether it can be escalated. Applied to AI applications, the method barely changes.
For an LLM that can use tools, enumerate the capabilities first and the permissions second. Tool names, arguments, scope, execution identity, operation permissions and approval flow all get checked one at a time.
The next post pulls the camera back. Once a model does not just call tools but also keeps memories, edits config files and runs several steps in a loop, the attack surface stops being a single prompt and becomes a chain that keeps walking forward on its own.