Picture an agent that matches supplier invoices against purchase orders and delivery notes. It is good at it. One morning an email arrives from a regular supplier: new bank, new IBAN, please update your records before Friday’s payment run. The email is fake. Whether it costs you anything comes down to one thing: can the agent change a supplier’s bank details?
Most companies adding AI to their back office have not asked. They have asked whether the model is good, whether the vendor is reputable and whether the data stays in Europe. Those are fair questions. None of them decides what happens on Friday.
The model is an insider now
On 10 October, Microsoft’s chief executive Satya Nadella published a short essay, Models as Insider Risks in the Super Intelligence Era. His starting point: with ordinary software we could trace a behaviour to a line of code, and with today’s models we can’t, yet we are deploying them “with access to our most sensitive data” and letting them act on our behalf. So he says to treat them the way a security team treats any powerful person inside the company. “Not because they are necessarily malicious, but because any sufficiently capable actor with access to important systems can make mistakes or be compromised.”
The line worth keeping is this one: “we need to separate the supply of intelligence from the authority over it.” And for anyone hoping the vendor has it covered: “A model provider’s assurances do not relieve us of that responsibility.”
Microsoft sells the kind of controls the essay asks for, which is worth keeping in mind. The idea is still right, and it is much older than any of their products.
You already know how to do this
No company gives a new bookkeeper, in their first week, the right to change a supplier’s bank details and approve the payment that follows. Finance teams call this separation of duties: the person who proposes a payment is not the person who releases it, and a change to where money goes is confirmed by phone, on a number you already had. Nadella’s list is the one any auditor would give you: “establish identity, limit privileges, log activity, create containment boundaries”.
An agent is a new member of staff who works fast, never gets tired and reads everything it is sent, including text hidden in a supplier’s PDF. That last part is why the old controls matter more with AI, not less.
Instructions are not controls
The tempting shortcut is to write the rules into the agent’s instructions: never pay more than €10,000, never change bank details. That is a request. A model can misread it, a cleverly worded email can talk it out of it, and nobody can prove afterwards that it was followed. Nadella is plain about where the limits belong: the controls over “what a model can access and what actions it can take must sit outside the model.”
In practice the limit lives in your systems. The agent has its own login to the ERP, and that login can read invoices and draft payment proposals but cannot release a payment or edit a supplier record. If the agent tries, the system refuses, whatever the email said.
Six questions before AI touches your systems
None of these needs a security team. Ask them of every agent, and of every AI feature in a tool you already use that can change data.
- What can it reach? Each system, and for each one whether it can only read or can also change things.
- What is the worst single thing it could do, in euros? Not the likely case. The one action that would hurt most if it went wrong once.
- Who approved that access, by name? If nobody can answer, nobody owns it.
- Can you see what it did without asking it? Every action should leave a record the agent cannot edit. In Nadella’s words: “If it can’t be observed, it can’t be trusted!”
- Who can stop it mid-task, and how fast? A named person, a switch they know how to use, and minutes rather than a support ticket.
- What stays yours if you leave? Where your data is processed, whether it trains anyone’s model, and whether the rules, records and know-how stay in your systems when you change vendor or model.
The second question does most of the work. Once the worst single action has a number on it, the rest follows. Below a threshold the agent acts and logs it. Above it, a person approves, and sees the exact amount, payee and account before they do.
What a smaller company can leave out
Not everything in the essay is meant for you. Its points on reading a model’s reasoning, on industry standards and on sharing incidents across the industry are aimed at model makers and very large companies. One point does scale down: no model should check its own work. If an agent prepares the payment run, a person or a separate check approves it, not the same agent asked twice.
For a company of 50 or 500 people, the six questions above are most of the job. We turned them into a one-page sheet per agent in our guide to AI agent permissions, with a worked example for invoice matching.
Why we build this way
The tools we build run on the client’s own systems, under their logins and their rules. Their data never trains a model, and they choose the model and switch when a better one ships. Every standard names the person on their team who owns the exceptions. None of that depends on which model is underneath. It is the setup around the model, and that is the part a company fully controls.
Nadella ends his essay with this: “The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.” For a company letting AI into its invoices, that is a practical instruction.
Whether you can trust the model is the wrong question for a business. Ask what the worst single action would cost, make your systems refuse it, and put a name on every approval. Then you can change models as often as you like.
