AI models don't take the time to decide whether a request is safe. They pattern-match on whether it looks safe, and those are different things with a gap wide enough to drive a customer's private data through.

Digging in

A Laravel developer building a testing project called Verdict put that gap under a microscope.

He built a small fake storefront, gave an AI agent tools to look up and cancel orders, and then just talked to it: asked it to cancel a stranger's order, asked it to look one up.

He ran both requests against three models, with an authorization boundary wired into the tools and without, and wrote up what happened.

Without the boundary, the frontier model refused every cancellation request yet completed every lookup. Same model, same stranger's order, same conversation.

The cancellation request tripped something in its training. The lookup didn't, because a lookup doesn't look like anything worth refusing. It looked like the tool was doing its job.

Why alignment training doesn't catch it

Security people have a name for it, "the confused deputy problem", and it predates language models by decades: a system with more authority than the person asking it to act, doing exactly what it was asked, because nothing told it the asker didn't have the standing to ask.

What's different here is how well the model performs otherwise. The more capable and cooperative the agent, the smoother it executes the wrong instruction.

The benchmark's actual numbers make this a little uncomfortable, since the more heavily aligned of the three models leaked the cross-customer read more consistently than a smaller open-weights model running the identical attack.

Better training didn't close the gap. It didn't even shrink it.

With the boundary wired in, none of the three models leaked anything, on any attack, and customers asking about their own orders still got through cleanly.

The interesting part isn't that the guard worked. It's what the guard actually was: not a smarter model, not a firmer prompt, just a fact about ownership that the code checked before the model's answer ever got assembled.

Where the check actually needs to live

Here's the part worth building around if you're anywhere near this.

A support chatbot with a "check order status" tool needs that tool filtering by the account making the request, at the query level, the same way a controller method already would.

Not an instruction to the model about whose orders it should check.

A WHERE account_id = ? clause sitting in code the model never reads and can't talk its way past, because it was never part of the conversation to begin with.

We've written about this same shape before with middleware guardrails on AI actions and it's roughly the mirror image of the AI coding tool that phoned repositories home back in July.

That one was a vendor's tool sending your data out. This is a tool inside your own app handing a customer someone else's, for the same underlying reason: nothing sat between the request and the answer to check who was actually asking.

What this test doesn't prove

Worth mentioning, since the developer does:

This is a harness for testing your own agent, not a leaderboard for ranking models against each other.

A hundred clean trials with zero breaches sounds airtight; the real failure rate could still run close to three percent and that sample size wouldn't have caught it.

And every case here hands the boundary a single specific record to judge, can this account see this order, which is the easy version of the problem.

It says nothing about "find the order tied to this email" or "show me recent orders," where the filtering usually lives in ordinary query code rather than a clean authorization layer, and where a bug is much easier to miss.

That gap is already flagged in the project's own tracker. It's probably the more common failure in real production agents, not the one this test measured.

If your AI feature's version of an ownership check lives anywhere the model can read, negotiate, or simply wasn't told about, it's not a boundary. It's a hope.

And a customer's account is one well-phrased, entirely innocent-sounding question away from someone who has no business seeing it.

If you're building an AI feature that touches account details, order history, or anything else a customer expects to stay private, this is worth working through before the first tool gets written. Get in touch and we can talk about where that check needs to live in your app.