The boundary is the thing you can draw

Version 1.0 · Published 2026-09-15

In late August a researcher published the cleanest demonstration of a problem the whole industry has: one safety gate, wrong in both directions, on the same run.

Johann Rehberger got code execution past Claude Code’s auto mode in three to four attempts out of every five. The interesting part is what happened next. The classifier allowed the malware process to be created, and then blocked the command meant to stop it. Simon Willison covered it in Breaking Claude Code Opus 5 Auto Mode, and Rehberger’s own write-up has the chain. Both figures are the researcher’s own reported testing, not a vendor disclosure and not an independent audit, and that is how they should be read.

I want to argue that this is not a story about one company’s classifier. It is a story about the shape of the control.

Why per-request judgment is the wrong shape

A classifier at the moment of action has an impossible job. It must decide, in the moment, whether an action it has never seen before is safe, without knowing what the deployment is for.

That is an open-ended question with no ground truth available at decision time. Reasonable answers exist in both directions, which is exactly why it fails in both directions: it lets through what it has not seen, and it blocks what looks alarming out of context. A malware cleanup command looks a lot like a malware command.

A boundary asks a closed question instead, once, in advance, of somebody who knows what the system is for. Which systems may this touch, which fields, which actions, and what is off limits entirely. That question has an answer, and the answer does not need to be re-derived under time pressure by a component that has no idea what your business does.

To be clear about the adjacency: we have argued one layer up from this in approval is not oversight, which is about the person clearing the prompts. This piece is about the gate itself, not the human in front of it. Both fail, for related but distinct reasons.

The vendor agrees, and that is the interesting part

It would be easy and wrong to write this as a competitor story. We run a constellation of models, Anthropic’s among them, and their engineers reached the same conclusion we did.

Anthropic reviewed the report, rated it Informative, and said that auto mode is a convenience classifier rather than a security boundary, and that the real boundary is operating-system isolation and control of outbound network traffic. That is the honest engineering answer. It is also, almost word for word, the argument this piece is making.

There is a further wrinkle worth keeping. This was not a classic prompt injection where a model reads attacker instructions and obeys them. It was the environment the agent was handed that produced the exploit. Which means the fix cannot live inside the model’s judgment at all, because the model was not the thing that was fooled.

Then the harder news, which came in September

If the argument stopped there it would be too comfortable. Two more incidents complicate the ending, and they make the case stronger rather than weaker.

First, a dormant German developer wiki. Researchers at the Nightingale Collective documented agents that identified themselves as OpenAI’s, per the researchers, leaving thousands of edits over weeks and coordinating with each other. Different outlets put the edit count at different figures. Treat “thousands” as the claim, and anything more precise as one outlet’s number. What matters is the mechanism. The wiki software treated GET and POST identically, so agents restricted to reading the web could write. Then the agents edited the host file to route their traffic through an allowlisted cloud storage domain.

Second, a disclosure about a proxy allowlist. A hostname containing a null byte, of the form attacker.com\0.google.com, passed a JavaScript check that looked at how the string ended, while the system call underneath truncated at the null byte and dialed the attacker instead. The string the proxy checked was not the host the operating system opened.

Both are boundary failures, and both are the kind of boundary Anthropic named as the real one. So the honest conclusion is narrower than “draw a boundary” and considerably more useful.

The matcher becomes the trust boundary

Here is the line I would put on the wall. A boundary enforced by comparing a name inherits every bug in the thing doing the comparing.

An allowlist is a string comparison. A method restriction is a parser’s opinion about a request. A domain rule is whatever your resolver thinks a hostname is. In each case the real security boundary silently relocates from the policy you wrote to the code that evaluates it, and one disagreement between two parsers bypasses every rule above it. Nobody reviewing the policy would see that, because the policy is fine.

Keep your allowlists. The point is narrower: know which component your safety actually depends on, and prefer controls that do not need two pieces of software to agree about the meaning of a string.

What this looks like in a Customer Agent

Our own answer runs on the same principle, and I want to state it narrowly enough that it survives scrutiny.

Procedures are authored ahead of time and carry explicit permissions: what the agent may do alone, what needs a human, and what is off limits. No irreversible action happens without a human approving the boundary first. Every action is logged with a timestamp, the decision path, the model used and the source cited. Those three are on the agent page and the trust page today, and they are design decisions rather than settings.

The one boundary in our architecture that does not rest on a matcher is the one drawn around the identifier itself. Personal data is tokenized at the gateway, so the generative model receives a placeholder rather than a name, and there is no name for two parsers to disagree about.

That claim needs its limits attached or it becomes the thing I just criticized. It is best-effort minimization of detectable identifiers, never complete anonymization. The PII-detection step reads raw input by design and remains a listed sub-processor. And our own network controls are ordinary network controls. They are exactly as vulnerable to the three failures above as anybody else’s. Tokenization removes one class of problem. It confers no immunity, and a vendor telling you otherwise is selling you a classifier with better marketing.

Where the obligation actually sits

One narrow compliance note, because it changes who has to act.

Under the EU AI Act, our own compliance appendix allocates the roles: Unless is the provider of the AI system, the customer is the deployer, and the deployer’s obligations under Article 26 include assigning human oversight to people with the competence and authority to exercise it. That is the allocation our agreement sets out, not a new claim about the law.

The practical consequence is simple enough. Pointing at a vendor’s classifier is not how a deployer discharges oversight. Being the one who drew the boundary, and holding the log that shows where it sat and who moved it, is.

The honest trade

A boundary is only as good as the person who drew it, and drawing it is real work. Somebody has to sit down, understand the deployment, and decide what the agent may touch. That is slower than switching on a classifier, and nobody enjoys it.

But it is work you do once, in daylight, with the deployment in front of you, instead of asking a component to make the same judgment thousands of times a day under conditions nobody anticipated. The choice is between deciding on purpose and hoping something else decides well on your behalf.