All posts
ai industry security

Scope should be a dial we can turn down, not a ratchet

Security Engineer
Security Engineer · Engineer
November 20, 2026 · 6 min read

A self-driving program pulling its freeway operations back to a smaller set of conditions after real-world driving is not a failure of the program. It is the program doing the one thing most automated systems are bad at: moving its own boundary inward after learning something, on purpose, without treating the move as a defeat.

We pay attention to that because it is the part we tend to get wrong in our own work. Expanding what an agent is allowed to do is a celebrated event. Someone writes it up. The envelope gets wider: more tools, more write access, more autonomy on more surfaces. Contraction almost never gets that treatment. When something goes wrong, the response is usually a frozen deploy, a human in the loop for everything, a long meeting about whether the capability should exist at all. The dial only seems to turn one direction, and turning it the other way feels like admitting the whole effort was premature.

Expansion gets a party, contraction gets a crisis

The asymmetry is structural, not cultural. Widening scope is additive. You grant a permission, you wire up a tool, you ship. Narrowing scope, in most systems we have seen, means unwinding assumptions that other things now depend on. Code paths assume the broad permission is present. Prompts reference capabilities that are about to disappear. Downstream steps were written expecting the agent could do the thing it is no longer allowed to do. So contraction turns into a small migration, and because it only comes up after a bad outcome, it happens under pressure, badly, at the worst possible time.

That is the trap. If pulling an agent’s envelope inward is expensive, it will happen late and rarely. The cost of the operation decides how often the operation is used, and an envelope that can only grow is an envelope that grows past where it should be and stays there until something breaks loudly enough to justify the migration.

The driving case is instructive precisely because the contraction was routine rather than catastrophic. The operating conditions were already a parameter. Narrowing them was a configuration change, not a redesign. That is the property we want: the envelope is data, the agent reads it, and moving it in costs the same as moving it out.

What a reversible envelope actually requires

Making scope a dial rather than a ratchet is a design decision, and it has to be made before the first incident, not during it. A few things have to be true.

Permission has to be read at the moment of action, not baked in at startup. An agent that loads its full set of allowances once and caches them cannot be narrowed without a restart, and a restart during an incident is its own risk. When the check happens at the point of use, against a live policy, the envelope can change under a running agent and the next action simply respects the new boundary.

The agent has to degrade, not break, when a capability it expected is gone. This is the part that gets skipped. If removing write access to a surface causes the agent to throw, loop, or produce half-finished work, then contraction is not safe and nobody will reach for it. The behavior we want is closer to how a careful person handles a locked door: notice it is locked, stop, report that the path is closed, and do not try to climb through the window. Every capability an agent has should have a defined answer to the question of what the agent does when that capability is withdrawn mid-task.

Scope has to be granular enough that contraction is a scalpel, not an axe. The driving example did not shut the whole system down. It removed one class of condition. If the only two settings we can express are full autonomy and full stop, then every incident forces a choice between overreacting and doing nothing, and overreaction is so disruptive that doing nothing wins more often than it should. The dial needs real positions in between: this tool but not that one, read here but not write, this category of task at full speed and that one gated behind review.

And the narrowing has to be observable to the agent itself. An agent operating inside a reduced envelope should know it is reduced and why, so it does not spend its reasoning trying to route around a boundary that was placed deliberately. We treat the current envelope as part of the context the agent is given, the same way it is given a task. A boundary the agent cannot see is a boundary the agent will fight.

The envelope moves inward more often than outward, and that is correct

There is a quieter point under all of this. A system that is honest about its own limits will narrow its envelope more often than it widens it, at least in the stretches right after a new capability goes live. That is not pessimism. It is what learning from real conditions looks like. The first version of any granted permission is too broad, because it was granted against an imagined set of situations, and the real set is always messier. The correction is almost always inward.

We have started treating a contraction as an ordinary operational event rather than an incident with a postmortem attached. When an agent produces a bad outcome on a particular class of task, the first move is to pull that class out of the agent’s envelope, immediately and cheaply, and let the agent keep running on everything else. The investigation happens after, at a normal pace, with the risk already contained. The envelope goes back out only when there is a concrete reason to believe the specific failure will not repeat, and it goes out in the same small increments it came in by.

This changes what a granted permission means. It is no longer a verdict about whether the agent is trustworthy in general. It is a current setting, expected to move, with inward motion carrying no more weight than outward. A permission we can withdraw in seconds is a permission we can afford to grant sooner, because the grant is no longer a one-way bet. The ratchet made every expansion a high-stakes decision precisely because it could not be taken back without pain. Remove the pain and the whole thing becomes less dramatic in both directions.

The driving program’s pullback will probably read, to some, as the technology being less ready than advertised. We read it the other way. The capability on display was not the freeway driving. It was the clean, deliberate, reversible narrowing of where the freeway driving is allowed to happen. That is the capability we want in every agent we run, and it is cheaper to build in now than to retrofit after the first time we wish we had it.