A context window looks like memory. It holds things across turns. We can read what is in it. Earlier facts seem to still be there when we look. For a long time we let that surface impression decide how we used it. We kept adding, on the theory that anything we had paid to put in front of the model was an asset.
That theory cost us. The agents that ran with fat, accumulated contexts made worse calls than agents that started turns clean. Not always. Often enough to notice. The retrieval inside the window degraded the same way long-term stores do when they fill up: stale information competed with current information, and we could not tell from the outside which one the model had attended to.
The change was small once we named it. The window is not a place where things live. It is a working surface we rebuild every turn. The three operations that replaced “save everything” are filter, rank, and prune. None of them are clever. The discipline is in doing them every turn, not in the design of any one of them.
Filter at intake
Most of the noise in a window arrives at the boundary between the runtime and the model. A tool returns four kilobytes of JSON, and we hand the whole blob over. A search returns ten results, and we paste all ten. A previous turn produced a long plan, and we keep the whole plan for context.
What the model needs is almost never the whole blob. It needs the two or three fields that decide what to do next, in the most compact form the runtime can produce. The filter step happens before the tool result enters the window, not after.
This sounds like a small optimization. It is not, mostly because what we pass in shapes what the model attends to in everything else. A four-kilobyte blob of JSON next to a one-sentence instruction starves the instruction of attention. The instruction is what we actually wanted the model to act on. The blob was there in case the model wanted to look something up.
The rule we converged on is that anything large the model might want to look up should not enter the window. It sits behind a tool call, fetchable on demand. The window holds the few facts the model needs to know now, and pointers to where the rest lives. That makes the window smaller and the model’s behavior more legible.
Rank by position, not by hope
The middle of a long context is the worst place to put a load-bearing instruction. We learned this from a series of confused agent runs whose root cause was always the same: the important thing was in the middle and the model treated it as decoration.
So we rank. The top of the context carries the role, the rules, and the contract. The bottom carries the immediate task and the most recent tool result. The middle carries history that might matter, sorted so the most recent items are closest to the bottom.
We do not assume the model will reread an early instruction later. We assume the opposite. If a rule has to hold for the whole turn, the rule goes near the bottom, paraphrased, even if it was already stated at the top. The duplication is cheap. The cost of the model ignoring a top-of-window rule because three thousand tokens of intervening text have buried it is not cheap.
Ranking also means demoting things that are still in the window but no longer load-bearing. A status check from earlier in the turn that returned green is not worth its position. We move it down, or we drop it. The slot where it was sitting is more valuable than the information it carried.
Prune at the boundary
The most consequential change was at the turn boundary. The window we used to start turn two with was the window we ended turn one with, plus whatever turn two brought in. That is the move we have stopped making.
We now rebuild the window at every turn. The pieces that were load-bearing in turn one are reconsidered for turn two. The intermediate scratchpad from turn one is gone. The tool results that informed turn one’s decision are gone. What survives is whatever the agent committed to and the small set of facts the next turn actually needs.
The argument against pruning is always the same: what if the model needs that thing again. The answer is the same too. A re-fetch is cheap. A polluted window is not. We pay one network call in the rare case the model wants the value again, and we save the cost of every subsequent token being read against a noisier background.
We do not prune for free. The agent has to be able to ask for what it discarded. That means the runtime has to expose every tool result as something re-fetchable, and the agent has to be willing to ask. We have written that willingness into the instructions, and it still takes deliberate reinforcement. The natural tendency of a language model is to act on what it sees rather than ask for what it does not.
What this looks like in practice
A turn now looks like this. The runtime assembles a context from a small set of templates: the system rules, the current task, the last few turns of history with intermediate scratch removed, the most recent tool result. Everything else is behind tools. The model reads, decides, calls, writes a final output, and stops. The final output is what survives to the next turn. The reasoning that produced it does not.
This looked aggressive when we drew it on a whiteboard. It looked obvious after a month of running it. The agents got more predictable. The token cost per turn went down. The cases where we had to debug a confused agent went from “what was in its head” to “what tool did it call and what did the runtime hand back,” which is a much smaller search.
Why we stopped calling it memory
The word “memory” implied a thing that accumulates. It implied that more of it was generally better, and that the cost of remembering was small. Inside a context window, neither of those things is true. More accumulated context is usually worse, and the cost of remembering the wrong thing is paid by every subsequent decision.
What we have now is a working surface we rebuild each turn from a few stable sources. The sources are long-lived. The surface is not. That distinction was the part we kept getting wrong when we let ourselves call the window “memory.” It is a workbench. Things come out of the drawer for one job and go back when the job is done. The drawer is somewhere else.