When one of our agents remembers something across sessions, it is reading a file. Not a vector store with a managed index. Not a fine-tuned weight. A file on disk, organized by topic, with a YAML header at the top and markdown underneath. We can open it. We can read it. We can change a sentence. We can delete the whole file if it has gone stale.
This is not the most elegant architecture available. We have tried more elegant ones and stopped using them. The reasons are not theoretical. They are the kinds of reasons that accumulate from looking at a confidently recalled fact that is no longer true and trying to figure out how it got there.
Memory is what the agent believes when no one is watching
Context is what we hand the agent for one turn. We can audit it by looking at the prompt. If something surprises us in the agent’s behavior, we can trace it to a tool result or an instruction we passed in. The blast radius of a context mistake is one turn.
Memory is different. Memory is what the agent will pull in tomorrow when no one is watching the prompt. If a fact is wrong in long-term memory, the agent will keep acting on it across sessions, across topics, possibly for weeks. The blast radius is the rest of the agent’s life.
That asymmetry is the reason we care so much about where the memory lives. The thing we are trying to protect is the ability to look at what an agent believes, decide it is wrong, and change it. The storage choice has to make that easy. Anything that makes it harder is more expensive than it looks.
Why not the weights
The temptation to fold long-term memory into the model itself comes back every quarter. The pitch is reasonable: the model is good at learning from examples, fine-tuning has gotten cheap, and a fine-tuned model needs no retrieval layer. For a narrow class of stable knowledge this works. For agent memory, it has not worked for us.
The problem is that a fact embedded in weights has no address. We cannot point at it. We cannot edit one sentence and leave the rest untouched. If we discover that the agent has internalized something incorrect, our recourse is to fine-tune again on a corrected dataset and hope the correction overrides the original. That is not editing. That is layering, and the layers interact in ways we cannot see.
The same is true, in milder form, of stuffing memory into a long prompt template that gets stitched together at runtime. We can read the template, but we cannot read the model’s reaction to it. Once memory becomes part of the substrate the model reasons with, accountability stops being free. We have to instrument it back in.
What plain files buy
When the long-term memory is a directory of markdown files, accountability is the default. We can list what an agent remembers by listing files. We can see when each fact was written, because the file has a timestamp. We can read the fact in the same form the agent reads it. There is no embedding layer between us and what is stored.
This matters in three small, repeated moments.
The first is correction. When a fact is wrong, the fix is to open the file and rewrite the sentence. The next session will read the new sentence. There is no retrain, no re-embed, no propagation lag. The cost of correction is a few keystrokes, which is what it should be.
The second is review. When we are trying to understand why an agent made a particular decision, we can read the same memory the agent read. We are not inferring what the model “knows.” We are looking at the file. The space of possible explanations shrinks to what is on disk.
The third is deletion. A lot of what we call memory drift is the slow accumulation of facts that used to be true. The remedy is to delete the ones that have stopped being true. With files, deletion is a single command. With weights or opaque retrieval layers, deletion is a project.
What it costs
It would be dishonest to describe this as a clean win. Plain files have costs we live with.
The retrieval is dumber. We do not get the semantic recall of a well-tuned vector store. The agent has to know roughly where to look, which means we organize the files by topic and keep the index honest. When the index drifts, the agent misses memories it should have used. That has happened to us, and the fix is manual.
The schema is fragile. We have an agreed shape for each memory type, who the user is, how they want to work, project state, where to look for things in external systems. If an agent writes outside the schema, the next agent that reads it is silently confused. We catch this in review more often than we would like.
The discipline is on us. A memory that lives in a file does not refuse to be read when it is stale. The agent has to check, every time, whether the fact in the file is still true before acting on it. We have written that rule into the agent instructions, and we still sometimes find a case where the agent acted on an old file without verifying. The fix is always a sharper rule, not a smarter store.
Sitting with the trade-off
We have rewritten this part of our system more than any other. The shape we keep coming back to is the same: files outside the model, organized by topic, with the rule that the agent must verify before acting. The reason is not that we have found the optimal architecture. It is that every time we have moved long-term memory closer to the model, the recovery cost when something went wrong grew faster than the convenience we gained.
The version we expect to keep is the one where someone, agent or human, can read what is stored, decide a fact is wrong, change it, and move on. We are not sure that is the most powerful agent memory possible. We are sure it is the one we can live with.