Four days ago we wrote that weights we didn’t build are a dependency we can’t read. On November 2, a group at MIT CSAIL (Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab) published early results on something they call weight-space programming: a general-purpose frontier model, GPT-6 Astra, reads a network’s raw weights and writes new ones directly. No optimizer, no gradient descent, and in some experiments no feedback from running the network at all.
The experiment that matters for us is the last one. Qwen3.5-0.8B, a small open-weight reasoning model, has a documented habit of looping: it repeats itself until it hits the token limit. In the paper’s practice set, 85 of its first 96 answers ran to the 32,768-token ceiling. Astra was given the weights in a Python kernel on a CPU, some practice problems, and a budget of 50 forward passes. It traced the looping to a single direction written by the MLPs in layers 15 and 19, and subtracted that direction from 64 output columns in each. That’s 131,072 weights, 0.016% of the model. MATH-500 went from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and the share of answers hitting the token limit fell from 93% to 33%.
The authors are careful about what this isn’t. Gemma 3 270M, which rarely loops, didn’t improve at all. Their summary line: “Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.” The sycophancy experiment failed from weights alone and only worked once the model got an attention-pattern tool on text it had generated. The results are one edit, one model, one set of evaluation settings, and the logs and edited checkpoint are published but we haven’t seen an independent reproduction yet. We take it as what they call it, very early, and still think it changes four things about how we treat the models we run.
Readable to a model is not readable to us
Our line last week was that a hash tells us which file we downloaded and nothing about what the file does with a shell. That’s still true for us. What’s new is that there is now a reader, and the reader is another model.
That matters because of what kind of output a reader like this produces. When Astra says the loop lives in one direction in layers 15 and 19, that is a claim made by a model about a model. It has exactly the trust status of a reasoning trace, which is to say it’s worth reading and not worth obeying. We argued last week that a trace is a debugging aid and not a control, because the trace and the action come from the same weights with nothing forcing them to agree. A weight-reading model has the same gap: its account of what a network does is produced by its own weights, not by the network it’s describing.
The thing that rescues the Qwen result from being just a story is that the claim came with a test. Subtract the direction, run the forward passes, watch the loop rate fall. The hypothesis is the reviewable artifact. The diff isn’t. Nobody can look at 131,072 changed floats and say whether they’re a loop fix or something else. We can look at “this direction causes looping, here are the forward passes that show it” and decide whether the evidence holds. If weight edits start arriving in our supply chain, from vendors, from community fine-tunes, from our own tooling, that’s the review standard: the edit ships with its hypothesis and the measurements that backed it, or it doesn’t ship.
A behavior-sized change is a diff-sized nothing
The second thing is arithmetic again. 0.016% of the weights moved a benchmark by forty points. A full retrain of this model and a surgical edit to it produce files with the same name, the same architecture, the same size, and completely different behavior. The model’s name has stopped carrying information. “Qwen3.5-0.8B” names a family of files, and the one we have might be the published one or might be fifty forward passes away from it.
We already decided last week to record the weights hash on every tool-call record, not a marketing name. This result is the reason that decision was right, and it also shows its limit. A hash catches every edit, including a loop fix we’d have wanted. It can’t distinguish a repair from a modification we’d object to. The same procedure that removed a loop could, in principle, remove a refusal, and several people reading the paper noticed that within hours. The method ran on a CPU with numpy. The cost of making a targeted behavioral change to an open-weight model just fell from a training run to a prompt and an afternoon.
So every edited checkpoint is a new dependency. It inherits nothing from its base: not the evaluation results, not the egress profile, not whatever trust the published model had earned on our traffic. That was already our rule for fine-tunes. It now applies to anything whose hash differs by a single column.
The budget was the method
The third thing is the detail we keep coming back to. Astra’s edits at 10 and 25 forward passes barely helped; MATH-500 moved to 12.3% and 14.1%. At 50 passes the loop went away. The quality of the edit tracked how much the editor was allowed to test it. Nothing about the model’s reasoning changed between budgets. What changed was the amount of verification it could afford.
We’ve written about verification becoming the job once generation got cheap, and about a published catalogue of machine-written proofs where the unverified half carried the error that took down its dependents. This is the same shape at a smaller scale and with a clearer dial. The scarce resource in weight editing isn’t the insight. It’s the forward passes. An editing tool that reports its hypothesis and skips the budget it tested with has told us half of what we need.
There’s a sharper corner here too. The authors drew practice problems from the GSM8K and MATH training splits and evaluated on held-out sets, which is the right discipline. Nothing forces anyone else to follow it. A model that can edit weights against a target can edit them against a benchmark, and a published score for an edited checkpoint now tells us even less than it did. Our Oct 30 position applies with more force: passing an evaluation is evidence about the evaluation, and the evaluation that counts is one on our own trigger distribution, run by us, after the hash we actually deploy was fixed.
What we’re changing
Most of our posture from last week survives this paper intact, which is a relief. What moves:
- Edits travel with their hypothesis. Any weight change we adopt, from anywhere, comes with the stated mechanism, the forward-pass budget it was tested under, and the before-and-after measurements. We re-run the measurements ourselves on a held-out slice before the hash is allowed near production.
- Weight-reading is debugging, not control. When we use a model to inspect another model’s weights, the output is a lead to test, filed next to reasoning traces and post-incident interviews. The control stays at the boundary: egress, syscalls, tool permissions.
- Checkpoint identity is the hash, and the hash is the unit of trust. Evaluation results, egress profiles, and the right to touch production attach to a specific hash. A new hash starts over, however small the diff.
- Our own small models get a new repair path. The classifiers and gates we run are closer in size to the 0.8B model in the paper than to anything frontier. A targeted fix for a known failure, with the hypothesis and the test attached, is cheaper than a retrain and easier to review than one. We’ll try it on something low-stakes first and treat the result as a checkpoint like any other.
The part that stays with us is how thin the change was. A direction in two layers, sixty-four columns each, and a model that had been failing most of its problems started passing most of them. That cuts both ways. The things we depend on are more repairable than we thought, and more alterable than we thought, by exactly the same hand. Our job hasn’t changed. It’s to make sure the hand that touches the weights leaves a record we can check, and that nothing acts on our systems until we’ve checked it.