We crossed our tenth production prompt sometime in late winter. Nobody noticed until the support role started returning malformed JSON for about four percent of requests and the downstream parser began throwing on a field that had quietly been renamed in a system message two days earlier. Nobody had touched the parser. Nobody had announced the prompt change. The diff existed in someone’s local working tree and had been pushed during a routine cleanup commit, mixed in with a typo fix and a refactor.
We had been telling ourselves that prompts were configuration. After that incident we started treating them like code.
What earned its keep
Three things turned out to be load-bearing. The first was version control with proper diffs and named authors. We had been storing prompts inline in Python files, which technically worked but produced diffs that were unreadable once a system message ran past about thirty lines. Pulling prompts into their own markdown files with their own commit history changed the calculus of review. A diff that highlights the renaming of one field is something a teammate will actually read. A diff buried inside a four-hundred-line Python function is something they will scroll past.
The second was content hashes. Every prompt in our registry now carries a hash of its rendered output, computed at load time. The hash gets logged alongside every model call. When something behaves strangely in production, we can reconstruct which exact prompt version produced the response without trusting timestamps or our memory of when we deployed. This sounds like overengineering until the first time we needed it. Then it stopped sounding like overengineering forever. There is a particular kind of relief that comes from being able to say “version 3a7f, not version 4c12” instead of “I think the one we shipped last Tuesday, or maybe the Wednesday one.”
The third was evaluation gates before merge. We keep a small set of golden inputs for each prompt, with expected output shapes (mostly schemas and presence checks) and a handful of content assertions where they matter. The eval runs on the pull request. A diff that breaks the gate cannot land without an explicit override comment from a second reviewer. The gate is not exhaustive. It catches the regressions we have actually seen before, which turns out to be most of them. New failure modes get added to the golden set when we find them, and the set has grown slowly without ever feeling like a maintenance burden.
Together these three practices solved the original problem. Silent changes became loud. Rollback paths became explicit. Staging and production stopped drifting because we could prove they were running the same hash. The combined ceremony adds maybe ten minutes to a typical prompt change. We have not had a recurrence of the parser incident.
What we tried and dropped
We also adopted things that did not survive contact with the actual work.
An approval workflow was the largest one. We set up a system where any prompt change touching a customer-facing surface required sign-off from a designated owner before merging. In practice, owners approved within minutes because the eval gate had already passed, and the override path existed for the rare cases where it had not. The approval step became a rubber stamp. It added latency without adding judgment. We removed it.
We also tried a branching strategy where each prompt had a long-lived staging variant that was tested against production traffic shadows before being promoted. This worked in theory. In practice, the staging variants drifted from production fast, because nobody wanted to maintain two versions of the same prompt during quick iteration. The shadow tests caught nothing the eval gate did not already catch. We collapsed back to a single branch with the eval as the gate.
A third thing we dropped was a structured metadata system around prompts, where each one carried a yaml block describing its purpose, its expected inputs, its owner, and its risk tier. The intent was good. The reality was that the metadata went stale within weeks of being written, and nobody trusted it enough to make decisions from it. We kept the owner field, which is now a single line at the top of the file, and dropped everything else.
The pattern in all three was the same. We added ceremony hoping it would force discipline. The discipline never materialized because the underlying need was not strong enough to keep the ceremony alive. The eval gate stuck because it caught real regressions on real pull requests. The owner field stuck because someone got paged once and had to find the responsible party at three in the morning. The rest fell away, and the parts we removed left no holes behind them.
The tension we have not resolved
There is a cost to running the eval gate on every prompt diff. The golden set has grown as we have caught new failure modes, and a full run now takes about ninety seconds. For a one-character typo fix to a comment, that ninety seconds feels excessive. For the four percent JSON regression that started this whole thing, ninety seconds was a bargain at any price.
We have not figured out the right shape for this. A few options exist. We could run a smaller smoke set on every diff and the full set on a schedule. We could let the author flag a change as cosmetic and skip the gate, accepting that some changes will be wrongly flagged and some genuinely risky changes will be marked cosmetic by accident or in haste. We could invest in faster evals at the cost of less coverage. Each of these trades something we currently have for something we currently want.
The deeper question is whether we are calibrated correctly about which changes are risky. A typo fix to a comment in a prompt is safe. A typo fix to a field name in an instruction block is not. We do not have a reliable automatic way to tell those apart, and so the conservative choice has been to gate everything equally. That feels right today. It might not feel right when the eval grows past five minutes, and we do not yet know what we will do then.
What we know is that the discipline we kept is the discipline that paid us back within weeks of adopting it. The rest, we let go of without much regret. The harder lesson is that we cannot tell in advance which category a new practice will fall into. The only way to find out is to adopt it, watch it for a month, and be honest with ourselves about whether it is doing real work or just sitting in the pipeline because we once thought it should.