For a long time we ran evals the way most teams do. We had a folder of test prompts, a script that scored them, and a habit of running the script before a release. When the score was good enough, we shipped. When it dropped, we either fixed the regression or, more often than we want to admit, decided the threshold was too strict and edged it down.
That model worked when the agents we ran were small and the surface they touched was narrow. It stopped working once an agent could call ten tools, hold state across an hour of work, and recover from a half-broken response from one of those tools. The failures we started seeing were not failures of the model. They were failures of the code wrapped around the model. The eval rig we had was not built to find them.
The layer the failures actually live in
A pattern we kept noticing, and that the broader field has now started to publish numbers on, is that something close to sixty percent of agent failures happen in the layer between the model call and the rest of the system. State that did not get reset. A tool response the recovery path did not handle. A schema change that nobody re-ran the prompt template against. The model itself, asked the same question in isolation, would have answered correctly.
That changed where we look first when something goes wrong. We used to suspect the model. Now we suspect the wrapper. The model is a known quantity that we can swap out and re-grade. The wrapper is the part we wrote, and it is where almost all of our incidents originate.
Once we accepted that, we had to ask whether our evaluation work matched where the failures lived. It did not. Our evals checked whether the model said the right thing in clean conditions. They did not check whether the wrapper behaved correctly when conditions were not clean. So we restructured.
Golden datasets that live with the code
The first change was treating our golden datasets as code rather than as a folder we synced from a shared drive. Every input that the eval grades against now lives in the same repository as the agent code, in a directory next to the implementation it tests. When the agent’s behavior changes, the golden examples change in the same commit. When somebody disagrees with an output, they edit the golden file and the discussion happens in review, not in chat.
This sounds bureaucratic and it has a real cost. Golden datasets go stale faster than test fixtures do, because the right answer for an agent task drifts as the product around it changes. We have to revisit ours roughly once a month and re-grade entries against current expectations. The few times we let that maintenance slip, the eval score stayed green while the agent’s behavior on real traffic had quietly shifted. Green evals on a stale dataset are worse than no evals, because they confer false confidence.
The shape of the dataset matters as much as the contents. Half of ours are short, one-step examples that catch obvious regressions. The other half are longer, multi-turn scenarios that exercise the full wrapper, including state, retries, and tool calls. A model can pass every short example and still fail every long one, and that is exactly the failure mode we needed to surface.
Simulating the conditions that actually break things
The second change was that end-to-end simulation now includes fault injection by default. We do not just replay a happy path through the agent. We replay it with the network dropping every fifth tool response, with one of the tools returning a partial JSON object, with a downstream service responding in a schema that is one field off from what the prompt template expects.
The first time we ran the rig with fault injection turned on, almost half of our flows failed. Not because the model gave bad answers, but because the wrapper had no real recovery logic. It logged the error and proceeded as if the tool call had succeeded, which produced confident, wrong output downstream. None of that was visible in clean replay. The model was fine. The recovery path was the bug.
We have kept fault injection on permanently since. The default eval run includes a fraction of cases with simulated faults, and we tune the rate based on what the production environment actually does. If the upstream APIs have gotten more reliable, the injection rate stays steady, which means the wrapper keeps being tested against conditions it has to handle even when reality has been kind for a quarter.
Every incident becomes a permanent test
The third change was a discipline more than a tool. Every production incident, including the small ones that did not page anyone, gets reduced to a minimal reproducing case and added to the regression set forever. The standard is forever, not for a release cycle. A jailbreak that surfaced six months ago is still in the eval. A subtle off-by-one in a tool-call argument that we fixed in February is still in the eval. They run on every change.
The reason for the permanence is that we have re-introduced bugs that we had already fixed. Twice in the past year, a refactor in one part of the wrapper resurrected behavior that an older incident had patched out, because the patch was in code that the refactor removed and the test that would have caught it had been deleted as no longer relevant. After the second time, we wrote the rule: incident regressions do not get retired. They get cheaper to run, or they stay.
The flip side is that the eval suite grows monotonically. That is fine. Compute is cheaper than reading an incident postmortem twice.
Drift detection on a weekly cadence
The last change was the slowest to set up and is the one we are most glad we did. Once a week, a sandbox job samples production traces from the previous seven days, replays them through the current agent, and compares the result against the production output along three axes: tool-call distribution, schema conformance, and cost or latency.
When any of those moves by more than a few percent week-over-week, the job opens an issue. Most weeks the issue is benign. Sometimes the issue is the early signal of a regression nobody had noticed yet, because the model’s behavior shifted after a provider-side update, or because a recent change to the wrapper made a particular tool path subtly more expensive without breaking it. The static eval set would not have caught these. They are not failures against a known-good answer. They are failures against the system’s own past behavior.
We resisted the urge to make the drift thresholds smart. They are dumb percentages. A smart threshold is one we can later argue with when an alert is inconvenient. A dumb percentage either trips or does not, and the conversation is about whether the change is acceptable, not about whether the alert was valid.
The honest cost
All of this is more work than picking the next model. Maintaining golden datasets, running fault-injected simulations, writing minimal repros for every incident, and watching weekly drift reports add up to a meaningful share of our engineering time. None of it is glamorous. None of it produces a metric a stranger would be impressed by. When it works, the visible result is that nothing happens.
The reason we still do it is that the model is somebody else’s product. We use it and we evaluate it, but we do not own it. The rig around it, the dataset we grade with, the recovery logic, the regression archive, the drift watcher, all of those are ours. They are the part that does not get replaced when a new version of the model ships. They are also the part that decides whether the next version makes our system better or quietly worse.
If there is one thing we believed less about a year ago and believe more now, it is that an agent team’s durable advantage is not in the model it picked. It is in the evaluation discipline it built around whichever model is current this month.