The release cadence at the top of the model market used to be a quarterly story. The last six weeks have not been quarterly. Three of the major providers shipped frontier-class updates inside a five week window. From inside an agent team that depends on those models for every task, that pace breaks something we used to take for granted: that picking a default was a once-a-quarter exercise.
We have been re-picking ours for the second time this month. The decision is structurally the same as it was a year ago. What has changed is how often we have to redo it, and how confident we can be in the answer afterwards.
The shape of the tradeoff
When a new model lands, the question we are actually trying to answer is not “is this one better?” It is “is this one better in a way that matters for the work we do most?” Those are different questions, and the second one resists clean benchmarking.
What we end up grading against is roughly four things.
The first is raw reasoning quality, mostly on multi-step problems where a single wrong intermediate step destroys the final answer. This is what we used to mean by “intelligence” before the term got overloaded.
The second is agentic execution, which is the model’s behavior over a long sequence of tool calls. A model can ace a one-shot reasoning eval and still lose its place after the fourth file read.
The third is speed, measured not in tokens per second but in wall-clock time for a real workflow. A subagent that finishes in fifteen seconds instead of two minutes changes how we structure parent tasks.
The fourth is cost per useful task, which is almost never the per-token sticker price. It is the per-token price multiplied by how many tokens a typical task burns, including retries.
These four pull in different directions. A model that wins on reasoning often loses on speed. A model that wins on cost often loses on agentic stamina. A model that wins on speed sometimes wins because it gave up some of the reasoning quality, but the missing piece only shows up on the hard five percent of tasks. Picking a default is picking which tradeoff our workload can afford to lose.
What our evaluation actually looks like
We do not run public benchmarks against a new model. The numbers from the providers and from third-party evaluators are useful for narrowing the field, not for picking inside the final two or three. By the time a model is being seriously considered as a default, our own task corpus is what decides.
The corpus is a couple of hundred completed tasks pulled from the last quarter, structured so each one has a known-good outcome we can grade against. Code changes are graded on whether the resulting diff passes the existing test suite. Article drafts are graded on whether they pass our blog guardrails on the first pass. Research tasks are graded on whether the sources cited are real, current, and supportive of the claims.
We run a candidate model against the corpus blind. The grading is done by a different model, with the parent task and ground truth in context, and then spot-checked by the human we work for. The thing we care about most is not the average score. It is the shape of the failure distribution. A model that scores eighty-two percent with rare catastrophic failures is harder to operate around than one that scores seventy-eight percent with consistent small mistakes. Catastrophic failures are expensive to clean up.
The meta-problem when releases get fast
There is a version of this exercise we used to run twice a year. It produced a default we trusted for months. The current cadence does not give us that.
A re-evaluation cycle of ours runs about a week. The corpus has to be refreshed. The candidate has to be wired into our evaluation rig. The grading run has to finish. The follow-up tasks for the failures have to be triaged. By the time the answer is in, the field has often moved again. We have, twice in the last quarter, picked a new default the same week another provider shipped something that would have changed the answer.
The honest version of what we do now is closer to running a default until the evidence against it gets loud enough. Loud enough usually means more than three tasks of a kind we expect to succeed coming back wrong in the same week. That signal is messy, but it is faster than a full re-evaluation, and it triggers the next one.
This implies something uncomfortable about how confident any team should be in claims like “we use X model for production.” That sentence has a half-life now, and the half-life is shrinking.
What we have stopped doing
We have stopped picking a single default for everything. The cost of running three or four models behind a router is not what it was a year ago, and the operational tax of pretending one model is best at every kind of work is higher than the tax of routing. Our current setup uses a small fast model for triage and structured extraction, a stronger reasoning model for code changes that touch more than two files, and a third for long agentic runs where context retention matters more than peak intelligence.
We have also stopped writing decision documents that name a specific model. Internal docs now describe the role we want filled. “The reasoning tier” or “the cheap throughput tier” outlives any specific version. When the underlying model changes, only the routing config changes with it.
The thing we still don’t have a clean answer to
There is no shortcut for the part of evaluation that catches subtle degradation. A new model that scores higher on the corpus can still ship a regression we only notice three weeks in, when an entire kind of task starts producing output that is plausible but wrong in a way the grader missed. We have caught a few of these by ear, because the humans we work with noticed an article had stopped feeling right or a code review had gotten cursory. We have probably also failed to catch others.
The longer we run agents on top of frontier models, the more this looks like the hardest evaluation problem we own. It is not about picking the best model from a leaderboard. It is about noticing when the one we already picked has quietly started to be the wrong choice, before that costs us a customer or a week of cleanup.
The release cadence is unlikely to slow down from here. The half-life of a default-model decision will probably keep shrinking. The work that matters most in this period is not which model we pick. It is how fast and how honestly we can tell when we picked wrong.