On November 9, two Amazon engineers published an account of turning off a frontier model in production. The workload was labeling tickets in one of the company’s larger internal ticketing systems: customer escalations, capacity requests, automated alarms, security findings, sorted into twenty categories. For a long time each ticket went to a general-purpose model with a long system prompt, which reasoned over the label set and wrote back JSON. Now each ticket goes to a small decision model, a build of Strands Decider on a single GPU instance, which returns a probability for every declared label and a confidence score. The post reports 70 percent fewer tokens per ticket, one output token instead of about 170, and an average call time that dropped from roughly 4.2 seconds to under 0.9.
The cost line is what made the headline. The sentence we keep rereading is a different one: “The part we’d copy regardless of model is the three-way switch.”
We have written twice already about decision models, once about what a typed output cannot catch and once about who owns the calibration loop. This post is about something narrower, and it is the part of the story that belongs to testing. Moving a decision off a large model is not a model choice. It is a migration, and the Amazon write-up is one of the clearest public descriptions we have seen of how to run one.
A swap is four changes wearing one name
The obvious framing is “we replaced model A with model B.” Read the caveats in the post and the change turns out to be bigger than that.
The production label descriptions did not fit the new model’s 4,096-token window, so the team rewrote all twenty labels as one- or two-sentence criteria. That is a new prompt. On long tickets the new path drops the oldest comments to fit, which the old path never did. That is a new input policy. And the team found that the order of keys in the JSON state changed the answer, moving output probabilities by up to 0.19, so they now send sorted keys. That is a new serialization contract.
So the swap was a new model, a new label text, a new truncation rule, and a new input encoding, all shipped under a single name. If agreement drops after a change like that, there are four suspects, and an offline benchmark score on the new model alone cannot say which one did it.
The key-order finding deserves its own line in every checklist we keep. In most of our pipelines, the JSON we hand to a model is assembled by whatever code happened to build the object, and its key order is an accident of insertion. We treat it as a format detail. For a model that reads the state and returns a probability, it is part of the input, and a 0.19 swing near a threshold is the difference between one label and another. Once a probability is the output, the bytes of the input are the contract. Canonical serialization stops being tidiness and becomes a correctness requirement, and it needs a test that asserts it, not a convention someone remembers.
The switch is the test
The mechanism the team says they would copy regardless of model is a configuration value read on every message, with three settings. In the first, only the old model runs. In the last, only the new one does. In the middle setting, both run in parallel on live traffic, the old model’s labels are applied, and the two are compared. The team then “judg[ed] every disagreement.” Rollback is described as a 45-second config change.
Their own summary of the result: “Running both on real traffic, then judging every disagreement, told us more than any offline eval would have.”
From the QA side, this is the right shape for a reason that goes beyond caution. An offline test set measures the new model against labels someone produced at some earlier point, on a distribution someone sampled at some earlier point. The dual-run mode measures the new path against the incumbent on today’s tickets, including the long ones that get truncated, the oddly formatted ones, and the categories that only show up during an incident. The disagreement set is a test suite generated by production itself, and it is exactly the set of cases where the migration changes behavior. Agreement is cheap to confirm and uninteresting to review. The disagreements are where all the information is.
Two things about this setup are easy to miss. First, it requires that the old path keep working while the new one is being judged, which means the old prompt, the old credentials, and the old budget stay alive for the length of the comparison. Teams that cut the old path the moment the new one looks good lose their reference before they have finished using it. Second, judging disagreements is human or reviewer work with a cost, and the post does not say who judged or how. That cost belongs in the migration plan, not discovered halfway through.
After the cutover, keep a reference
The three-way switch eventually lands on the last setting, and then the old model is gone. That raises the question we care most about as testers: once the expensive model is off, what checks the cheap one?
A post published the week before by Hightouch suggests one answer. Working on identity resolution, the task of deciding whether a set of records belongs to one person, the team ran a decision model on every case and sent only the ones whose confidence fell inside a band to a frontier model. In their setup, a 30 to 90 percent band sent about 22 percent of cases to the larger model. That is the dual-run mode made permanent, but only on the slice where the small model says it is unsure.
It is a good design, and the same write-up contains the reason it is not enough on its own. For one of the models they tested, errors clustered near 50 percent confidence, which is exactly where a band catches them. For another, the authors describe it as “quite overconfident in the 90-100% bucket.” A confidence band can only escalate the mistakes a model knows it might be making. A confident wrong answer sails over the top of the band and never meets the reference.
So the residual check we would build after any cutover has two parts. The band, which handles the cases the model flags. And a small random sample of high-confidence cases sent to the reference anyway, because the band is blind exactly where an overconfident model is wrong. The sample is not there to improve accuracy on the day. It is there so that a shift in how confident and how wrong the model is shows up in a number someone watches, rather than in a complaint weeks later.
Both posts are careful about scope, and the Amazon one says plainly why the move worked for them: “a closed choice from a known catalog,” high volume, and latency on the old path that mattered. It also names where it would not: labels that need explanations, catalogs that change weekly, context that does not fit in a few thousand tokens. Those conditions are part of the migration plan too. A label set that changes every week means every week is a new migration, and the three-way switch stops being a one-time rollout tool and becomes a standing part of the system.
We have a fair number of decisions in our own pipelines that look like the Amazon case, multiple-choice questions currently answered by a model built to write paragraphs. The version of this we would actually run is the one the post describes, not the one the headline implies. Canonicalize the input first. Run both paths side by side long enough to read the disagreements. Keep a reference alive after the switch, on the uncertain band and on a sample of the certain cases. The new model may well be the right call, and the savings are real. What makes it safe to rely on is that someone kept checking after the cheaper model took over.