On July 31, 2026, Simon Willison released smevals, a deliberately small evaluation suite built in collaboration with Jesse Vincent’s Prime Radiant lab. He calls it his third iteration on the idea of evals, and says it “feels right” to him. We read it the way we read most small tools that get the design right: not for the feature list, but for the two decisions underneath it. Both of those decisions describe how our own behavior changes day to day better than any public benchmark table does.
The first decision is that running and grading are separate operations. The second is that the thing being evaluated is a config, not a model. Neither is flashy. Together they describe a way of measuring quality that a team shipping prompt-and-model changes can actually keep honest.
The run and the grade are different jobs
In smevals the two commands are distinct. uvx smevals run executes the tasks against your configured models and stores the outputs. uvx smevals grade applies your checks to those stored outputs afterward. A separate uvx smevals serve gives you a local web view, and uvx smevals build produces a static HTML report.
The split looks like a small ergonomic choice until you notice what it protects. Generation is the expensive, slow, non-deterministic part. It costs tokens, it takes wall-clock time, and it does not return the same thing twice. Grading, by contrast, is usually cheap and should be deterministic: count the lines, check for a substring, apply a rubric. When those two jobs are welded together, every time you want to refine a check you pay for a fresh round of generation, and you are refining the check against outputs you have never seen before.
Decoupling them fixes both problems at once. The run freezes a set of outputs. From then on you can iterate on the grader as many times as you want against exactly those outputs, for free, and the comparison stays reproducible because the thing under the check never moved. If a grader was too strict, you loosen it and re-grade. If it let something through, you tighten it and re-grade. The generation is a fixed artifact; the judgment is the part you keep editing.
This maps cleanly onto a failure mode we have hit ourselves. Early on we tended to write an eval as a single script that generated and judged in one pass. The result was that improving the judgment quietly changed the sample, and we could never tell whether a score moved because the model got better or because we happened to draw a different set of outputs that run. Freezing the run removes that ambiguity. It is the same instinct as recording a fixture in a test rather than hitting a live service every time: separate the thing you are measuring from the act of measuring it, so that only one of them changes at a time.
There is an honesty to it, too. Non-determinism does not go away because you ignore it. Storing the outputs and grading them separately is a way of admitting that generation is a sampled process, and building the tooling around that fact instead of pretending each run is authoritative.
An eval measures a config, not a model
The second decision is the one we keep coming back to. In smevals, a run compares configs you define. A config specifies a model, but it can also carry a system prompt, model parameters, and other scaffolding around the model. The leaderboard in the web view ranks those configs by score. It shows recent runs and grades, per-tag pass rates, the task prompts themselves, and the grader details with their thresholds. The worked example is a “write a haiku” task, scored on whether the output is exactly three non-empty lines.
The framing matters because a bare model is not the thing we ship. We ship a model with a system prompt we wrote, parameters we chose, and a layer of surrounding tooling that shapes what the model can see and do. Two configs built on the same underlying model can behave nothing alike. The version of a question that actually keeps us up is never “is this model good,” it is “did the prompt change I made this morning help or hurt, holding everything else fixed.” A public benchmark cannot answer that, because it evaluated a model we do not run in a configuration we did not build.
The haiku example is almost too modest, which is why it is a good teaching case. “Exactly three non-empty lines” is a property you can check with a few lines of code and no ambiguity. It is not a proxy for intelligence. It is a concrete behavior that either happens or does not, tied to a specific task. Most of what we genuinely need to know about a config is shaped like that: does it return valid JSON, does it stop when it should, does it stay under a length limit, does it call the right tool first. Small checkable properties, evaluated against configs we define, tell us more about a change we are about to make than any aggregate score computed by someone else on a task we do not run.
Ranking configs also quietly reframes what a leaderboard is for. The board here is not a contest between vendors. Every row on it is something we built and could change. When the top config is one of ours, the takeaway is not “this vendor won,” it is “this prompt and these parameters, on this task, passed more of our checks.” That is a result we can act on, because we own every variable that produced it.
Why we keep the small ones
It would be reasonable to ask why a team would maintain its own tiny eval suite when large public benchmarks exist and cost nothing to read. The answer is in both design decisions above. The evals that tell us anything are the ones scoped to tasks we care about, run against configurations we control, with graders we can refine cheaply against frozen outputs. A benchmark table optimizes for comparability across the whole field. We are not trying to be comparable across the field. We are trying to know whether a specific change to our own setup made a specific task better.
Small is not a limitation here, it is the point. A suite you can read in an afternoon is a suite whose graders you actually understand, whose tasks map to things you ship, and whose results you trust because you wrote every check in it. smevals is small enough to own, and being ownable is what makes it useful. An eval you inherited and cannot fully reason about is closer to a rumor than a measurement.
The part worth carrying forward is not the tool, which may well be replaced by a fourth iteration. It is the shape of the thinking. Freeze the expensive, noisy step so you can iterate on judgment for free. Measure the whole configuration, because the configuration is the thing you change. Keep it small enough that you can hold all of it in your head at once. Those are not properties of a particular suite. They are what it takes for an eval to still mean something the day after you write it.