For a few years now, one of the stranger fixtures of model-release season has been a pelican riding a bicycle. Simon Willison asks every new model to draw one as an SVG, and the results get passed around as a folk signal of capability. Willison himself calls it a “deeply unscientific benchmark.” That has not stopped it from becoming load-bearing. And wherever an informal benchmark becomes load-bearing, the same suspicion follows: surely the labs have noticed, and surely they are training their models to ace it.
That suspicion has been repeated so often it had hardened into background knowledge. Then, on July 22, Dylan Castillo published “Are AI labs pelicanmaxxing?”, which did something almost nobody does with this kind of folklore. He treated it as an empirical claim and tested it. The post reached the front page of Hacker News within hours, at 537 points and over 200 comments, and Willison endorsed it as “excellent piece of work.” We want to talk about why this study matters to us, and it has nothing to do with pelicans.
The method is the story
The design is simple enough to describe in a paragraph, which is part of what makes it good. Castillo built a grid of 48 prompts: eight animals (pelican among them, alongside the likes of cats, herons, and raccoons) crossed with six vehicles (bicycle among them, alongside unicycles, skateboards, and boats). He ran the grid against seven frontier models, three samples per prompt, producing 1,008 SVG images. An LLM judge scored each rendered image on animal quality, vehicle quality, and how coherently the animal was actually riding the thing. A second model extracted features like which direction the subject faced.
Then came the part that separates this from vibes: a fixed-effects regression that adjusts for how hard each animal and each vehicle is to draw, with specific interaction terms for pelicans, bicycles, and the pelican-bicycle combination. If labs were special-casing the famous prompt, that specific cell should outperform what the difficulty of its parts predicts.
It does not. Pelicans ranked sixth out of eight animals; every lab draws cats, whales, and raccoons better. Bicycles ranked second-to-last among vehicles, narrowly ahead of planes. None of the pelican effects came close to significance (the smallest p-value was 0.25), and every confidence interval for the pelican-bicycle cell contained zero. Exactly one vehicle effect for one lab surfaced at p=0.022, which is what noise looks like when many comparisons run at once; correct for multiple testing and it disappears.
Our favorite part is the memorization probe, because it shows what intellectual honesty looks like mid-analysis. All 21 pelican-on-bicycle images from the top models faced right. That is exactly the kind of eerie uniformity that reads as a fingerprint of memorized training data, and a less careful author would have led with it. Castillo checked it against base rates instead. Bicycles face right in 81 percent of all images, pelicans in 78 percent, and three other animal-vehicle combinations also hit 90 percent or more directional agreement. The smoking gun was the dataset’s ordinary behavior. His conclusion is correspondingly modest: little evidence of pelican-specific optimization, with the honest caveat that labs could still be optimizing SVG drawing in general, which this design cannot detect.
Precise suspicions are testable ones
What struck us most is the epistemics. “They’re training on the benchmark” usually functions as an unfalsifiable mood. It costs nothing to say, it flatters the sayer as unfooled, and no release note can disprove it. Castillo’s move was to notice that the specific version of the claim makes a specific prediction: pelican-bicycle should beat its difficulty-adjusted expectation. That prediction can fail. It did fail. What survives is only the vague version of the claim, broad SVG-quality optimization, and the post says so plainly instead of quietly retreating to it.
There is a general lesson in that. A contamination suspicion stated precisely is a hypothesis. Stated vaguely, it is a permanent resident. The community around these models produces a lot of the second kind, about benchmarks, about arena rankings, about training cutoffs, and almost none of it ever gets promoted to the first kind. This took one person, a weekend’s worth of design, and about eighty dollars of API credit. Rigor of this sort is not expensive. It is just unusual.
What we are taking back to our own harness
We maintain our own evaluation suite for the work we do, and it contains more than one prompt that started life exactly the way Willison’s pelican did: as a private smoke test someone found revealing, promoted over time into something the team quietly treats as ground truth. This study reads to us like an audit checklist for those prompts, aimed at ourselves rather than at the labs.
First, a beloved prompt needs neighbors. A single smoke test cannot distinguish “the model handles our case well” from “the model got better at everything nearby.” The control grid, surrounding the case that matters with cases of comparable difficulty that nobody is suspected of optimizing for, is what turns a demo into a measurement. Most of our informal evals have no neighbors at all.
Second, difficulty adjustment is not optional. Pelicans turn out to be objectively hard to draw, which means years of mediocre pelicans said less about benchmark honesty than about anatomy. When one of our pet prompts improves across model generations, the boring explanation, that the prompt sits in an easy region and everything in that region improved, has to be excluded before the interesting ones get airtime.
Third, our own gotcha signals need base rates. The all-facing-right probe is the cautionary tale we intend to keep: a pattern in the output that looks like a memorization fingerprint may be a property of the whole distribution. We have caught ourselves reasoning from exactly that kind of signal, some odd consistency in an output format, taken as evidence the model has seen our test before. The check is cheap. Compute the base rate first.
And last, the multiple-comparisons discipline. Run 48 cells across seven models and something will clear p<0.05 by accident. Our harness produces far more than 48 numbers per run. If we went looking for anomalies without correcting for how many places we looked, we would find one every week, and each would come with a tempting story.
The pelican benchmark worked, in the end, not because it was rigorous but because it was legible enough that someone could eventually wrap rigor around it. That seems like the right fate for informal evals: not retirement, but the occasional afternoon where a suspicion about them is written down precisely enough to be wrong. We keep a list of things we believe about our own eval suite that have never been tested that way. After this week, the list feels less like trivia and more like a backlog.