The most-traveled code path in the LLM ecosystem got roughly a thousand times faster this week, and not because anyone discovered a new algorithm. GigaToken, an MIT-licensed tokenization library released by Marcel Røed, benchmarks GPT-2 tokenization at 24.53 GB/s against 24.8 MB/s for the widely used HuggingFace tokenizers on a 144-core AMD EPYC. That is a measured 989x. It ships drop-in compatibility layers that mimic the HuggingFace and Tiktoken interfaces, covers most common BPE tokenizers, and went straight to the front page of Hacker News.
The BPE math is unchanged. The gains come from what the author plainly describes as engineering: replacing the general-purpose regex engine that normally handles pretokenization with SIMD-optimized code, minimizing branching, caching pretoken mappings to exploit their long-tailed distribution, and cutting the overhead of crossing the Python to Rust boundary. Every one of those techniques has been well understood for a decade or more. What is new is that someone applied them, all at once, to a layer everyone depends on and almost nobody measures.
Where a thousandfold actually comes from
The headline number deserves a closer look, because it teaches something beyond “new library fast.”
The speedup is not constant across hardware. On the 144-core EPYC it is 989x. On an Apple M4 Max it is about 1,268x. On a 16-core desktop Ryzen it is 106x. The new implementation is roughly consistent per core; what varies is the baseline, which slows down dramatically as core counts rise. A benchmark ratio always measures two things at once, the code being celebrated and the code being replaced, and here the extreme end of the ratio says as much about how badly the incumbent scales under contention as about how well the challenger vectorizes. One hundred times faster on a normal machine is the honest floor. It is still enormous.
The second lesson is in the list of techniques, because none of them touch the algorithm. Pretokenization is usually outsourced to a regex engine, which is general-purpose machinery solving a narrow, fixed problem. Specializing it and vectorizing it is exactly the move Mitchell Hashimoto described the same week in “Everyone should know SIMD,” which was on the front page beside GigaToken: broadcast constants into lanes, loop over chunks, operate in parallel, reduce, handle the tail. His worked example, scanning codepoints in a terminal emulator, yields about 5x from what is essentially a for loop with wider lanes. It is not a coincidence that both posts found an audience in the same week. The easy wins in this ecosystem have moved down the stack, and low-level performance literacy is suddenly worth more than it was.
The third lesson is the boundary. Part of GigaToken’s gain comes from reducing Python to Rust interop overhead, which is a polite way of saying that the previous hot path crossed a language boundary far more often than the work required. We have internalized a shorthand that “the core is in Rust, so it is fast.” Language choice is not a performance property. The boundary between a scripting language and a systems language is itself a cost center, and when the unit of work per crossing is small, the crossing dominates the work.
Fast enough until the loop changed
Tokenization earned its invisibility honestly. For years it was genuinely fast enough. Encoding a prompt took microseconds against a model forward pass measured in seconds, so no profiler ever flagged it, so nobody looked.
Then the loops around it changed. Pretraining corpora grew to the point where tokenization is not a rounding error but a phase, days of wall-clock time across fleets of CPUs before training can begin. Context windows grew from four thousand tokens to hundreds of thousands. Agent systems started serializing entire tool transcripts through the encoder many times per session. The layer stayed the same speed while everything flowing through it grew by orders of magnitude.
The Hacker News discussion split in a revealing way. Skeptics pointed out that tokenization is a fraction of a percent of total inference compute, which is true. The author’s own numbers show around 8 to 10 percent reduction in time-to-first-token on an 8B model at long input lengths, which is also true. Both sides are right because the cost of a layer is not a fixed number, it is a function of which loop the layer sits inside. In a training pipeline it is throughput and the unit is CPU-days. In interactive inference it is latency on the critical path before the first token appears. In a batch analytics job it may genuinely be nothing. Arguing about whether tokenization is expensive without naming the loop is arguing about nothing.
We recognize this pattern from inside our own harness. What appears on our dashboards as “model latency” is really a sum: prompt assembly, serialization of tool results, transport, encoding, and then, finally, the model. The model gets the blame because the model has a dashboard. The glue between the pieces reports nothing, so it costs nothing, officially. When we have actually traced a slow agent turn end to end, the model’s share was consistently smaller than the attribution suggested. Instrumentation determines blame, and blame determines where optimization effort goes. That is how a thousandfold sits undisturbed in plain sight.
Systems humility, applied
The uncomfortable part of this story is the date. This is 2026. Tokenization is arguably the single most-executed code path in the entire ecosystem, running in every training pipeline, every inference server, every evaluation harness, every cost estimator. The library it just lapped is excellent, widely used, and written by capable people. And still, three orders of magnitude were available to anyone who profiled the layer instead of assuming it was done.
Nobody was incapable of this work. Nobody was looking, because the layer had “already optimized” stamped on it by reputation. That should make us suspicious of every component we treat the same way: the JSON serializer, the schema validator, the diff engine, the transport framing under our tool calls. Each is mature, each is written in a fast language, and each has been fast enough for so long that it has aged out of every profile we run.
The practical conclusion we are taking is narrow and actionable. Profile by layer, not by symptom, and deliberately include the layers with good reputations, because reputation is exactly what keeps them out of the flame graph. Treat any component that predates a few orders of magnitude of load growth as unproven at the new scale, whatever its pedigree. And when a speedup claim arrives with a spectacular ratio, ask what the baseline was doing, because the answer is often the more useful fact.
The models will keep getting faster on their own schedule, driven by people with far more resources than us. The glue is ours. If a thousandfold was hiding in the most-traveled path in the ecosystem, we are not inclined to assume our own least-traveled ones are clean.