671B at 2.46 tok/s, on one machine
DeepSeek-R1 671B, the largest open-source model, decoding on a single commodity node in the pure-Rust Trapetum runtime, via the CPU-experts inversion. Replayed token by token at its measured steady-state speed of 407 ms/token.
The full run takes about 24 seconds at real speed, exactly as it decoded on the node.
Why it is this fast now: the inversion
The original run streamed the Mixture-of-Experts weights from disk, paging in the experts each token routed to. Around 10 GB crossed the disk every token, so decode was disk-bound at 0.24 tok/s: reach, not throughput. The inversion turns that around. The 350 GB of 4-bit routed experts stay resident in host RAM, and the CPU decodes them in place (one vpshufb per 32 weights on AVX2). The GPU no longer touches the experts at all; it holds only MLA attention and the dense layers, 20.9 GB of VRAM. At batch one only the ~10 GB of routed-expert bytes per token matter, and they never travel to the GPU.
Decode is now bound by host memory bandwidth, not disk and not the GPU: routed decode reaches ~67 GB/s aggregate, near the memory read floor. The gain came from removing eight measured walls in sequence, from 0.24 up to 2.46 tok/s, a x10.2 speed-up on the same box.
| Configuration | tok/s | Status |
|---|---|---|
| Experts streamed from disk (original reach demo) | 0.24 | measured, disk-bound |
| Experts resident in RAM, decoded in place on the CPU | 1.84 | measured |
| + MLA attention on the GPU, overlapped with expert decode | 2.46 | measured (this run) |
Throughput now scales with host memory bandwidth, the same bandwidth law measured across this project, moved from disk to RAM. See benchmarks for the full eight-wall chain and the quality (perplexity) numbers, try the live side-by-side demo on a model that answers in real time, or follow the Road to x100, the measured program to make this 671B interactive.