Chasing x100
Can DeepSeek-R1 671B be made interactive on hardware you can own? This is the engineering log of that chase: every number here was measured, including the ideas the data killed and the wall we did not expect. It started at 0.24 tok/s. It now runs the full 671B at 2.46 tok/s on a single commodity node, a x10.2 gain, once we stopped fighting the wrong wall. The answer was not to shrink the model until it fits, but to stop moving it: keep the experts in RAM and bring the computation to them. Here is exactly how far it goes, and where the real wall turned out to be.
The theory: shrink the model until it fits in fast memory
Each token must touch the 8 routed experts of 58 MoE layers: about 10 GB of weights. From disk (2-7 GB/s) that is roughly 4 seconds per token. From RAM or VRAM (100-1000 GB/s) it should be tens of milliseconds. So the plan was: shrink the experts until the model fits in fast memory. That theory turned out to be only half the story, as the measurements below show.
Both smaller formats fit in memory a single box can hold. 152 GB fits 192 GB of DDR5 or a Mac Studio's unified memory. We built them, measured their quality, and put the 671B fully in RAM to test the theory. The result was a surprise.
What the data killed, and what survived
Measured: adjacent tokens share only 28% of their experts (routing log committed to the repo), capping the gain at x1.16-1.37. Not worth the machinery.
A/B/A measured: no effect, the kernel readahead is ignored by the network mount. Still open on local NVMe.
Experts to 3-bit: PPL 6.82 to 7.18, +5.2%, near lossless, 228 GB. Experts to 2-bit: +21% (beam-4), aggressive. So there is a real quality knob: 3-bit is the sweet spot, 2-bit trades quality for size. 4-bit stays the reference.
New CBKA format, CUDA kernel validated on a 4090 at 4.5e-4 max error against a bit-faithful reference, end-to-end selftest exports decode in the pure-Rust runtime. Old 4-bit artifacts stay loadable, byte-identical.
HuggingFace throttles the download of that repo to a stall after ~250 GB, across three hosts and three protocols. The 2-bit/3-bit code is ready; only the weight transfer blocks a fresh export. Waiting on a faster source.
On AWS (384 GB RAM), the whole 4-bit model was loaded into memory, zero disk paging. Speed went from 0.24 (disk) to just 0.33 tok/s, a mere 1.4x. Fitting in RAM was not the answer: on a single GPU the wall is streaming 10 GB of experts across PCIe to the card every token, not reading them from disk.
See the quality trade with your own eyes
The measured curve, experts only, everything else untouched (V2-Lite proxy, wikitext-2): 4-bit is the reference, 3-bit +5.2% (near lossless, 228 GB), 2-bit +21% (152 GB). Below are real greedy continuations of "The capital of France is" at the two extremes so you can feel what 2-bit costs:
Note the honest wart: the bf16 base model loops on greedy decode here (small base models do that); the 2-bit one happens to ramble more freely. Perplexity, not vibes, is the metric: +19.2% is a real cost, and whether it is acceptable will be judged on the full 671B side by side. Raw results in the repo.
The kernel is ready
| Validation (RTX 4090, this week) | Result |
|---|---|
| 2-bit additive GEMV vs bit-faithful CPU reference | max err 4.5e-4, PASS |
| 3-bit additive GEMV vs bit-faithful CPU reference | max err 4.7e-4, PASS |
| Format roundtrip (encoder vs Rust decoder) | exact, 0.0 |
| End-to-end: export, load, decode in pure Rust | 4-bit, 2-bit, 3-bit all decode |
| Existing 350 GB 4-bit artifact compatibility | byte-identical loading |
The kernels and formats are done. What the measurements settled is where they matter.
Where the wall actually fell: inversion
The RAM surprise named the real wall: on a single GPU the bottleneck is moving 10 GB of experts across PCIe onto the card every token, not reading them from disk. Fitting the model in fast memory does not help if you still shuttle it to one GPU. So we stopped shuttling. Inversion: keep the routed experts resident in host RAM and decode them in place on the CPU, with the same fused 4-bit codebook kernel ported to AVX2 (one vpshufb per 32 weights). The GPU holds only what it is good at, the MLA attention and the dense layers (20.9 GB VRAM). At batch one the ~10 GB of routed-expert bytes never travel; only the 14 KB activation moves.
The result, measured on a single commodity node (64 vCPU, one L40S, 497 GB RAM): the full 671B decodes at 2.46 tok/s, a x10.2 gain over the 0.24 tok/s baseline. It got there by removing eight successive walls, each measured, not one clever trick: 0.24 → 0.44 → 0.96 → 1.31 → 1.35 → 1.67 → 1.84 → 2.46 tok/s as an x86 decode kernel, a register-resident tile transpose, GPU-side attention absorption, and a warmed codebook cache each fell. The routed decode itself now reaches ~67 GB/s aggregate, near the memory read floor, so from here throughput scales with host memory bandwidth: an EPYC or a Mac Studio would go higher (projected, not yet measured).
| Quality (wikitext-2 perplexity, full model) | PPL | vs fp16 |
|---|---|---|
| fp16 (reference) | 5.70 | baseline |
| our 4-bit (all K=16) | 6.10 | +7.1% |
| mixed precision (shared experts + head at 8-bit) | 5.78 | +1.5%, +0.7% size |
| full 8-bit (all K=256) | 5.70 | +0.03% |
Honest quality, four measured numbers: full 8-bit is essentially lossless (+0.03%), and the mixed-precision format captures most of that recovery, from +7.1% down to +1.5%, by lifting only 0.7% of the tensors (the shared experts and output head, which every token passes through) to 8-bit. "Lossless 4-bit" means identical to the 4-bit artifact; the 4-bit itself carries that +7.1%, which the mixed format nearly closes for a negligible size cost. The x100 chase did not reach x100 on one box; it reached a measured x10.2, by inverting the data path rather than shrinking the model, and it now scales with memory bandwidth. All raw numbers in the repo.