Press the model, keep the oil.
Run real LLMs privately, at 4-bit, on your own GPU.
Pure Rust, no Python at runtime, nothing leaves your machine during inference. Trapetum presses the weights into a small 4-bit code and decodes it straight inside the matmul, so the full weight matrix is never materialized. (The name is the Roman olive press: crush the fruit, keep the essence.)
$ ./trapetum llama-2-7b.cbk # pure Rust, no Python loaded llama-2-7b.cbk (3.5 GB) in 3.1 s prompt: "The capital of France is" logits vs HuggingFace (worst) 7.9e-3 top-1 agreement with HF 6 / 6 greedy continuation = HF 16 / 16 tokens OK decode throughput 135 tok/s energy 2.58 J/token (2.1x less than fp16)
curl -fsSL get.neuralboot.com | sh
Linux. Windows and details
Real Llama-2-7B, 4-bit, in pure Rust. No Python at runtime.
What do these numbers mean?
- 16/16 tokens match HuggingFace: given the same prompt, the 4-bit model emits the exact same next 16 tokens (greedy) as the original fp16 model. Proof the quantization, the kernel and the Rust pipeline are correct, not approximate (worst-case logit gap 7.9e-3).
- 135 tok/s on one RTX 4090: generation speed, one token at a time (batch 1), on a single consumer GPU. How fast it “types” the answer.
- 3.5 GB on disk (4.7 GB VRAM), vs ~13.5 GB fp16: the weights shrink from ~13.5 GB (16-bit) to 3.5 GB (4-bit), so a 7B model fits on a small or consumer GPU instead of needing a datacenter card.
- 2.1× less energy per token: each generated token costs 2.1× less energy than fp16 (2.58 vs 5.45 joules per token, measured from GPU power draw). Cheaper and greener to run.
NVIDIA CUDA
Runs on any modern NVIDIA GPU, not just the 4090
The RTX 4090 above is only our reference card. Trapetum is CUDA and runs on any
Ampere, Ada or Hopper GPU. Benchmarked on RTX 4090, A40 and H100; other Ampere cards such as
the A100 are supported but were benchmarked on the A40. Recompile with one
-arch flag, or ship a single multi-arch binary. The 4-bit memory win is
universal (a 7B in ~3.5 GB fits an 8 GB card); the decode speedup follows the
bandwidth law, largest where memory bandwidth is scarcest (the A40, then the RTX 4090), parity on an H100.
See the per-GPU table.
AMD + APPLE · COMING SOON
AMD and Apple GPUs are on the way
The fused 4-bit decode is being ported beyond NVIDIA. An AMD backend (ROCm / HIP) and an Apple backend (Metal, for M-series GPUs) are in active development: the pure-Rust runtime and the codebook format are already portable, so the remaining work is the kernel itself. The 4-bit memory win carries over unchanged; the decode speedup will again follow the bandwidth law. Want one of these first? Tell us your GPU.
It reproduces HuggingFace, token for token
Loaded from a 3.5 GB .cbk file, the runtime decodes the same quantized weights as
HuggingFace and reproduces its greedy generation exactly, with no Python in the loop. The terminal
output above is a real run, not a mockup: worst-case logit gap 7.9e-3, greedy continuation identical
for 16 of 16 tokens. Full model-level results.
An honest Pareto, one machine
Speed, memory, accuracy and energy for the same Llama-2-7B on an RTX 4090, batch-1 decode, iso-context. The pure-Rust runtime is the fastest and the lowest-energy path.
Honest limits
- It does not beat dense fp16 on accuracy. Nothing does; quantization trades a little perplexity for a lot of memory.
- The kernel speedup is a memory-bandwidth law: largest where memory bandwidth is scarcest, near parity on an H100 where fp16 is already near roofline.
- Batch-1 decode for now. Batched throughput is an open problem, marked as such.
Everything is public, measured on real hardware, and reproducible with one command, including the negative results.