Pure-Rust LLM runtime

Big models,
run small.

Trapetum presses a model's weights to a quarter of their size and reads them straight in that compressed form, so a big model runs faster, on far less memory, using far less power. Same answers, token for token.

See it live: same model, compressed vs not, racing on two identical RTX 4090s.

In one sentence

When a model writes one word at a time, the machine spends almost all its effort just reading the model's weights out of memory, not doing math. Trapetum shrinks those weights to a quarter of their size and reads them compressed, so every word comes out faster, on far less memory, using far less power. The answers stay the same.

What it does

Four measured wins from one move: read fewer bytes.

2.34×

faster decode, fused 4-bit kernel

3.9×

smaller on disk: 3.5 GB vs 13.5 GB fp16

~47%

less energy per token

1.32×

lossless speculative decode, measured wall-clock

One move causes all four: the fused kernel reads a quarter of the weight bytes straight out of the codebook.

End to end that is 2.0× (7B) to 2.45× (13B) faster per token. Weights drop from 13.5 GB to 3.5 GB on disk for a 7B model (4.7 GB peak VRAM at runtime); a 2-bit 70B fits a single 24 GB consumer card and stays sharper than an fp16 7B. The speculative-decoding pass on top (measured 1.32× wall-clock, 164 vs 125 tok/s) is token-for-token identical to plain decoding, for about 2% extra memory.

The bandwidth law

The kernel speedup follows GPU memory bandwidth: largest exactly where bandwidth is scarcest.

GPU	Bandwidth class	Speedup vs cuBLAS fp16
RTX 4090	~1.0 TB/s, consumer	2.20×
A40	~0.7 TB/s, prosumer	2.34×
H100	~3.3 TB/s, datacenter	0.99× (parity)

The 4-bit memory win is universal; the decode speedup is largest on bandwidth-limited consumer and prosumer cards, and ties on a bandwidth-rich H100 where fp16 is already near roofline. Full per-GPU table.

Runs real models, in pure Rust

No Python at inference. Eight architectures compressed to 4-bit and decoded coherently by the same runtime, including the hard modern ones.

Llama-2-7B Llama-3.1-8B Mistral-7B Qwen2.5-7B DeepSeek-R1-Distill-7B Gemma-2-9B Phi-4 · 14B DeepSeek-V2-Lite · MLA + MoE

Export is torch-free too: weights compress straight from safetensors, no PyTorch or transformers dependency. See which of these models fit your GPU.

What it unlocks

The point is not a faster benchmark. It is what becomes possible when a frontier model fits on hardware you already own.

Frontier models on your own hardware

70B on a single 24 GB consumer card. The 671B DeepSeek-V3 on one big box, by streaming experts from memory on demand.

Private by default

Everything runs locally and offline. No cloud, no account, no data leaving the machine.

Cheaper, greener serving

Roughly half the energy per token. Lower cost per request and a smaller footprint at scale.

New architectures, quickly

Mixture-of-Experts, Multi-head Latent Attention and logit softcaps are already ported and running. The kernel generalizes.

The fine print. Trapetum optimizes decode at batch 1, the local single-user case. Speed figures are versus cuBLAS fp16 on the same GPU; prefill is a memory win, not a speed win. The speculative-decoding wall-clock is now measured: 1.32x lossless on Llama-2 (vs the 1.50x projected ceiling); large-vocab pairs sit at parity pending device-side argmax. The full negative-to-fixed arc is published in the speculative-decoding benchmarks. Measured on RTX 4090, A40 and H100 with a fixed seed. Code and raw numbers are public.

Convinced by the numbers?

Every figure above reproduces with one command. The next step takes one line.

Run it See the numbers Read the paper