AirLLM's 4GB 70B Trick Is Real, and Beside the Point

AirLLM hit the Hacker News front page again this weekend with the same headline that's been stopping scrollers since late 2023: run a 70B model on a single 4GB GPU, no quantization required. The claim is real. It's also one of the best examples in local AI of a technically true statement that answers the wrong question — and the interesting part in 2026 isn't the 70B trick at all. It's what the same idea does to sparse mixture-of-experts models.

The trick is bandwidth, not magic

A transformer only ever computes one layer at a time. A Llama-style 70B model is roughly 80 decoder layers of about 1.7GB each at fp16, so if you load layer 1, run it, evict it, load layer 2, and repeat, your peak VRAM is a couple of layers plus activations — comfortably under 4GB. AirLLM shards the model into per-layer files on disk, streams them through the GPU in sequence, and wraps the whole thing in a from_pretrained() one-liner:

from airllm import AutoModel

model = AutoModel.from_pretrained("meta-llama/Meta-Llama-3-70B")

None of this is new physics. FlexGen formalized offloading-based inference in early 2023, DeepSpeed's ZeRO-Inference did disk offload before that, and Hugging Face Accelerate has shipped device_map disk offload for years. What Gavin Li's project got right was packaging: one pip install, automatic layer splitting, per-layer flash attention, and support for basically every open model family — Llama, Qwen, Mistral, Phi, Gemma, DeepSeek. That's worth something. Twenty-six thousand GitHub stars say the ergonomics mattered more than the novelty.

Now do the arithmetic

Here's the part the headline omits. Generating one token requires a full forward pass, which means every one of those 140GB of fp16 weights has to transit to the GPU — per token. Your 4GB card can't cache anything meaningful between passes, so throughput is bounded by transfer bandwidth, not compute.

Run the floors: a top-end NVMe drive sustains about 7GB/s sequential, so streaming weights from disk costs at least 20 seconds per token before the GPU does any math. If you have enough system RAM to cache the whole model and feed it over PCIe 4.0 x16, you're still looking at roughly five seconds per token as a hard floor. AirLLM's 4-bit block-wise compression cuts the traffic by three-quarters — the project claims about a 3x speedup, which is consistent with the bandwidth math — but you're still in seconds-per-token territory. Commenters in the HN thread reported everything from multi-second to multi-minute token times depending on hardware, and the thread's sharpest criticism was exactly this: the memory problem gets solved by converting it into a worse latency problem.

Compare the alternative every practitioner actually uses. A Q4 GGUF of a 70B model under llama.cpp fits in about 40GB and runs at conversational speed on a 64GB Mac or a CPU box with enough RAM. Yes, quantization costs some quality. But "slightly degraded and usable" beats "bit-exact and 20 seconds per token" for every interactive workload that exists.

Where it actually earns its keep

That doesn't make AirLLM hype. It makes it a narrow tool that keeps getting marketed as a general one. The honest use cases:

• Full-precision ground truth. If you're evaluating how much a quantized deployment degrades, you need fp16 baseline outputs from the same weights. AirLLM produces those on hardware you already own, overnight, for free.
• Offline batch work. Scoring a few hundred examples with a big model where a 12-hour wall clock is fine and an API bill or a GPU rental isn't.
• Touching weights you couldn't otherwise load. Poking at a 405B base model's raw behavior on a workstation is genuinely not possible any other way at ~8GB VRAM.

Practical caveats before you pip install airllm: the first run decomposes the checkpoint into layer shards on disk, so budget roughly double the model's footprint in free space — for a 70B fp16 model that's ~280GB total. The bundled examples use MAX_LENGTH = 128, and long contexts make everything slower because prefill also streams the full stack. This is a batch tool; treat it like one.

The MoE plot twist

The 2026 releases are where the approach stops being a party trick. AirLLM v3 added FP8 support and, more importantly, expert-level streaming for sparse models: for a mixture-of-experts architecture it loads only the experts each token actually routes to. The repo now claims DeepSeek-V3's 671B in about 12GB of VRAM and — as of July — Moonshot AI's newly open-sourced 2.8-trillion-parameter Kimi K3 in under 4GB.

That changes the bandwidth equation fundamentally. K3 activates 16 of 896 experts per token, so per-token weight traffic is a small fraction of total parameters rather than all of them. Streaming a dense 70B moves 140GB per token; streaming a sparse model moves something proportional to its active parameters. Sparsity is exactly the property that makes offloading stop being absurd.

Two caveats before you get excited. Expert routing differs per layer and per token, so those reads are scattered, not sequential — NVMe random-read throughput, not the spec-sheet sequential number, is your real ceiling. And the disk footprint is brutal: K3's weights run to multiple terabytes, so the constraint just migrates from VRAM to your SSD budget. I'd also note the trillion-scale VRAM figures currently come from the project's own changelog; independent benchmarks haven't caught up yet.

Still, the direction is right. Open frontier models are converging on extreme sparsity — 671B, 1T, 2.8T totals with tens of billions active — and layer-plus-expert streaming is a legitimately plausible path to running them on enthusiast hardware for non-interactive work. My verdict: the "70B on 4GB" framing that keeps AirLLM on the front page is the least interesting thing about it, and for interactive use you should keep running quantized models under llama.cpp or Ollama. But as sparse models eat the frontier, the streaming crowd is quietly building the only bridge between trillion-parameter open weights and the hardware most of us actually own. Worth watching — just not waiting 20 seconds a token for.