Your SSD Is the New VRAM for Local LLMs

A weekend Show HN made a claim that sounds like a typo: Swiftlet, a Swift + Metal runtime, runs Qwen3-Next-80B in 4.3 GB of peak RAM on a Mac, and a 35B Qwen on an iPhone 17 in about 2.5 GB. The numbers are real, but the headline buries the actual story. Nothing got compressed 10x. What changed is which hardware resource gates local inference — and for a specific class of models, it's no longer RAM. It's your SSD.

The models did half the work

Swiftlet only runs one family: the ultra-sparse mixture-of-experts hybrids Alibaba started shipping with Qwen3-Next-80B-A3B in late 2025 and continued with Qwen3.6-35B-A3B this spring. The "A3B" is the load-bearing part: of 80 billion parameters, only about 3 billion fire per token. The 80B routes each token to 10 of 512 experts; the 35B to 8 of 256, plus one shared expert. Three-quarters of the layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so the KV cache doesn't balloon with context either.

That architecture creates a memory profile older models never had: a small dense core (attention, routers, embeddings, shared expert) that every token touches, and a huge cold mass of expert weights where any single token touches under 4% of it. Dense-model tricks — llama.cpp-style mmap, layer-by-layer streaming à la AirLLM — turn into page-cache thrashing or single-digit tokens per minute, because every token needs every weight. Sparse routing is what makes leaving 90%+ of the model on disk survivable.

Explicit reads beat mmap, and it's not close

Swiftlet keeps the dense core resident — about 1.3 GB for the 35B, 2.5 GB for the 80B at 4-bit — and repacks the routed experts into fixed-stride .qpack containers so every expert fetch is exactly one pread from SSD into a Metal-visible buffer. Hot experts live in a bounded LFU pool with recency weighting; the author measured 43–70% hit rates. No mmap, no page cache, no kernel deciding what to evict.

That design choice has independent support. TurboFieldfare — the Gemma-focused project Swiftlet credits for the streaming design — published measurements in July showing roughly 0.5 tok/s with mmap versus ~4 tok/s with explicit pread on a cold 8 GB Mac. The kernel's page cache is optimized for workloads that aren't "randomly fault in 100 MB of weights per token while the GPU waits." Bypassing it is the whole ballgame.

The resulting numbers, from the author's README on an M5 Mac: 7–11 tok/s for the 35B (18 GB on disk, 2.6 GB peak RAM), 4.5–5 tok/s for the 80B (42 GB on disk, 4.3 GB peak). These are single-source figures — nobody's independently reproduced them yet — but they pass a sanity check, which matters more than it usually does.

Here's the check. Three billion active parameters at 4 bits is roughly 1.5 GB of weights per token; subtract the resident dense core and a 43–70% cache hit rate and you're pulling somewhere around half a gigabyte to a gigabyte off SSD per token. Apple's internal NVMe sustains several GB/s on large sequential reads — which fixed-stride packing gives you. That lands you at mid-single-digit tokens per second. The claimed 4.5–5 tok/s isn't suspiciously fast; it's about what the disk can feed. This is now a storage-bandwidth-bound workload, and that's falsifiable arithmetic, not vibes.

It also defuses the HN thread's loudest worry: drive wear. NAND flash wears out from program/erase cycles — writes. Inference streaming is pure read traffic. Read disturb is real but managed by the controller, and it's marginal next to what a normal Spotlight reindex does in writes. The legitimate costs are elsewhere: 42 GB of disk, sustained I/O power draw, and thermals on a phone.

The lineage, and the "first" asterisk

None of the ingredients is new. Apple's own LLM in a flash paper laid out flash-offloaded inference in December 2023; Eliseev and Mazur's Mixtral-offloading work used LRU expert caching the same month; PowerInfer-2 ran a 47B sparse Mixtral variant on an Android phone back in 2024 — so Swiftlet's "first time a model of this class has run natively on a phone" needs qualification. But PowerInfer-2 needed a 24 GB flagship. Doing it in 2.5 GB, on a stock iPhone, under iOS's per-app memory ceiling — where an 18 GB resident model isn't slow, it's jetsam-killed before the first token — is the part with no real precedent.

And Swiftlet isn't alone. TurboFieldfare for Gemma, SharpAI's SwiftLM, TurboQuant-MLX pushing a 122B through a 16 GB Mac mini — a recognizable pattern has formed in 2026: A3B-class open models plus fast NVMe plus explicit expert streaming. When three independent projects converge on pread-into-LFU-pool within months, that's an architecture finding its runtime, not one person's hack.

Should you actually use it

If you own a 48–64 GB Mac, no — MLX with the whole 4-bit model resident will thrash this on throughput. Swiftlet's audience is everyone else: the 8–16 GB MacBook Air majority, and iOS developers who until now couldn't ship anything smarter than a 3–4B dense model inside an app. The workflow is a git clone and swift build -c release, a swiftlet-repack pull from Hugging Face, then either the CLI, the SwiftletCore Swift package, or an OpenAI-compatible local server. Apache 2.0, macOS 14+/iOS 17+.

Calibrate expectations before you clear 42 GB of disk. Decode at 4.5–5 tok/s is fine for chat and painful for agents; the README doesn't publish prefill numbers, and prompt processing over long contexts is exactly where disk-bound inference should hurt most — HN commenters flagged this and it's the open question I'd want answered first. One tok/s on the iPhone is a proof of concept, not a product. And the author's own caveat is the most honest line in the README: with 3B active parameters, these models "chat and write like large models but recall facts like small ones." You're getting a large model's fluency and instruction-following, not its knowledge.

Still: genuine shift, not hype. For two years the local-LLM ceiling was set by how much unified memory you paid Apple for. Ultra-sparse MoE quietly broke that pricing model, and Swiftlet is what the break looks like in practice — capability per gigabyte of RAM stopped being the constraint, and capability per gigabyte per second of SSD took its place. Storage bandwidth is about to start showing up in inference benchmarks, right next to memory bandwidth. Watch prefill numbers and independent repros; if those hold up, the 16 GB Air just became a legitimate LLM machine.