Your SSD Is the New VRAM

A weekend-sized project just made the "how much RAM do I need for local AI" question obsolete — or at least badly framed. Swiftlet, roughly 10,000 lines of Swift and Metal under Apache 2.0, runs Alibaba's Qwen3-Next-80B-A3B at 4-bit in 4.3GB of peak RAM on a Mac, decoding at 4.5–5 tokens per second on an M5. It also runs the newer Qwen3.6-35B-A3B on an iPhone 17 in about 2.5GB of RAM — the first time, as far as anyone in the Show HN thread could tell, that a 35B-class model has run natively on a phone.

The headline number is real, but it's doing sleight of hand. You still need 42GB of SSD for the 80B model (18GB for the 35B). What Swiftlet actually demonstrates is a shift worth internalizing: for the current generation of open models, inference is turning from a memory-capacity problem into a storage-bandwidth problem. Your SSD is becoming the new VRAM.

The trick is old — the models finally cooperate

Streaming weights from flash isn't new. Apple researchers proposed exactly this in the "LLM in a flash" paper back in December 2023, and llama.cpp has memory-mapped weights forever, quietly paging them off disk when RAM runs short. Community projects tried expert offloading against Mixtral in early 2024. It never mattered much, because the models fought back. Mixtral-8x7B routed every token to 2 of 8 large experts — roughly 13B active parameters per token. Stream that from disk and you're reading gigabytes per token. mmap-based paging under memory pressure was a thrashing lottery, not a strategy.

What changed is model architecture, on two fronts at once.

First, fine-grained sparsity. Qwen3-Next-80B has 512 small experts and routes each token to 10 of them; only about 3B of its 80B parameters activate per token. The dense core that every token touches — attention, routers, embeddings — is just 2.5GB at 4-bit (1.3GB for the 35B). Swiftlet keeps that resident, repacks the experts into fixed-stride .qpack containers so any expert is one pread() away, and holds a bounded cache with LFU-plus-recency eviction. Expert reuse across tokens turns out to be substantial: the cache hits 43–70% of the time. That's the difference between streaming as a stunt and streaming as an architecture.

Second, linear attention. Three-quarters of Qwen3-Next's layers use Gated DeltaNet with a fixed-size recurrent state instead of a growing KV cache. That's the half of the memory bill quantization never touched — the reason "32GB is enough" claims always collapsed at 32K context. With a constant-size state, the RAM footprint genuinely doesn't grow with context length.

Swiftlet credits TurboFieldfare, which pulled the same move on Gemma 4 26B in July, as design inspiration. Two independent projects landing on the same technique within weeks is usually the tell that the substrate shifted underneath everyone.

What you'd actually do with it

Swiftlet ships as a Swift library, a CLI (swiftlet chat, generate, and repack, which converts MLX-format checkpoints into its streaming container), and an OpenAI-compatible server — so existing tooling points at it with a base-URL change. It needs Apple Silicon on macOS 14+ or iOS 17+, and the author validated every layer of the forward pass against mlx-lm in both float32 and int4, which is more rigor than most inference side projects bother with.

Be honest about what 4.5–5 tokens per second buys you, though. Interactive chat at that speed is unpleasant, and prefill is the real killer: streamed experts mean prompt processing crawls too — commenters pegged a 10K-token prompt at around half an hour on an M5. That rules out the most tempting use case, pointing a coding agent with a repo-sized context at it. Where it fits is asynchronous work on machines that could never touch these models before: overnight batch summarization, classification queues, privacy-constrained document processing on a base-spec 8GB or 16GB Mac. Qwen3.6-35B-A3B scores 73.4% on SWE-bench Verified; getting that class of model on hardware Apple sells as entry-level, at 7–11 tokens per second, is a real capability, not a demo.

The iPhone number is a demo — 1 token per second isn't a product. But it's the existence proof that matters, and the SSD-wear panic that greets every disk-streaming project is mostly noise: reads don't meaningfully wear NAND, writes do, and inference is read-only. Sustained multi-GB/s reads do cost battery, which is the honest objection on a phone.

The uncomfortable part for the RAM-tier business model

My read: this is a genuine shift, not hype, with one asterisk — it's a capability-per-gigabyte breakthrough, not a speed one. Nothing here helps you if you need 40 tokens per second. But the constraint that defined local AI for three years — model size bounded by RAM — just became negotiable, and the trade you make instead (disk space plus patience) is one most developers can afford.

Expect the technique to stop being a standalone project quickly. llama.cpp already offloads MoE tensors to CPU RAM; SSD-tier streaming with cache-aware expert packing is the obvious next backend feature there and in MLX. The more interesting second-order effect lands on model labs: once expert streaming is a mainstream deployment target, expert-cache hit rate becomes a design metric. Routing locality — reusing the same experts across adjacent tokens — will get optimized the way KV-cache efficiency did in 2024–2025.

And it puts quiet pressure on Apple's own segmentation. The 8GB-to-128GB RAM ladder has been the local-AI upsell for two years. Swiftlet is an Apache-licensed argument that the ladder matters less than the NVMe controller — which, conveniently for Apple and less so for anyone who just bought 128GB, is fast on every machine they sell.