How a 20B Model Hits 120 tok/s on an iPhone

A 20-billion-parameter model generating 120 tokens per second on a phone would have sounded like a typo two years ago. That's the headline claim behind Maple-Preview, a new open-weights reasoning model from DeepGrove that hit the Hacker News front page this week. And unlike most "frontier model on a potato" demos, this one mostly survives scrutiny — because it's not one trick, it's two tricks that multiply.

Maple-Preview is a 20B-parameter mixture-of-experts model with roughly 1B parameters active per token (20B-A1B), 24 layers, 256 experts with 8 active, and a 131k context window. The weights are ternary — every weight is -1, 0, or +1 — and the whole checkpoint is 5.31 GB under an MIT license. DeepGrove claims 218 tok/s on a base M4 Mac mini and the 120 tok/s iPhone figure from the announcement, and says the model is 5–16× faster than efficient competitors like Gemma 4, Qwen3.5, and gpt-oss.

Why the number is plausible

On-device decoding is memory-bandwidth-bound: every generated token requires streaming the active weights through the memory bus. That's where the two tricks compound. MoE sparsity means only ~1B of the 20B parameters are touched per token. Ternary packing means those parameters cost about two bits each instead of sixteen. Multiply them and each decode step reads roughly 300 MB of weights — against the 50–120 GB/s of memory bandwidth in a modern iPhone or base M4. That puts the theoretical ceiling in the low hundreds of tokens per second, which makes both the 218 tok/s Mac number and the 120 tok/s phone number physically consistent rather than magical.

The checkpoint size passes the same smell test. A 20B model shipped in BF16 would be ~40 GB; 5.31 GB is exactly what you'd expect from genuinely packed ternary MoE weights plus higher-precision attention and embeddings. This isn't a "ternary" model that secretly ships full-precision tensors.

The architecture shows the same discipline on the other bandwidth hog, the KV cache. Maple interleaves sliding-window attention (512-token window) with global attention at a 3:1 ratio, Gemma-style, so three-quarters of the layers have a KV cache that stops growing after 512 tokens. Without that, a 131k context on a phone would be dead on arrival regardless of how small the weights are.

Native ternary finally cashes a real check

The interesting part isn't quantization — it's when the quantization happens. Microsoft's BitNet line argued back in 2024 that ternary models should be trained that way from scratch, and proved it at 2B parameters with b1.58-2B4T. DeepGrove's own earlier Bonsai was a 500M-parameter native-ternary proof of concept trained on under 5B tokens. What nobody had shipped was native ternary at a scale where the model is actually useful. Maple-Preview, if its claims hold, is that model — and bolting ternary onto a fine-grained MoE (the DeepSeek-style many-small-experts recipe) is the move that gets capacity back without paying for it at decode time.

The contrast with post-hoc compression is instructive. PrismML's Ternary-Bonsai-27B — confusingly named, but an unrelated post-training ternary quantization of Qwen3.6-27B — manages about 11 tok/s on an iPhone 17 Pro, and independent analysis found its tool-calling accuracy collapsed from 80 to 66 at 1-bit while math barely degraded. Squeezing a trained full-precision model into two bits destroys exactly the fine-grained discrimination that post-training taught it. Training in ternary from day one sidesteps that failure mode, at least in principle. The gap between 11 tok/s and a claimed 120 tok/s for models in the same weight class is the strongest single argument that native-ternary-plus-MoE is a different category, not a better compressor.

Where the claims get soft

Now the caveats, because there are real ones. The benchmark story — Pareto-frontier results on LiveCodeBench v6, AIME 2026, HMMT 2026, and GPQA Diamond — is published as an image on the model card with no eval harness, and HN commenters noted the comparisons target Qwen3.5 rather than the newer 3.6. The iPhone figure comes from the team's own announcement, doesn't specify which iPhone, and hasn't been independently reproduced yet. Coverage so far, including Latent.Space's AINews, relays DeepGrove's numbers rather than confirming them.

The capability critiques are more predictable but worth taking seriously. Early users caught it confidently inventing etymology and hallucinating a nonexistent Seattle observation deck. Of course it did: 5 GB of weights can only memorize so much world knowledge, whatever the reasoning benchmarks say. And DeepGrove is upfront that this preview received minimal agentic post-training and only small-scale RL. This is a reasoning engine, not an encyclopedia and not yet an agent.

What to actually do with it

If you're building on-device or local-first, the adoption path is concrete. The checkpoint runs today through Transformers, vLLM, and SGLang, and there's already a community WebGPU demo — a weekend on an M-series Mac tells you whether the reasoning quality is real for your workload. Treat the model as a compute core, not a knowledge store: pair it with retrieval for facts and route tool-calling to something proven until the non-preview release ships with real agentic training. Phone deployment is the frontier, not the default — 5.31 GB of weights plus KV cache is a tight fit inside iOS app memory limits, and the mobile inference stack that produced the 120 tok/s demo isn't in your hands yet.

The strategic read: this squeezes dense small models hardest. The default local stack — a dense 4B–9B model at 4-bit — now competes with something that has 4× the capacity, comparable memory, and multiples of the decode speed. Apple already ships its on-device foundation model at 2 bits per weight; the direction of travel is unanimous, and cheap-tier cloud inference APIs should be nervous about every workload that fits in 6 GB.

My verdict: genuine shift, preview-grade execution. The physics checks out, the lineage from BitNet through Bonsai is real research rather than launch-week theater, and the MoE-times-ternary compounding is an insight others will copy within months. But hold the confetti until third parties reproduce the throughput and the benchmarks escape the PNG. Watch for independent evals and a GGUF-style mobile runtime — those two things turn this from an impressive demo into the new baseline for local inference.