Linear Attention Just Graduated to Frontier Scale

When Moonshot AI dropped the Kimi Linear paper last October, it was easy to file it under "yet another linear attention variant" — a genre with a decade-long habit of great benchmarks and zero production deployments. Nine months later, that read has aged badly. The same architecture now sits at the core of Kimi K3, Moonshot's 2.8-trillion-parameter frontier model released in mid-July, and the paper is back on Hacker News because people are retroactively realizing what it was: a dress rehearsal, run in public, for the biggest bet yet on hybrid linear attention.

What KDA actually changes

The core mechanism is Kimi Delta Attention (KDA), a linear attention layer that maintains a fixed-size recurrent state instead of a growing KV cache. It builds on Gated DeltaNet — the delta-rule update that overwrites stale associations in the state rather than just accumulating them — and adds channel-wise gating, so each feature dimension decays at its own learned rate instead of the whole memory fading uniformly. The paper pairs that with a chunkwise kernel built on a specialized diagonal-plus-low-rank transition matrix, which is the unglamorous part that makes the whole thing fast on real GPUs rather than just in FLOP counts.

The architecture is deliberately not pure. Kimi Linear interleaves three KDA layers for every one full-attention layer (Multi-Head Latent Attention, DeepSeek-style, with no positional encoding — KDA handles position implicitly). That 3:1 ratio is the load-bearing design decision: the periodic full-attention layers preserve exact global retrieval, the thing linear attention has always fumbled, while 75% of the stack runs with constant memory per token.

The numbers from the 48B-total, 3B-active proof-of-concept — open-weighted on Hugging Face and trained on 5.7T tokens against a full-MLA twin under identical conditions — are the part worth taking seriously. Up to 75% less KV cache. On the RULER long-context benchmark at 128k, it scores 84.3 while running roughly 4x faster. At 1M tokens, time-per-output-token improves about 6x over full MLA. And on quality, it doesn't just hold the line: the paper reports it beating the full-attention baseline across short-context, long-context, and RL-style evals. Matched-training comparisons at this scale are rare and expensive, which is exactly why this paper mattered more than the dozen linear-attention papers before it.

The graveyard it climbed out of

Linear attention's history is a pattern of near-misses. Linear Transformers and Performer (2020) traded too much quality. RWKV and Mamba proved recurrent-style models could be competitive but never displaced transformers at the top end. The hybrid idea — mostly-linear with periodic full attention — got its first frontier-scale tests in 2025: MiniMax-01 at 456B parameters, then Qwen3-Next, which also picked Gated DeltaNet as its linear layer at a similar 3:1 ratio. Tellingly, MiniMax then retreated, shipping M2 on full attention after deciding the retrieval and infrastructure headaches weren't worth it.

So the field genuinely split, and Moonshot's contribution was to resolve the argument with receipts: a controlled ablation at meaningful scale, open kernels in the flash-linear-attention library, and then — the part nobody else had done — betting the flagship on it. K3 keeps the recipe, using MLA on every fourth layer with KDA everywhere else, across a 1M-token context window and a 2.8T-parameter MoE. vLLM's K3 support shipped within a week, with fused KDA decode kernels and prefix caching adapted for hybrid attention. That last bit matters: prefix caching, quantization, speculative decoding — all the serving machinery assumed a homogeneous KV cache, and it's now being rebuilt around hybrids. Infrastructure follows architectures that ship, not papers.

Why your inference bill cares

For anyone serving long contexts, KV cache is the binding constraint. At 1M tokens, cache size — not compute — dictates how many concurrent requests fit on a GPU and how fast decode runs, and agentic workloads make it worse: long-horizon coding sessions, tool-use loops, and RL rollouts all decode many tokens against huge accumulated contexts. That's precisely the regime where a 75% cache reduction and 6x decode speedup stop being benchmark trivia and start being the difference between one node and four. It's no accident this came from Moonshot, whose whole product line leans on 1M-context agentic use — or that K3's cache-hit pricing lands at $0.30 per million tokens.

If you want to kick the tires without touching K3's supernode-class hardware requirements, the 48B-A3B Kimi Linear checkpoints run on vLLM today as an OpenAI-compatible endpoint; the KDA kernel needs fla-core >= 0.4.0 and torch >= 2.6. The honest trade-offs: a fixed-size state is still lossy, so workloads that demand exact recall of arbitrary details from deep context lean hard on those every-fourth-layer MLA blocks — which also means prefill on those layers is still quadratic. This is a large constant-factor win, not an asymptotic escape. And hybrid-model tooling, while moving fast, remains younger than the transformer ecosystem it's disrupting.

The verdict

Call this one a genuine shift, not hype. The precise claims are Moonshot's own and deserve the usual skepticism, but the architecture has now survived the only test that counts — a frontier lab shipping its flagship on it — and the recipe (delta-rule linear layers, fine-grained gating, ~3:1 hybrid, hardware-aware kernels) has independently converged across Qwen and Kimi. Full attention isn't dying; it's being demoted to a seasoning, applied every fourth layer. For models that need to serve 256k-plus contexts economically, pure softmax attention is starting to look less like the default and more like a luxury you have to justify.