Cloudflare's Quantization Math Is Right. The Disclosure Isn't

Cloudflare published the kind of engineering post most inference providers won't: an itemized list of exactly how it degrades — sorry, optimizes — the models you're paying for. On Workers AI, Moonshot's Kimi K2.6 now runs with its KV cache quantized from BF16 to FP8, and Z.ai's GLM 5.2 runs with weights compressed from FP8 to INT4 for the decode path. The numbers are genuinely good. The disclosure model around them is the part worth arguing about.

The math checks out

Start with what Cloudflare actually did, because the engineering is sound. Halving the KV cache from 16-bit to FP8 (e4m3) doubles how much context fits on a node: for Kimi K2.6 on H200s, from roughly 686,000 tokens to about 1.37 million. That matters more than raw speed, because on trillion-parameter mixture-of-experts models the cache — not the weights — is what runs out first under concurrency. In Cloudflare's benchmarks, BF16 falls over at 64 concurrent requests with an out-of-memory error while FP8 keeps going, hitting 2,192 tokens per second — about 41% above BF16's peak — at roughly 30% lower cost per token. The per-request tax is real but small: 137 vs. 125 tok/s at concurrency 1.

The GLM 5.2 story is weight compression: the checkpoint shrinks from 705 GB to 421 GB, per-GPU memory drops from 88 GB to 52 GB across an 8-way tensor-parallel deployment, and single-request decode jumps 55% because decode is memory-bandwidth-bound and INT4 moves fewer bytes. Cloudflare splits prefill and decode into separate pools and quantizes each where it's cheap — FP8 for compute-bound prefill, INT4 for bandwidth-bound decode. That disaggregated design is the current state of the art, and it's telling that even Cloudflare — which spent 2025 promoting Infire, its in-house Rust inference engine — says all of this runs on SGLang. When a company that built its own engine to beat vLLM serves its flagship open models on an open-source framework, that's a signal about where trillion-parameter MoE serving has consolidated. Writing your own engine buys you efficiency on Llama-class models; at frontier MoE scale, you run what the ecosystem runs.

The most underrated piece is the KV cache integrity check. When hundreds of requests share paged cache memory on one GPU, a reallocation bug means your request can silently decode from someone else's context — a cross-tenant leak with no error message. Cloudflare now tags every physical cache page, records which pages and tags each request expects, and aborts the request on any mismatch, for under 1% overhead. Every multi-tenant inference operator should steal this. Most serving-stack failures of this class have historically surfaced as "the model got weird," which is exactly the failure mode nobody debugs.

The part the blog post doesn't say

Here's the criticism that stuck when the post hit Hacker News: none of this is on the model page. If you pick Kimi K2.6 off the Workers AI catalog, nothing at the point of purchase tells you the KV cache is FP8 or that GLM's decode weights are INT4. You learn it from a blog post, if you happen to read it. One commenter called the practice borderline fraudulent; that's too strong for a provider that published a benchmark table, but the instinct is right. Quantization isn't a backend implementation detail anymore — it's a product spec, like an SLA.

We know this because the ecosystem already fought this battle once. When Kimi K2 launched, third-party hosts served it at whatever precision fit their hardware, tool-call accuracy quietly cratered on some of them, and Moonshot AI had to ship the K2 Vendor Verifier — a public harness that replays about 4,000 tool-call requests against each provider and compares the results to Moonshot's own API — to name and shame endpoints into honesty. OpenRouter now labels quantization per endpoint for the same reason. Model authors building public verification tooling to police their own hosts is a remarkable institutional failure, and it's the direct consequence of treating precision as a blog-post topic instead of a product field.

Cloudflare's accuracy defense is that quantized and baseline models score within about a point of each other on GSM8K, ARC, MMLU, MMLU-Pro, and tool-call validity checks. True, and also exactly the evidence the skeptics on HN picked apart: no coding benchmarks, and nothing that measures long-horizon agentic work — which is what Kimi K2.6, a model Moonshot built explicitly for agentic loops, is for. Short-form QA benchmarks are the easiest possible test for quantization, because a slightly perturbed token distribution rarely flips a single answer. In a 200-step agent run, small perturbations compound: one malformed tool call mid-trajectory and the task fails in a way MMLU will never show you. Notably, Moonshot itself ships K2.6 with INT4 quantization-aware training baked into the MoE layers — quantization done with the model author's participation. Post-hoc cache quantization by a host is a different animal, and pretending the two carry identical risk is where "no change in model accuracy" becomes marketing.

What to actually do with this

If you're running chat, summarization, extraction — anything short-horizon — take the deal. A 30% cost reduction for a within-noise accuracy delta is free money, and Cloudflare's capacity math means fewer 429s at peak. This is the right trade and most providers make it silently; at least this one showed its work.

If you're running long agentic workloads, treat provider precision as a variable you test, not a fact you're told. Concretely: run the K2 Vendor Verifier (or a scaled-down version of your own tool-call suite) against every endpoint you're considering, including the model author's first-party API as the reference. Diff tool-call validity rates, not benchmark scores. If a provider won't state its serving precision when asked, that's your answer.

And if you're a provider: the lesson from Cloudflare's post isn't the FP8 math — everyone's KV cache will be FP8 within a year, and Blackwell makes low-precision formats the default path. The lesson is that the first provider to put "KV cache: FP8-e4m3, weights: INT4 (decode), verified against first-party API" directly on the model card turns transparency into a selling point instead of a Hacker News liability. Quantization won the technical argument a while ago. Disclosure is the only fight left, and right now everyone's losing it on purpose.