#Llm Inference

Your Agents Are Waiting on the CPU, Not the GPU

Agentic workloads moved the inference bottleneck off the accelerator, and hardware ratios are quietly following.

Article · 5d ago5

How a 20B Model Hits 120 tok/s on an iPhone

DeepGrove's Maple-Preview stacks native ternary training on MoE sparsity, and the arithmetic largely holds up.

Article · 1w ago1

DeepSeek V4 Flash on One AMD GPU Took Nine Patches

A single 192 GB MI300X now serves the full checkpoint — proof of AMD's hardware edge and its lingering software gap.

Article · 1w ago1

Quantize the Decode, Not the Prefill

Cloudflare's production numbers for Kimi K2.6 and GLM 5.2 show where trillion-parameter inference actually gets cheaper.

Article · 1w ago5

Linear Attention Just Graduated to Frontier Scale

Kimi Linear looked like another attention paper in October — now its KDA design underpins a 2.8T-parameter flagship.

Article · 2w ago2

The Real Cost of 'Just Use vLLM,' According to Netflix

Netflix's Model Runtime team details the version pinning, GIL workarounds, and preemption bugs behind production LLM inference.

Article · 2w ago2

Hetzner Is Quietly Commoditizing LLM Inference

A free, OpenAI-compatible endpoint from Europe's cheapest host says more about inference margins than any benchmark.

Article · 2w ago1

KTransformers Turned CPU Offloading Into Real Infrastructure

The Tsinghua-born framework dissolved into SGLang and LLaMA-Factory, making trillion-class MoE models a single-machine problem.

Article · 2w ago1

AI's Energy Problem Isn't Your Chatbot Query

The reassuring per-prompt numbers measure the chatbot era, just as agents change the math by 200x.

Article · 3w ago2

VRAM Beats TOPS for 2026 Local AI GPUs

Memory capacity and bandwidth decide what models you can run; software maturity still decides how painful it feels.

Article · 4w ago2

Ditching HBM: Inside the Monolithic 3D AI ASIC

PhantaField’s Sophon PFG-1 puts 330 GB of DRAM on-die, bypassing the GPU memory wall for low-batch LLMs.

Article · 1mo ago2

The Real Cost of the Open-Weight Price Collapse

Why the 50x price gap between open-weight APIs and frontier models is rewriting the build-versus-buy playbook for developers.

Article · 1mo ago2

Disaggregating LLM Inference: Inside AMD's ATOM and ATOMesh Stack

AMD's native ROCm serving stack splits prefill and decode to eliminate head-of-line blocking on Instinct hardware.

Article · 1mo ago0

Why Noam Shazeer’s Move to OpenAI Reshapes the Frontier

The co-author of the Transformer paper leaves Google, signaling a shift from brute-force scaling to architectural efficiency.

Article · 1mo ago0