#Inference

Nvidia's Switchyard Finally Makes Model Routing Boring

Nemotron 3.5 Lightning gets the headlines, but the Rust proxy quietly rewires how agent workloads buy compute.

Article · 2d ago1

Nvidia's Real Product Here Is the Router, Not the Model

Nemotron 3.5 Lightning is a capable workhorse, but NeMo Switchyard targets the escalation logic every agent team keeps hand-rolling.

Article · 2d ago1

NVIDIA's Switchyard Matters More Than Its New Model

Nemotron 3.5 Lightning is a solid agent workhorse, but the open-source router is what changes agent economics.

Article · 2d ago4

ECS Finally Learns to Schedule a Slice of a GPU

AWS closes a year-old gap for G6f slices — worthwhile for small-model inference, once you've done the pricing math.

Article · 6d ago0

Your SSD Is the New VRAM for Local LLMs

Swiftlet streams sparse Qwen experts off disk, fitting an 80B model into 4.3 GB of Mac RAM.

Article · 1w ago1

Your SSD Is the New VRAM

Swiftlet streams MoE experts from SSD, fitting an 80B Qwen in 4.3GB of RAM — and a 35B on an iPhone.

Article · 1w ago1

Cloudflare's Quantization Math Is Right. The Disclosure Isn't

FP8 caches and INT4 weights are becoming table stakes; what providers tell you at purchase time is the real fight.

Article · 1w ago0

AirLLM's 4GB 70B Trick Is Real, and Beside the Point

Layer streaming makes giant models fit; disk bandwidth decides whether you'd ever want to run them that way.

Article · 1w ago1

A 26B Model in 2 GB of RAM, Courtesy of Your SSD

TurboFieldfare streams Gemma 4's experts from flash, and the trick will outlive the tool.

Article · 2w ago2

Google Split Its TPU in Two Because Agents Broke Inference

Cloud Next 2026's bifurcated silicon, flattened networks, and managed sandboxes reveal what agentic workloads actually demand.

Article · 2w ago2

Discarded Teslas Still Deliver Local AI VRAM

Enterprise cards at $60–200 remain usable for intermittent inference and training if you tolerate older stacks and power draw.

Article · 1mo ago0

Popping the CPU-GPU Latency Bubble in Inference

Pipelined decoding techniques show that software optimization, not just raw hardware scaling, is the key to maximizing GPU utilization.

Article · 1mo ago2

OpenAI Jalapeno and the Shift to Custom Inference Silicon

Custom ASICs are replacing general-purpose GPUs for running large language models to survive the crushing cost of scale.

Article · 1mo ago7

The LLM Cost Cliff Your Budget Isn't Ready For

Per-token prices are collapsing, yet AI bills keep exploding. The two facts aren't a contradiction, and confusing them will wreck your business case.

Article · 1mo ago1

OpenAI's Jalapeño Chip Is a Bet on Inference Economics

A custom Broadcom-built ASIC for LLM inference puts OpenAI on the same vertical-integration path Google and Amazon paved years ago.

News · 1mo ago2