Your Agents Are Waiting on the CPU, Not the GPU

Red Hat published a post this week arguing that "the CPU is back" for LLM inference, and the Hacker News response was predictably allergic: GPUs generate the tokens, CPUs don't, end of story. The skeptics are right about the headline and wrong about the substance. Nobody serious is claiming a Xeon will out-decode an H200 — NVIDIA's own numbers put an H200 north of 7,000 generation tokens per second on Llama 3.1 8B, and no CPU gets within two orders of magnitude of that. The claim worth your attention is different: in agentic systems, token generation is no longer where the latency lives. And that changes how you should be provisioning hardware, starting now.

Where the latency actually went

The load-bearing evidence isn't Red Hat's — it's a paper from Intel and Georgia Tech researchers that profiled five real agentic workloads end to end. Their finding: CPU-side tool processing accounts for 50–90% of total latency, topping 90% in the worst cases. Not model inference. The stuff around it — sandboxed code execution, retrieval, parsing, orchestration between agents, validating and dispatching tool calls.

Anyone who's run coding agents in production will recognize this immediately. When your agent fires off a tool call that compiles a Rust crate, runs a test suite, or greps a monorepo, the GPU sits idle while a CPU does the work. Chain five of those into a single task and your wall-clock time is dominated by compile-test loops, not decode throughput. One HN commenter nailed the historical rhyme: the best human programmers were always bottlenecked by the compile-test loop too. Agents inherited our bottleneck.

This is why the "that's not inference, that's orchestration" objection misses. Technically true, practically irrelevant. Your users experience end-to-end task latency. Your finance team experiences the bill for GPUs that are idle 50–90% of the time. Call it what you want — it's a CPU problem sitting in the middle of your inference budget.

The ratio everyone is re-deriving

The strongest signal that this is real: every silicon vendor has independently converged on the same math. Intel's CEO has said the CPU-to-GPU ratio is shifting from 1:8 in training toward 1:4 and eventually 1:1 as workloads move to agentic inference — self-serving for Intel, sure, but AMD published essentially the same argument about its EPYC line. And NVIDIA, the company with the least incentive to talk up CPUs, has been shipping racks at 1 CPU per 2 GPUs since the GB200 NVL72 (36 Grace CPUs, 72 Blackwell GPUs) and kept that ratio for Vera Rubin, while making the Vera CPU markedly beefier. When the GPU vendor fattens the CPU, believe the workload data.

For developers this cashes out as an instance-shape problem. Standard cloud GPU shapes were specced for training-era assumptions — a handful of vCPUs per GPU, enough to feed the data loader. Run a multi-agent workload on one of those and you'll see it in your traces: GPU utilization sawtoothing while tool executions queue on starved cores. Intel calls this the "GPU idle tax," and the fix is unglamorous capacity planning — profile the CPU-seconds per GPU-second your actual agent traces consume, then pick shapes accordingly. If you're paying H200 prices for a card that spends most of each task waiting on a sandbox, adding cores is the cheapest inference optimization available to you.

The narrower case for CPU-only serving

Separate from the ratio argument, there's a real but bounded case for serving models on CPUs alone. vLLM ships a CPU backend with the features that matter — continuous batching, PagedAttention, chunked prefill — behind the same OpenAI-compatible API as the GPU backend, so moving a deployment is a config change, not a rewrite. On recent Xeons, AMX instructions do the matmul heavy lifting; quantizing to INT4 with llm-compressor shrinks the memory-bandwidth bill, which is the binding constraint for CPU decode.

Where this actually pencils out: small models with modest traffic. A fine-tuned 1–8B model — the SmolLM2 class, or a domain-specific distillation — handling internal classification, routing, or extraction at tens of requests per minute doesn't justify a five-figure accelerator. Edge and air-gapped deployments, where the GPU was never going to be installed anyway, are the other honest case. The trade-off is stark and shouldn't be sugarcoated: tokens per second and tokens per watt still favor GPUs by a wide margin at any real utilization. CPU serving wins only when the hardware is already there, the model is small, or the alternative is not deploying at all.

One caution if you evaluate this path: CPU inference benchmarking is currently a swamp of vendor-specific numbers, partly because CPUs share cores, memory bandwidth, and cache with everything else on the box — isolation takes NUMA-aware pinning that GPU benchmarks never needed. Red Hat's genuinely useful contribution here isn't the manifesto, it's tooling: a reproducible evaluation framework built on GuideLLM. Run your own numbers on your own model and traffic shape before believing anyone's slide deck, including theirs.

The verdict

"The CPU is back" is marketing. What's actually happening is subtler and more useful: the unit of deployment is shifting from a model to a system — models, tools, sandboxes, orchestrators — and the system's bottleneck has moved to the parts GPUs can't run. The winners are whoever sells server CPUs into the rebalance (Intel needed this narrative badly; NVIDIA hedged early with Grace and Vera) and any team that re-derives its CPU:GPU ratio from real agent traces instead of training-era defaults. The losers are capacity plans that treat "inference" as a synonym for "GPU." If you run agents, pull up your traces this week and check where the milliseconds actually go. There's a decent chance you've been optimizing the fast part.