The Tokio/Rayon Trap Kills Async Concurrency

Async/await won because it looks like regular code. You write sequential logic, sprinkle await, and the runtime promises to keep the machine busy. In Rust that promise usually means Tokio. The syntax is easy. The operational model is not.

The trap is structural. Async/await conflates asynchrony (yielding while you wait on I/O) with concurrency (running multiple units of work without starving each other). A cooperative executor only switches tasks at .await points. Put a 50ms JSON parse, a crypto proof, or a big collection walk in the middle of an async function and that OS thread stops serving everyone else. Latency spikes across unrelated connections. Cores sit idle. The hardware is fine. Your scheduling model is not.

Teams keep rediscovering this in production. The standard answer is the same every time: keep Tokio for I/O, ship CPU work to a dedicated pool such as Rayon. That split is the trap, not the fix.

Why the syntax lies to you

Blocking code and async code look almost identical. That is the marketing win and the production failure. Under the hood you are writing a state machine the compiler desugars for you. Control flow is hidden. Hardware realities (cache lines, core affinity, preemption) are hidden. Scheduling decisions get pushed back onto the developer the moment the workload is not pure network wait.

Rich Hickey’s easy-vs-simple distinction still applies. Familiar syntax is easy. Untangled structure is simple. Async/await is the former. Rob Pike has made a related point about language design: async/await is smaller for implementers to ship than something like goroutines and channels, but it leaves “colored functions” and a pile of complexity for the programmer. When an environment also offers a second concurrency model (thread pools, work-stealing, manual channels), the combination becomes actively dangerous.

In a Tokio multi-threaded runtime, one long CPU stretch does not just slow that task. It blocks the worker thread until the next await. Other tasks pinned to that worker wait. Tail latency goes nonlinear. You measure “the network is fine” while the process is effectively single-threaded for stretches of tens of milliseconds.

The human-in-the-loop scheduler

The industry response is manual partition: I/O on Tokio, compute on Rayon (or a custom spawn_blocking pool). Every function now needs a classification decision. Does this belong on the async executor or the compute pool? Crossing the boundary means cloning or moving data, waiting on oneshot channels, and carefully avoiding re-entrancy that can deadlock.

Postmortems from teams at PostHog and Meilisearch have described exactly this untangling cost. The abstraction that was supposed to hide concurrency complexity turned application developers into the scheduler. You police boundaries by hand. You ferry messages between two runtimes with two mental models. If that is the steady state, async/await has failed at its job.

This is not a Rust-only pathology. Node.js has the same cooperative story. The difference is that Rust’s performance culture pushes people toward mixed I/O-and-CPU services, so the failure mode shows up earlier and harder.

Unbounded spawn is OOM with extra steps

A second failure mode rides along for free. tokio::spawn is cheap. When a downstream database or dependency slows under load, the accept loop keeps taking connections and spawning tasks. Memory and task counts are unbounded by default in most of these setups. Queues grow. The process burns RAM until the OOM killer ends the argument.

Queues do not fix overload. They delay the crash and make it more catastrophic. Infinite capacity is a lie that the defaults pretend is true. Backpressure has to be explicit: bounded channels, admission control at the edge, load shedding that fails fast. None of that falls out of “just await.”

Work-stealing is not free fairness

When people hit these stalls they ask for smarter schedulers: preemption, work-stealing, steal from the busy core. Fairness sounds good until you look at cache behavior at scale.

Moving a task to another core abandons L1/L2. A main-memory fetch is on the order of 100+ nanoseconds. At massive core counts the steal path itself becomes a bottleneck. When WhatsApp pushed Erlang’s BEAM hard on machines with 100+ cores, idle threads fighting over the global run-queue lock burned cycles that should have been useful work (as described by Robin Morisset). Even with better locks, the locality tax remains.

If you have already been forced to manually partition I/O threads from CPU threads to survive production, a generic work-stealing algorithm has already lost. You understand your topology better than the runtime does. Shared-nothing, thread-per-core designs exist for that reason: pin work, own the queue, avoid stealing unless the win is proven.

What this means in a real Rust service

If you ship Tokio today, treat CPU work as hostile to the executor until proven otherwise.

• Audit hot paths. Anything that can run tens of milliseconds without an await (serde of large payloads, compression, image work, crypto, graph walks) is a stall candidate. Profile with tokio-console or simple timing around suspect blocks; do not trust intuition.
• Isolate deliberately. Prefer a bounded spawn_blocking pool or a dedicated Rayon pool with a clear ownership story. Move ownership of the data; avoid shared mutable state across the boundary.
• Bound everything. Bound the accept path, the channel into the compute pool, and the outbound client concurrency. Default unbounded is a production bug waiting for traffic.
• Do not dual-run two full mental models forever. If half your logic is “must not touch the async threads” and the other half is “must not block,” you are paying the tax the language feature claimed to remove. That is a design smell, not a badge of sophistication.
• Measure tail latency under mixed load. Average QPS lies. Inject realistic CPU bursts in load tests and watch p99/p999 while I/O continues.

A minimal shape many teams converge on:

// CPU work stays off the async workers
let handle = tokio::task::spawn_blocking(move || {
    heavy_parse_or_hash(payload)
});
let result = handle.await??;

That is necessary, not elegant. Every call site is a reminder that the abstraction leaked.

Longer term, shared-nothing thread-per-core frameworks and explicit state-machine styles try to put control back in the open instead of hiding it behind compiler sugar. Whether any particular project wins is secondary. The constraint is the point: if the runtime cannot see your CPU work, you must schedule it yourself.

The judgement

Async/await is a solid I/O multiplexing tool. It is a weak general concurrency model. The Tokio-plus-Rayon pattern is widespread because the failure mode is real, not because the pattern is clean. Every time you hand-classify functions into “async-safe” and “compute pool,” you are doing the job the abstraction promised to do for you.

For pure network services with tiny handlers, cooperative async still earns its keep. For mixed workloads (most real backends), plan the partition on day one, bound the queues, and stop pretending that async fn means “this will share the machine fairly.” Easy to write. Expensive to operate. That is the trap.