Google's HEIR Turns Encrypted AI Inference Into a Compiler Problem

Google's security team used the phrase "practical private AI" this week, and for once the load-bearing word isn't a new model or a new chip. It's a compiler. In a post by Jeremy Kun, Google laid out where its HEIR project — an open-source compiler toolchain for fully homomorphic encryption, first announced in 2023 — has landed: four working demos of AI inference on encrypted data, a bench of hardware-accelerator partners, and a claim, backed by an accompanying paper, that HEIR has become the de facto compiler for FHE work in both academia and industry.

That framing is the right one, and it's worth being precise about why. The cryptography behind FHE has been improving steadily since Craig Gentry's 2009 breakthrough showed it was possible at all; overheads that started at nine-plus orders of magnitude are now down to roughly a thousandfold for many workloads. What hasn't improved is the developer experience. Building an FHE application has meant picking a scheme (CKKS for approximate arithmetic, BGV/BFV for exact integers, CGGI/TFHE for boolean circuits), hand-arithmetizing your program for it, tuning parameters that trade security against noise growth, and marrying yourself to one library's API. Change any of those choices and you start over. That's the problem HEIR attacks: it's built on MLIR, takes annotated Python or Torch models at the top, and lowers them through scheme-level dialects down to backends like OpenFHE and Lattigo — or, increasingly, to purpose-built silicon.

The demos tell you the real scope

Look at what Google actually got running: a deep-learning recommendation model, credit-card fraud detection, an encrypted adaptation of the Kitsune network-intrusion detector, and a hotword detector. These are small models with structured inputs. There is no encrypted Gemini here, and there won't be soon — a roughly 1,000x compute overhead is survivable for a fraud-scoring MLP and fatal for transformer-scale inference. Google's own benchmarks are reported for single-threaded CPU execution, which is the polite way of saying latency is still the whole ballgame. That's also why the post spends so much time on hardware partners: Belfort, Niobium, Cornami, and Optalysys, a cottage industry of FHE-accelerator startups seeded in part by DARPA's DPRIVE program. HEIR's bet is that a shared IR lets your application code stay put while the silicon underneath gets fast — the same decoupling move LLVM pulled for CPUs.

The skeptic's counterpoint writes itself, and it showed up immediately in the Hacker News thread: if you want private cloud AI today, trusted execution environments already ship. Google's own Private AI Compute, announced last November, runs Gemini-class models inside hardware-attested enclaves on TPUs. Apple's Private Cloud Compute and startups like Tinfoil make the same play. TEEs handle LLM-scale workloads at near-native speed; the price is trusting the hardware vendor and the attestation chain. FHE's guarantee is categorically stronger — the server computes on ciphertext and mathematically cannot read your data — but you pay for it in latency and model size. These aren't competitors so much as two ends of a dial, and for the next several years the honest mapping is: TEEs for big models, FHE for small high-stakes ones.

Apple already showed the adoption pattern

If you want to know how FHE inference actually reaches production, don't extrapolate from the phrase "private AI" — look at Apple, which quietly became the largest FHE deployment on earth with iOS 18. Live Caller ID Lookup and Enhanced Visual Search both use homomorphic encryption (BFV, via Apple's open-source swift-homomorphic-encryption) to query server-side databases without revealing the phone number or photo embedding being looked up. Narrow feature, small computation, clear privacy story, invisible to the user. That's the template. Meanwhile Zama — the best-funded FHE startup — has pointed most of its TFHE stack at blockchain confidentiality, which tells you how hard it's been to find general-purpose commercial demand. Google standardizing the tooling layer is a bid to lower that bar: fewer cryptographers required per deployment.

What this means if you build things

The practical question isn't "should I encrypt my AI pipeline" — it's whether you have a workload shaped like the demos. The profile: a small model (think logistic regression, GBDTs, compact MLPs or CNNs), genuinely sensitive inputs, a server that shouldn't be trusted with plaintext, and tolerance for seconds rather than milliseconds. Fraud scoring, health-signal classification, network telemetry analysis, and private lookups all fit. If that's you, the HEIR workflow is: express the model in Python with annotations marking secret inputs, let the compiler handle arithmetization and parameter selection, and serve ciphertexts against an OpenFHE or Lattigo runtime.

Know the trade-offs going in. CKKS arithmetic is approximate, so verify your model tolerates the precision loss. Data-dependent branching doesn't exist under encryption — control flow becomes arithmetic, which is why compilers matter so much here. Ciphertext expansion means real bandwidth costs. Key management shifts to the client, which is the point, but it's now your problem. And note what FHE inference protects: the user's input, not your model — weights sit on the server in plaintext.

The judgment call: this is a genuine infrastructure milestone dressed in slightly oversold clothing. "Private AI" in 2026 headlines means LLMs, and FHE isn't touching those on any roadmap you should plan around. But the pattern Google is executing — standardize the compiler, let hardware vendors compete below the IR, port the academic literature into one toolchain — is exactly how a technology crosses from papers to production. FHE spent fifteen years as a cryptographic curiosity because every application was a bespoke research project. A real compiler ends that. The first mainstream FHE features will look like Apple's: small, specific, and boring — and developers who learn the shape of the workload now will be the ones who ship them.