Gemini Robotics ER 2 Turns Robot Control Into Agent Engineering

Google DeepMind shipped three robotics models today, and the one that matters to most developers is the one that doesn't move a single motor. Gemini Robotics 2 — the vision-language-action (VLA) model that actually drives humanoid arms and five-fingered hands — is restricted to early-access partners. Gemini Robotics On-Device 2 is trusted-testers only. But Gemini Robotics ER 2, the "embodied reasoning" model that plans tasks, watches video feeds, and orchestrates other robots and models as tools, is live right now in the Gemini API and Google AI Studio, with a private preview on the Gemini Enterprise Agent Platform.

That split is the story. DeepMind is betting on a hierarchical stack — a cloud "brain" that reasons and delegates, sitting above low-level controllers that execute — and it's handing out the brain while keeping the hands locked up. If you build software, that's an invitation. It's also a tell about what's actually hard.

Robotics development just became agent engineering

ER 2 is built on Gemini 3.5 Flash, takes interleaved text, image, video, and audio in a 128K context window, and outputs text — which in practice means plans, tool calls, and spatial annotations. It plugs into the Gemini Live API's bidirectional streaming endpoint, so the model can keep reasoning about upcoming steps while the robot executes the current one instead of freezing between thoughts.

Look at what that toolkit is: function calling, streaming multimodal sessions, tool orchestration. It's the exact same set of primitives you'd use to build a customer-support agent. DeepMind's flagship demo makes the point explicitly — ER 2 drives Boston Dynamics' Spot by calling Spot's existing navigation and manipulation APIs as tools, fetching objects from natural-language commands. No VLA model in the loop at all. If your robot — or your warehouse conveyor, or your lab instrument — already exposes an SDK, ER 2 will treat it like any other function schema.

This collapses a real barrier. The previous generation, Robotics-ER 1.5 and 1.6, already offered pointing and trajectory reasoning through the API, but the loop was fundamentally request-response: send a frame, get coordinates back. ER 2's continuous video understanding changes the shape of the system you'd build. The model tracks task progress from a live feed, detects when a step has actually finished, and decides when to advance — the supervisory loop that today lives in hand-rolled state machines and ROS behavior trees. That's the layer robotics startups have been reimplementing badly, over and over, for a decade. Middleware vendors selling "orchestration for robot fleets" should be nervous; robot OEMs with clean APIs just became more valuable.

Read the benchmarks before you fire your safety operator

DeepMind's own numbers argue for enthusiasm and against autonomy in the same breath. Moment-finding — pinpointing the video frame where a critical event happens — hits 91.3% accuracy with a 0.96-second mean absolute error, and DeepMind claims roughly 4x the execution speed of larger competing models. That's genuinely useful: fast, cheap "did the gripper actually close on the part?" checks are the difference between a demo and a deployable cell.

But progress classification — bucketing task completion into five levels from 0 to 100% — sits at 57.4%. That leads ER 1.6 and frontier competitors, and it's nearly triple the 20% chance baseline, yet it still means the model misjudges how far along a task is more than four times out of ten. If your architecture trusts ER 2 to decide "step complete, move on" unsupervised, you will ship a system that confidently advances past failures. The right pattern for now is ER 2 as proposer, with cheap deterministic checks — force sensors, fiducials, barcode scans — as verifier. The model card is blunt about the ceiling too: not for safety-critical applications, full stop.

The safety work that is real: improved scores on Safety Instruction Following and Human Proximity benchmarks, including halting a humanoid when a person approaches. Worth noting, and worth not relying on — a cloud round-trip is nobody's e-stop.

Multi-robot collaboration, minus the fleet manager

The most forward-looking capability is heterogeneous multi-robot coordination: DeepMind shows an Apptronik Apollo 2 humanoid and a Franka F3 Duo arm handing work off through what it calls shared semantic understanding — one model instance reasoning about what each embodiment can do and sequencing them through a workflow neither could finish alone.

The prior art here is instructive. Multi-robot coordination has meant explicit task-allocation frameworks — Open-RMF-style traffic management, auction algorithms, carefully engineered shared world models. Those systems are robust precisely because they're narrow. ER 2's pitch is that a language model's world knowledge can replace the engineered ontology: the arm "knows" it can't walk, the quadruped "knows" it can't screw in a bulb. That's a much more flexible abstraction and a much less predictable one, and anyone who's watched an LLM agent loop on a stale observation knows the failure modes transfer.

Hype or shift?

Shift — but a narrower one than the launch video implies. Against the vertically integrated end-to-end camp — NVIDIA's GR00T line, Figure's Helix, Physical Intelligence's π models — Google is making the opposite bet: that the reasoning layer generalizes across embodiments even if motor control doesn't, so ship the reasoner broadly and license the motors carefully. The gated VLA's own numbers back the caution — 92% on unscrewing a light bulb, 44% on tying a trash bag, 32% on using a dustpan. Dexterity is not solved, which is exactly why the hands aren't in your API tier.

What is solved enough to build on: a streaming, tool-calling, video-native planner that any developer can wire to hardware they already own, this afternoon, with the same skills they use for chat agents. Start with a robot that has a real SDK, keep ER 2 out of the safety path, verify its progress claims with dumb sensors, and treat the 128K context as your world model. The demos oversell the autonomy. The API undersells the shift.