The Approve Button Is Not a Security Boundary

Scale X has published the numbers behind "Continue? Y/N," a 60-second browser game where you play the human in the loop for an AI coding agent. Commands stream past — some routine, some built to steal credentials or wreck the machine — and you approve or deny each one against the clock. Across more than 40,000 runs and 409,000 individual decisions, players approved one in three threats. Seven percent approved everything.

It's a game, not a field study, and the caveats matter (we'll get to them). But this finding doesn't stand alone. It lands on top of two decades of security-UX research saying the same thing, plus telemetry from the agent vendors themselves. Put it all together and the conclusion is hard to dodge: the per-command approval prompt — the thing every coding agent ships as its primary safety mechanism — is not a security boundary. It's a consent form.

What 409,000 decisions actually show

Mean accuracy was 66.3 percent. Only about a fifth of players caught every threat while also not blocking legitimate work, and that second clause matters: a benign rm -rf dist/ was denied 45 percent of the time. Fatigue cuts both ways. The reflexive denier is failing at supervision as thoroughly as the reflexive approver; they're just failing closed.

The breakdown by threat type is the interesting part. Obviously destructive commands like rm -rf ~/ were missed only 11.7 percent of the time. Credential reads — an agent poking at ~/.aws/credentials — got through 35 percent. The runaway winner was npm run analyze, missed 64.7 percent of the time. Overall, exfiltration payloads hidden behind familiar npm script names were approved 52.5 percent of the time, against 28.4 percent for the same class of attack presented naked. Wrapping a payload in a boring name roughly doubles its success rate — and in the game, the payload was visible in the logs. The label did the work anyway.

We've run this experiment before

Warning fatigue is one of the oldest replicated results in usable security. Windows Vista's UAC prompts became such reliable click-throughs that Microsoft rebuilt the model for Windows 7 around showing fewer of them. Akhawe and Felt's 2013 "Alice in Warningland" study measured real browser telemetry and found users clicked through roughly 70 percent of Chrome's SSL warnings. Later lab work showed habituation is neurological: the brain stops fully processing a warning it has seen before, and the effect transfers to warnings that merely look similar.

What's changed is the rate. A browser shows you a few warnings a week. An agent in a tight loop generates dozens of prompts an hour, each one interrupting work you actually care about. If most people clicked through rare, high-salience browser warnings, expecting better from a developer mid-flow on their fortieth approval of the afternoon was always fantasy.

The vendors know. Anthropic's engineering post on Claude Code sandboxing frames permission prompts squarely as an attention problem, and its answer was to make prompts rarer, not better: OS-level filesystem and network isolation — bubblewrap on Linux, Seatbelt on macOS — that cut permission prompts by 84 percent in internal use, since open-sourced for other agent builders. When the company whose product popularized the approval prompt ships machinery to route around it, treat that as the industry's real position.

The npm finding is the whole story

A command name is not an action. Approving npm run analyze approves whatever package.json maps it to, plus the lifecycle scripts of every package it touches. Nobody reads all that at prompt speed; the approval is a vibe check on a string.

Attackers already operate on this. The August 2025 s1ngularity compromise of Nx slipped a malicious postinstall script into one of npm's most popular build tools — and, in a first, the payload invoked whatever AI CLIs it found on the machine (Claude Code, Gemini, Amazon Q) with their safety-bypass flags to hunt the filesystem for secrets. Some 2,349 credentials ended up base64-encoded in public GitHub repos. The command a human would have seen, if they saw anything at all, was npm install.

This is also why prefix-matched auto-approve lists — Cursor's auto-run config, Claude Code's Bash(npm run:*)-style rules — inherit the flaw rather than fixing it. They approve names too. Shell indirection, npm scripts, and a quietly edited package.json all launder arbitrary behavior through an approved name.

What the game can't tell you

Threat prevalence in the game was 34 percent, absurdly high; players were told threats were coming; there was a countdown clock. You could argue those conditions were rigged against the humans. I'd argue the opposite: real threat base rates are a rounding error, and vigilance decays with rarity — automation complacency is the TSA's problem, and now it's yours. The players were self-selected Hacker News readers primed to hunt for attacks, and they still missed a third. Fair objections surfaced in the HN thread all the same: some prompts are ambiguous (cat ~/.zshrc is benign or hostile depending on context the game doesn't give you), 40,000 runs is not 40,000 people, and this is one vendor's unreviewed telemetry from a company selling developer security. Hold the decimals loosely. The direction, though, is corroborated from every side that has data.

What to do instead

Treat the approval prompt as UX, not enforcement, and put enforcement where it can't get tired:

• Sandbox by default. Filesystem isolation scoped to the project, via devcontainers or the OS primitives Claude Code now ships. If the agent can't read ~/.aws, the prompt asking to read it doesn't need to exist.
• Egress is the choke point. Exfiltration needs a network. A deny-by-default domain allowlist stops whole classes of attack no human reviewer will reliably catch, including every npm-script trick above.
• Get credentials out of reach. Long-lived secrets in dotfiles and env vars are what made s1ngularity profitable. Short-lived, scoped tokens turn a successful exfiltration into a stale string.
• Spend human attention where base rates support it. Deploys, schema migrations, force-pushes: rare, high-stakes, reviewable. Approving those is a decision. Approving the fortieth npm run of the day is a reflex.

The uncomfortable read on the per-command prompt is that it was never really a control. It's liability transfer: the agent did it, but you approved it. This data says you'll approve about a third of the worst things it tries. Better to build the boundary out of something that doesn't blink.