AI Crawlers Broke the Web's Oldest Bargain

The week Nick Gray published his traffic postmortem, his analytics dashboard recorded 5,977 pageviews. His server answered 2.5 million requests and delivered 1.28 million full pages. That's 214 page loads he couldn't see for every one he could — on PatronView, a 1.5-million-page database of American museum donors he built from IRS 990 filings. The post hit the Hacker News front page today, and the comments did something unusual: instead of debunking the headline number, operator after operator confirmed it. One runs a site showing nearly a million monthly "unique visitors" in Cloudflare against a few dozen real daily users. Another manages a 500-user GitLab instance that had to lock down repository browsing entirely because distributed crawlers were burying it.

The 99% figure sounds like an outlier next to Imperva's industry-wide finding that automated traffic passed 53% of all web traffic in 2025. It isn't. Both numbers are true, and the gap between them is the actual story.

Bot share is a function of site shape

Human traffic scales with your audience. Crawler traffic scales with your URL count. A site with 1.5 million pages and a niche audience of fundraising researchers presents an enormous surface to every crawler on earth while attracting a few hundred humans a day — so its bot share converges on 99% no matter what it does. Imperva's 53% is an average dragged down by high-traffic commercial sites where humans still show up in volume. If you run anything with a big page inventory and a small audience — a package registry mirror, a docs site, a data catalog, a public Git forge — you're on the long tail, and the long tail is where the bots live.

Which means the first practical lesson is about observability, not blocking. JavaScript-based analytics — Plausible, Fathom, Google Analytics — only count clients that execute your beacon, and almost no crawler does. Gray's stats saw under half a percent of his real request volume. If your capacity planning, caching strategy, or "who is hammering the DB" investigations start from a JS analytics dashboard, you're reasoning from a rounding error. Server logs and edge analytics are the ground truth; the pageview dashboard is a survey of the minority of visitors polite enough to run your tracking script.

The referral contract is dead

The deal that made crawling tolerable for thirty years was simple: you burn my bandwidth, you send me readers. Gray put numbers on how unevenly that contract now performs. Googlebot crawled 46 pages per visitor referred. Bingbot ran 406:1. Claude-SearchBot fetched 420,680 pages in one week — 4.63 GB — and sent him 12 humans, a measured ratio of 35,000:1. Amazon's Amzn-SearchBot, which feeds Rufus and Alexa answers, was reading 117,000 pages a day as his single biggest crawler and had referred exactly zero visitors. Ever.

This isn't one angry site owner's anecdote. Cloudflare's own Radar data measured Anthropic's crawl-to-refer ratio at roughly 70,900:1 during one June 2025 week, with later snapshots in the four-to-five-figure range. In the HN thread, the operator of a financial-data site reported Claude-SearchBot pulling ~205,000 pages in 72 hours for a single referral. The asymmetry is structural: an AI assistant's product is the answer, consumed in place. There's no mechanism for traffic to flow back, so it doesn't.

Gray's response is the most useful takeaway in the piece: he turned crawl-per-referral into his firewall policy. A crawler that never sends a visitor gets blocked. That's a metric any site can compute from logs plus referrer data, it's vendor-neutral, and it neatly separates Bing (406:1 but growing referrals — keep) from Amazon (infinite — block). Expect it to become the standard yardstick, because it prices exactly the thing site owners actually sell: attention routed back to them.

What holds and what leaks

The defensive picture is less cheerful. The credible parts of Gray's stack: block by ASN rather than IP (IPs rotate; networks don't), challenge rather than block wherever humans might be inside, and use challenge solve rates as feedback — his ran 0.24% across 106,437 challenges, meaning the rules were taxing bots, not people. The well-behaved crawlers genuinely honor a 403: after he blocked Claude-SearchBot, Anthropic's traffic fell from 60,000 requests a day to about 25 attempts.

Two findings deserve wider attention. First, your defenses have a cost you should profile like any other dependency: Cloudflare's JavaScript Detections script was adding 2,875 ms on a mid-range phone — ten times his site's own JavaScript — and turning it off took his Lighthouse score from 58 to 99. Security tooling that outweighs the application it protects is a bug, not a posture.

Second, the ceiling on all of it: residential proxy networks. Gray watched scraping traffic arrive from 124,000 unique IPs in a day, spread across American home ISPs, with each address making a handful of requests under his rate limit. Geo rules can't touch it, ASN rules can't touch it, and per-IP rate limits are designed around. When one wave came from 361,844 Chinese IPs in a single day — 3.6 million requests — he could block the country. When the same shape returned on Comcast addresses, he had nothing left but heuristics like challenging frozen Chrome 118 user-agents. Every defense that works today works because most scrapers are lazy, not because the determined ones can be stopped.

There's also an uncomfortable second-order effect. Gray's ruleset — like most Cloudflare configurations — exempts cryptographically verified bots, which in practice means Google, Bing, and Apple. Every site that adopts this posture raises the drawbridge behind the incumbents. A startup building a new search index in 2026 starts from behind ten million WAF rules that Googlebot walks through.

Charge rent

The whack-a-mole isn't a strategy; it's a holding pattern. Scraping keeps getting worse because scraper costs fell faster than defenses improved, and residential proxies broke network-level identity for good. The durable fixes both replace guessing with declaration: Web Bot Auth, now an IETF draft, has crawlers sign requests with Ed25519 keys so identity stops depending on IP ranges — and pay-per-crawl puts an HTTP 402 price tag on access, so the crawl-to-refer imbalance gets settled in cash instead of referrals. Gray says he'd happily sell Amazon those 3.5 million pages a month at a fair rate. That's the right instinct. The open question is whether crawler operators adopt signed, paid access before site owners finish walling off the long tail — because the current trajectory ends with the open web readable only by the three crawlers everyone whitelisted in 2010.

Until the market exists, steal the metric. Pull your crawl-to-refer ratios out of your logs this week. You'll learn more from that one number than from anything in your analytics dashboard — which, as we've established, can't see the war at all.