Your CDN Failover Plan Dies With the Dashboard

On November 18, 2025, a database permissions change at Cloudflare caused a Bot Management feature file to double in size, trip a hardcoded 200-feature limit, and panic the Rust-based proxy at the heart of the network — Result::unwrap() on an Err, propagated to every machine in the fleet. X, ChatGPT, Spotify, Discord, and a large slice of the web returned 5xx errors for roughly three hours; full recovery took until 17:06 UTC. Cloudflare's own postmortem called it the company's worst outage since 2019.

Nine months later, the detail that should still bother you isn't the unwrap. It's this: customers who wanted to route around the problem couldn't log in to do it, because Turnstile — Cloudflare's CAPTCHA — sits in the dashboard's own login flow and was down too. Teams using Cloudflare Access for SSO were locked out of their internal tools at the same moment. The vendor you bought for resilience failed, and it took your ability to respond down with it.

That's the threat model worth designing against, and it's more specific than "add a second CDN."

The bundling is the single point of failure

Nobody buys "a CDN" from Cloudflare anymore. You get authoritative DNS, reverse proxy, WAF, TLS termination, zero-trust auth, serverless compute, and a CAPTCHA, all behind one control plane. That bundle is genuinely great value — it's also why November 18 hurt the way it did. A data-plane bug in one product (Bot Management) surfaced as login failures in a completely different one (the dashboard), because Turnstile was on the critical path for both.

We've seen each piece of this before. The 2016 Dyn DDoS taught everyone to run two authoritative DNS providers — advice most teams promptly forgot once DNS came bundled free with the proxy. Fastly's June 2021 config bug showed a single CDN vendor could blank out major media sites in under an hour. And November 18 didn't happen in isolation: it landed weeks after the AWS us-east-1 DynamoDB DNS failure and the Azure Front Door outage, a fall in which every major edge and cloud provider took a turn. The base-rate argument for treating your provider as fallible has stopped being theoretical.

The new lesson from November is about control planes. Your incident plan almost certainly begins "log in and change X." If your login, your DNS, and your broken proxy all live with the same vendor, that plan has a dependency cycle in it.

What to build, in order

The tempting response is full active-active multi-CDN. For most teams that's the wrong first move — it doubles your config surface, forces you to the lowest common denominator on WAF rules and edge logic, and dilutes your cache hit ratio across two networks. Do the cheap, high-leverage layers first.

1. Break-glass access (a day of work). Keep at least one admin path that doesn't traverse your edge vendor: an IdP configured outside Cloudflare Access, API tokens stored where on-call can reach them offline, and a printed-or-mirrored runbook. On November 18, the dashboard was mostly up — only the Turnstile-gated login was broken — and the API kept working for many operations. Teams with tokens in hand could act; teams who only ever used the web UI couldn't.

2. Split DNS from the proxy (a week). Run two authoritative DNS providers and delegate to both at your registrar. The catch is synchronization: Route 53 doesn't speak AXFR zone transfer, so the practical pattern is infrastructure-as-code with octoDNS or Terraform pushing identical zones to both providers on every change. Note the plan-tier trap: pointing external DNS at Cloudflare's proxy via CNAME (rather than using their nameservers) requires a Business plan or above. Keep TTLs on your critical records at 300 seconds or less — failover you can't propagate isn't failover.

3. A direct-origin escape hatch (a week, with gotchas). A origin.example.com record that bypasses the proxy entirely sounds trivial, but two defaults will silently break it. If your origin serves a Cloudflare Origin CA certificate, browsers hitting it directly get a trust error — that cert only validates behind Cloudflare's edge, so you need a publicly trusted cert from Let's Encrypt or similar on the origin itself. And if you've locked your origin firewall to Cloudflare's IP ranges (as their hardening guides recommend), your escape hatch is firewalled shut. Both are the kind of thing you only discover during an outage — or a drill.

4. A warm fallback that isn't your origin. Here's the trade-off the multi-CDN pitch glosses over: if your CDN offloads 90% of traffic, failing over sends 10x load at an origin sized for the other 10%. Unless you've provisioned for that, "failover to origin" means "choose which way to be down." For many products the honest fallback is a static degraded-mode page — status, cached critical content, an apology — served from a provider on different infrastructure. Your status page, obviously, belongs off your primary CDN too, along with the monitoring that decides you're down: external probes hitting the origin directly, not through the edge.

Full active-active multi-CDN with health-checked DNS steering sits above all of this, and it's real — but it's a staffing commitment, not a config change. If a minute of downtime costs you less than an engineer-month per quarter, the tiers above buy most of the resilience for a fraction of the price.

Failover you haven't rehearsed is a liability

The uncomfortable counterweight: most outages, including Cloudflare's, are self-inflicted configuration changes. A second provider is a second place to make that mistake, and an unrehearsed emergency DNS flip — wrong record, stale runbook, cert expired eight months ago — can extend an incident rather than end it. If you build any of the layers above, drill them on a schedule, timed, with the person on rotation rather than the person who built them. The first run will be embarrassing. That's the point of doing it on a Tuesday afternoon instead of during the next November 18.

Cloudflare's remediations — treating internal config files as untrusted input, more kill switches — are the right fixes for last November's bug. They will not fix the next one, at Cloudflare or anywhere else. The providers have accepted that they'll fail occasionally; their postmortems say so plainly. The only unresolved question is whether, when it happens, you're locked out of your own recovery — and that part, unlike the Rust panic, is entirely yours to fix.