The 16-Year-Old SQLite Bug That Ate Tailscale's Data

SQLite is, by its own accounting, the most widely deployed database engine on Earth, backed by a test harness that achieves 100% machine-code branch coverage and runs billions of checks per release. And for sixteen years — every version from 3.7.0 in July 2010 through 3.51.2 this past January — it shipped a race condition that could silently destroy committed data.

Tailscale hit that race nineteen times between August 2024 and February 2025, and this week published the retrospective on how its engineers and SQLite's developers finally cornered it. The bug is patched (SQLite 3.51.3, with backports to 3.44.6 and 3.50.7), but the story matters beyond the fix. It's the clearest demonstration in years that "rare" is not a property of a bug — it's a property of the bug multiplied by your workload. And the server-side SQLite renaissance is quietly changing everyone's workload.

A checkpoint that lies

In WAL mode, SQLite appends committed transactions to a write-ahead log file, then periodically runs a checkpoint that copies those pages back into the main database and resets the log. The WAL-reset bug is a race between that checkpoint and a concurrent write: if the write lands at exactly the wrong instant, the checkpointer becomes convinced it has copied pages that it never actually copied. The WAL gets reset anyway. Those pages — committed, fsynced, acknowledged to the application — are simply gone.

That failure mode is about as hostile as database bugs get. There's no error at write time, no error at checkpoint time. The damage surfaces later, if ever, as an integrity_check failure or as data that just isn't there. Tailscale's first symptoms were corruption reports on backup databases and hour-long recovery windows on affected control-plane shards.

"Cosmic-ray rare" depends on who's asking

SQLite's own documentation, updated after the fix, says the bug's occurrence rate is at or below "the expected occurrence rate of SSD malfunctions and/or cosmic-ray hits," and tells developers that upgrading is prudent but "not an emergency." For the median SQLite deployment — one process, one connection, automatic checkpoints — that's fair. The race needs two or more connections, in separate threads or processes, writing or checkpointing at the same instant.

Tailscale's control plane is not the median deployment. It takes manual control of checkpointing to get fast, consistent backups, and runs those checkpoints aggressively against a busy multi-connection database. That configuration turned a cosmic-ray event into a roughly monthly incident. Both readings of the bug are true at once: negligible for a phone app, an operational crisis for a company that leaned on WAL internals. When a vendor says a failure is rare, the honest follow-up question is: rare under whose access pattern?

How you find a sixteen-year-old race

Nobody read the code and spotted this. The SQLite developers — engaged through a professional support contract, which is how SQLite's development gets funded — couldn't reproduce it synthetically, and even after the fix had to write test logic that deliberately forces the interleaving in order to verify it.

What actually worked is a playbook worth stealing wholesale. Tailscale ran PRAGMA integrity_check against backups routinely, which is why they knew corruption was happening at all. They built a statement-level transaction log and replayed it between backups; when replay didn't apply cleanly, that proved committed writes had become invisible — a transactional-guarantee violation, not flaky hardware. Then the SQLite team wrote tmstmpvfs, a shim around SQLite's virtual filesystem layer that logs low-level file operations with timing, and Tailscale deployed it to production and waited for the next incident. The resulting telemetry let SQLite developer Dan Kennedy pin the exact interleaving; per SQLite's docs he nailed it on March 3, 2026, and 3.51.3 shipped ten days later.

There's a sharper lesson buried here about testing. Branch coverage — even SQLite's fanatical version of it — measures which code runs, not which schedules run. Thin-interleaving concurrency races are exactly the class that deterministic simulation testing was invented for, the approach FoundationDB pioneered and TigerBeetle and Antithesis have since productized. SQLite's harness, arguably the best in open source, doesn't explore schedules. Sixteen years of latency is what that gap costs.

And a postscript that should worry you: two months after deploying the fixed SQLite, Tailscale's new detection alert fired. The race happened again — harmlessly this time. It had been live in their workload all along.

What to do about it, concretely

Check what you're actually running — SELECT sqlite_version(); at runtime, not what your package manager claims. Affected: 3.7.0 through 3.51.2. Fixed: 3.51.3 and later (3.52.0 was withdrawn for an unrelated issue; 3.53.0 is current), plus backports 3.44.6 and 3.50.7. Bundled copies are the trap: Python's stdlib sqlite3, better-sqlite3, distro packages, and Docker base images all pin their own SQLite, and most are still behind.

Then triage honestly. Single connection, or WAL with default checkpointing in one process? Upgrade on your normal cadence. But if you run SQLite server-side with multiple writer connections across threads or processes — Litestream-style replication setups, background jobs calling wal_checkpoint, Rails 8's SQLite-in-production defaults, any multi-process web deployment — you're in the bug's actual habitat. Patch now, and adopt the detection posture regardless of version: integrity_check on every backup, and restore verification that replays or diffs rather than trusting the file copy.

The envelope moved

This shouldn't scare anyone off SQLite in production — if anything, the response is the advertisement. Support money bought access to the two people on the planet who could instrument the VFS layer, and once real telemetry existed, the fix took days, with backports for stable branches. Compare that to how fsyncgate dragged on in Postgres.

But Tailscale's own closing line — running boring technology in a non-standard way is a risk — cuts deeper than they let on. The whole server-side SQLite movement is precisely about running it in formerly non-standard ways: more processes, more writers, more aggressive checkpointing, replication bolted on top. The well-trodden path is being rerouted through territory where bugs like this live, and sixteen years of embedded deployments tell you almost nothing about what's waiting there. Expect more of these. Patch this one.