DuckDB Beat SQLite 100x at a Game SQLite Wasn't Playing

A benchmark post from Traceway, an open-source observability platform, hit the Hacker News front page last week under the title "Choose DuckDB rather than SQLite." The numbers are striking: on the same $16/month Hetzner box, swapping SQLite for DuckDB as the storage engine moved every performance cliff by roughly two orders of magnitude. Log ingestion jumped 15x. Dashboards that choked at 100k rows in SQLite stayed under a second at 10 million. A billion metric points fit in 10.8 GB.

The numbers are probably directionally right. The title is wrong, and the gap between the two is where the actual engineering decision lives.

Why nobody who knows both engines is surprised

SQLite is a row-oriented transactional engine. Every row lives together on disk, which is exactly what you want for "fetch user 4823 and update their last_login." DuckDB is a columnar, vectorized engine: each column is stored and compressed contiguously, and queries process values in batches that stay hot in CPU cache. Ask it to aggregate one column across 100 million rows and it reads a fraction of the data SQLite would, then chews through it at close to memory bandwidth.

An observability backend — sequential appends, then aggregations over time ranges — is close to the ideal columnar workload. The HN thread said it bluntly: this is choosing a hammer over a wrench for driving nails. The DuckDB team has never claimed otherwise; their own docs position the engine as OLAP-first, and DuckDB's academic lineage (it came out of CWI, the same Amsterdam research group behind MonetDB) is entirely analytical.

So treat the 100x as confirmation, not discovery. What the post actually demonstrates — and this part is genuinely useful — is that the confirmation holds on cheap commodity hardware with a 4 GB memory cap, not just on the fat analytics boxes where columnar engines usually get benchmarked. If you've been assuming embedded analytics means "ship the data to ClickHouse," a $16 VPS running an in-process engine is a real alternative now.

Keep the usual vendor-benchmark salt handy, though. These are Traceway's own numbers, single-shot runs rather than medians, generated from uniform-random data, and the authors admit they didn't measure reads under concurrent write load — which for a live dashboard product is the load. They also found mid-benchmark that their own ingest path, not either database, caused every anomaly they chased. That's an honest disclosure, and also a reminder of how often "database benchmark" measures the harness.

What the benchmark doesn't show you

If DuckDB dominates this hard, why isn't it simply the new default embedded database? Three reasons, and they're the ones that bite in production rather than in benchmarks.

Concurrency. SQLite in WAL mode lets one writer and many readers from separate processes coexist against the same file. DuckDB's model is stricter: a process that opens the database read-write takes an exclusive lock, and other processes can't even read until it lets go. Practitioners in the HN thread called this the single biggest productivity killer — closing the CLI so a script can query the same file gets old fast. DuckDB knows it: the Quack remote protocol, which turns a lock-holding process into a server for others, shipped in beta and per the official concurrency docs isn't slated to stabilize until v2.0 this fall. Until then, multi-process access is SQLite's game.

Point writes. Columnar storage makes single-row inserts expensive. Rough practitioner consensus puts DuckDB around three orders of magnitude slower than SQLite for individual-record writes — fine if you batch (Traceway ingests via buffered OTLP batches, which is why their write numbers look great), painful if your app does one INSERT per user action.

Maturity. SQLite is 25 years old, runs on billions of devices, and has one of the most brutal test suites in software. DuckDB hit 1.0 in mid-2024 and is a large modern C++ codebase; it's improved fast, but reports of production crashes still surface, and you should architect as if the process can die. SQLite is also a single small C file that compiles anywhere, including the minimal Alpine images where DuckDB's footprint hurts.

The pattern that actually wins

The framing "rather than" imagines one slot for one embedded database. The emerging pattern in local-first and self-hosted apps is two slots: SQLite for application state — sessions, config, anything touched row-at-a-time — and DuckDB for the analytical surface, sitting beside it in the same process.

The seam between them is thinner than it sounds, because DuckDB can attach a SQLite file directly:

INSTALL sqlite;
ATTACH 'app.db' (TYPE sqlite);
SELECT date_trunc('day', created_at) AS day, count(*)
FROM app.events
GROUP BY day ORDER BY day;

That's a real migration path: keep your OLTP schema exactly where it is, point DuckDB at it for the dashboard queries that were timing out, and only move data into DuckDB's native format (or Parquet) when scan speed on cold data starts to matter. No ETL pipeline, no second server. Traceway's own architecture — swap the storage engine behind an interface, keep SQLite as the option for small installs — is closer to this than their headline admits.

And DuckDB isn't the only occupant of the embedded-OLAP slot. chDB embeds ClickHouse's engine in-process, and ClickHouse itself remains the answer once you outgrow one node — which is exactly the comparison Traceway says it's running next.

The verdict

DuckDB earned its result, and the broader shift is real: embedded analytics is now a solved problem at scales that used to demand a warehouse, and a mid-2020s stack that ships log search or usage dashboards through a client-server OLAP cluster for a single-tenant install deserves a second look. But "choose DuckDB rather than SQLite" is bad advice as stated. Choose DuckDB for the workload SQLite was never built for, keep SQLite for the one it owns, and let the ATTACH statement do the diplomacy. The engines aren't rivals. Your queries just needed sorting into the right pile.