One Log Line Costs journald 50KB of Disk Writes

Log one line, pay 50 kilobytes. That's the deal systemd-journald has quietly been offering every Linux box that persists its journal, and thanks to some genuinely careful measurement in a GitHub issue that hit Hacker News this week, we can finally put hard numbers on a complaint sysadmins have been making for six years.

The setup: security researcher ValdikSS mounted an ext4 loop device dedicated to /var/log/journal, watched the block device's raw write counters alongside journald's cgroup io.stat, silenced everything else on the system, and logged a single logger -p info test. The journal entry itself — message plus all of journald's metadata fields, exported as JSON — is 752 bytes. What hit the block layer over the following sync window was roughly 87KB, with around 49–74KB of it attributed directly to journald's cgroup. On btrfs the same single line cost about 110KB of cgroup-attributed writes and over 400KB at the block layer — with chattr +C already applied to disable copy-on-write.

That's a write amplification factor somewhere between 65x and 500x depending on your filesystem. For a four-word log message.

Why this measurement matters

The original issue, filed in January against systemd 257.9 on Debian 13, showed a VM doing ~50 IOPS to log two HAProxy lines per second. On its own, that's the kind of report that gets bounced — an earlier incarnation of this exact complaint, issue #15292 from 2020, was closed after maintainers argued that iotop numbers are misleading because they're captured before kernel write coalescing.

That objection is dead now. Block-device sector counters and cgroup I/O accounting sit after every coalescing mechanism the kernel has. This is what actually reached the disk. The methodology is the story as much as the numbers: if you want a bug like this taken seriously, this is how you instrument it, and you can reproduce it yourself with DefaultIOAccounting=yes and a watch on /sys/fs/cgroup/system.slice/systemd-journald.service/io.stat.

Where the bytes actually go

Two design decisions compound here, and neither is a bug in the fixable-by-Tuesday sense.

First, the journal file format is a database, not a log. Appending one entry doesn't append one record — it writes the entry object, updates two hash tables, bumps header counters, and (the killer) updates the entry-array chain for every data object the entry references. A typical entry carries around 20 fields (_PID, _BOOT_ID, _SELINUX_CONTEXT, and friends). The format deduplicates the field values, but each of those shared data objects keeps its own list of referencing entries, scattered across the file. One log line means a dozen-plus small writes at discontiguous offsets — and on a block device, dirtying one byte in a page costs you the whole 4KB page at writeback. Do the math on 15–20 scattered touches and the 49KB figure stops being mysterious.

Second, journald does all of this through mmap. The debugging surfaced a detail that surprised even people who thought they knew journald: SyncIntervalSec= (default five minutes) only delays the fsync/fdatasync — the data writes themselves go to the persistent file on every single entry. There is no write-behind buffer. Andy Lutomirski, the x86 kernel developer, showed up in the issue thread to say he'd made the same architectural mistake in a database years ago, spent years fighting it, and concluded mmap-based durable appends are simply the wrong design — plain pwrite to an append-only file would behave far better under the page cache. Vito Caputo, who was paid to work on journald at CoreOS, said much the same on HN: the format disperses tiny datums across the file with no consideration for block-oriented storage, and his own contributions only ever addressed read performance, never the write path.

In other words: this isn't a regression. It's the architecture, and it's been the architecture since 2011.

Who actually gets hurt

If you're on a beefy server with local NVMe, you'll likely never notice. The people who should care:

• Anyone paying for IOPS. On cloud volumes with provisioned or baseline IOPS, a chatty service logging ten lines a second is quietly consuming a triple-digit IOPS budget for kilobytes of actual information.
• SSD-wear-sensitive desktops. ValdikSS's broader crusade started because an idle desktop had written 38.7TB to its SSD in two years — journald is one of several offenders, but a large one. One commenter measured 7GB of journald writes in 15 minutes during a pathological episode.
• Edge and embedded devices. eMMC and SD cards have small erase blocks and no wear-leveling headroom. Journald with persistent storage on a Raspberry Pi is close to a card-killer, and the btrfs numbers rule out that pairing entirely for logs.

The mitigation that actually works is refusing to let journald own persistence:

# /etc/systemd/journald.conf
[Journal]
Storage=volatile
ForwardToSyslog=yes
RuntimeMaxUse=64M

Journal stays in /run (tmpfs, zero disk writes), and rsyslog — or any plain-text syslog daemon — handles the durable copy with sequential appends that the page cache batches into a handful of blocks per thousand lines. You keep journalctl for the current boot and structured queries; you lose cross-boot journal history, which the text logs cover anyway. What doesn't work, per the thread's testing: Compress= (short lines never trigger it), chattr +C on btrfs (still 110KB per line), and lowering SyncIntervalSec expectations (it was never buffering your writes to begin with). One genuinely useful scrap: the amplification is mostly per-sync-window, not per-line — a burst of ten messages cost barely more than one. Steady low-rate trickles, like a health check every two seconds, are the pathological case.

Don't wait for the fix

As of this writing, no systemd maintainer has responded to the issue. Given that the 2020 version was closed on methodology grounds and the real fix means rearchitecting the write path — something WAL-shaped, where entries append sequentially and indexes get materialized in batches, the way SQLite solved this same problem in 2010 — I wouldn't bet on this changing inside a major release cycle or three. The measurement is solid, the cause is structural, and the workaround is a three-line config change. On any machine where disk writes cost money or lifespan, make it this week.