ZeroFS and S3 Files: same mount, opposite bets

Mount an object store and treat it like a filesystem, and two products now offer that story with very different guts. Amazon S3 Files (built on EFS) and ZeroFS both expose POSIX paths over a bucket. Shared interface, opposite design. The real question is not “which is faster on a microbenchmark” but what role the bucket is allowed to play: public object identity, or private persistence for a log-structured filesystem.

If files must remain ordinary S3 objects that existing tools can GetObject, S3 Files preserves that identity. If the bucket can be an opaque durability layer, ZeroFS trades direct S3 access for packing, compression, client-side encryption, and multi-cloud backends. That split drives cost, security, and every workflow that touches the data outside the mount.

Opposite layouts under the same API shape

S3 Files keeps a one-file, one-object contract. images/cat.jpg on the mount is the same key in the bucket. Active data and metadata live in AWS’s low-latency “high-performance storage” tier; the bucket is the durable, interoperable view. Changes are meant to flow both ways. Export starts after about 60 seconds without writes, so a busy writer delays S3 visibility. After export, standard S3 tooling works. S3-side edits flow back into the filesystem. On concurrent modification before sync, S3 wins and the file-side version moves to lost+found.

ZeroFS does not keep that identity. Metadata sits in an LSM tree; file contents are split into extents, compressed, encrypted, then packed into immutable segment objects. An S3 client sees ciphertext segments and SST/manifest objects, not pathnames. You cannot fetch a ZeroFS path with the S3 API or scan a segment as Parquet. In return, small files do not need one data object and one PUT each: extents share segment objects. The project is open source (AGPL-3.0, Rust), serves NFS and 9P for files and NBD for blocks from one process, and targets S3, S3-compatible stores, Azure Blob, and Google Cloud Storage.

Neither model gives you immediate S3 visibility of a just-written path. S3 Files exports asynchronously; ZeroFS never exposes mounted files through the object API at all.

Write path, cold path, cache

Both mounts lean on the client page cache. After that, the paths diverge.

On S3 Files, an NFS write hits high-performance storage and is durable there immediately; S3 export is asynchronous after write inactivity. Cold access can trigger import: listing a directory loads every object’s metadata and asynchronously copies files below an import threshold (128 KiB by default) into the high-performance tier. AWS has said a first listing of 1,000 objects may take several seconds. Larger files can stay in S3; reads of at least 1 MiB may go direct to S3. Sequential traffic also gets Linux NFS read-ahead.

On ZeroFS, over 9P, fsync uploads sealed data segments and flushes LSM metadata to object storage before success returns. The local RAM and disk cache is not write-back for durability: writes still reach object storage when segments seal and metadata flushes. Newly sealed segments can enter cache from bytes already in hand, so read-after-write need not issue a GET. A cold miss resolves extents through the LSM tree and coalesces adjacent frames into ranged GETs. It pays the first object-store round trip for what you asked for, without importing the rest of the file or every small file in the directory. Once the working set fits locally, data reads generate few S3 requests. Prefetch spans within and across segment objects.

Threat-model note: S3 Files keeps a clear object identity and AWS-managed high-performance tier. ZeroFS makes the bucket opaque and puts encryption and packing on the client path. Recovery and mount both require ZeroFS and the encryption password; structural metadata that remains visible is documented separately. If your incident response assumes “open the bucket and read the files,” only one of these layouts still works.

Cost and what you pay for

S3 storage and request charges apply either way. S3 Files adds high-performance storage and file access charges. The AWS pricing example used in the comparison at publication listed roughly $0.30 per GB-month for high-performance storage (with a 10 KiB minimum billable size per file), $0.03 per GB for file reads from that tier (including exports to S3), and $0.06 per GB for file writes into it. That is on top of S3 itself and synchronization-related costs.

ZeroFS’s bill is object storage and requests, plus the compute and cache that run the ZeroFS process. Packing and compression reduce object count and bytes for small-file heavy trees; encryption is mandatory (XChaCha20-Poly1305 on extents, data key wrapped with Argon2id; zstd or lz4 before encryption, codec changeable without migration). You pay for a daemon and cache disk; you do not pay AWS’s high-performance storage tier.

Rough rule of thumb from the economics, not a spreadsheet: many small, frequently rewritten files under S3 Files can tax both the per-file minimum on the high-performance tier and the export path. ZeroFS is built to pack those workloads. Large, infrequently rewritten objects that other services must read with GetObject favor S3 Files’s one-to-one layout.

What developers actually choose

Pick by interoperability first, then by durability semantics and ops surface.

Choose S3 Files when:

• Paths must remain real S3 keys for Lambda, Athena, ETL jobs, CDN origins, or partner tooling that only speaks S3.
• You want AWS-managed high-performance storage and NFS 4.1/4.2 without running a userspace filesystem process.
• Bidirectional sync with the bucket (and S3-wins conflict rules) matches how humans and pipelines already edit objects.

Choose ZeroFS when:

• The bucket is allowed to be private state. No requirement for path-level GetObject.
• You need client-side encryption of file contents, multi-cloud or S3-compatible backends, or block devices (NBD) under ZFS/ext4/VMs on top of object storage.
• The workload is small-file dense or you care about packing and compression more than object identity.
• You accept running and monitoring a process (NFS/9P/NBD, cache, optional leader/standby HA over the same bucket).

Adoption shape for ZeroFS is familiar: config with storage URL and credentials, password for encryption, mount via NFS or 9P (or NBD for blocks). Packaging exists for apt/dnf, install script, Docker, and a systemd unit. The project stresses POSIX testing (pjdfstest, xfstests), kernel builds on the mount, stress-ng, ZFS on NBD, and Jepsen-style checks including crash and HA scenarios. That is a stronger correctness story than most “S3 as FS” experiments, but it is still a log-structured system over high-latency storage: cache sizing, fsync behavior, and segment/metadata flush paths matter more than they do on local disks.

Caveats that bite in production:

• S3 Files export lag (60s idle before export begins) breaks “write then immediately process via S3 API” pipelines unless you redesign around the delay or stay entirely on the NFS side until export completes.
• ZeroFS pathnames are not S3 keys. Backup, forensics, and cross-service access go through ZeroFS (or through whatever you export yourself), not through the raw bucket.
• Object storage latency and consistency are not local disk. Both designs cache; neither removes the need to size cache and think about cold starts.
• Forum-level friction shows up with non-kernel NFS servers (mount options, rpcbind expectations, orchestration tools that assume classic NFS). Plan mount and share topology deliberately, especially under hypervisors and storage UIs that are picky about NFS versions.

The bet, not the mount

S3 Files is AWS’s answer to “make S3 look like files without lying about object identity.” ZeroFS is the open-source answer to “make object storage a durable backend for a real filesystem and block layer, with packing and encryption first.” Calling both “filesystem on S3” collapses the only decision that matters.

If other systems must treat the data as ordinary objects, S3 Files is the honest fit and ZeroFS is the wrong abstraction. If the mount is the product interface and the bucket is just durable ciphertext, ZeroFS’s layout is the more coherent engineering trade: fewer objects, encryption by default, multi-cloud, NBD when you need blocks. The interface looks the same. The bucket does not.