Fixing the DHT: How IPFS Achieved Sub-Second Publishing

Decentralized systems have long paid a heavy performance tax for their trustless, permissionless nature. If you have ever tried to publish content or update application state on IPFS, you know the drill. You run the publish command, and then you wait.

A simple write to the Distributed Hash Table (DHT) historically took anywhere from 13 to 20 seconds, sometimes stretching into minutes. For interactive applications, real-time debugging, or delay-sensitive developer workflows, this latency was a massive bottleneck.

With the release of Kubo v0.39.0, this paradigm has changed. By shipping an optimization called "Optimistic Provide," IPFS has cut content publishing latency by over an order of magnitude. The engineering behind this change shows how moving from rigid deterministic guarantees to statistical heuristics can make decentralized protocols competitive with traditional centralized infrastructure.

xychart-beta
  title "IPFS Content Publishing Latency"
  x-axis ["Traditional Provide", "Optimistic Provide"]
  y-axis "Latency (seconds)" 0 --> 20
  bar [20, 1]

The Anatomy of the DHT Publish Bottleneck

To understand why publishing was so slow, we have to look at how Kademlia-based DHTs handle content routing. IPFS uses the Amino DHT, where storing a record requires identifying the 20 closest peers to the data's Content Identifier (CID). In this context, closeness is calculated using an XOR distance metric between a peer's unique ID and the CID.

Traditionally, this process consists of two distinct phases:

1. The DHT Walk: The initiating node queries its local routing table for the closest known peers, asks them for even closer candidates, and repeats this cycle iteratively to find the 20 closest peers in the entire network.
2. The Follow-Up: Once the walk terminates, the initiator pushes the provider record to all 20 of those closest peers to ensure redundancy and data availability.

This architecture is highly resilient, but its termination condition was incredibly rigid. During the DHT Walk, the algorithm insisted on waiting for responses from the three closest discovered peers before declaring the walk complete.

In a permissionless network, nodes constantly join and leave. Because of this high churn, those specific three nodes were frequently offline, slow, or unreachable. The initiating node would hang, wait for timeouts, and backtrack to query more distant peers, even though it had already discovered the actual 20 closest active peers. This rigid waiting was the primary source of the multi-second delay.

The Mechanics of Optimistic Provide

Optimistic Provide, developed by the ProbeLab team and published at IEEE INFOCOM 2024, solves this bottleneck by replacing strict deterministic waiting with three statistical mechanisms.

1. Passive Network Size Estimation

To determine if a peer is "close enough" to the target CID, a node must know the scale of the network it is querying. Kubo nodes now locally estimate the global network size using a lightweight, bias-corrected proximity model. Crucially, this calculation piggybacks on the existing routing table refresh mechanism. It requires no additional network requests and introduces zero overhead.

2. Predictive Termination

Armed with an accurate estimate of the network size, the node can calculate the probability that its current set of discovered peers represents the true network-wide closest peers.

Instead of waiting for the walk to finish entirely, the node uses a 90 percent confidence threshold. Once it is 90 percent certain that a discovered peer is among the true 20 closest, it immediately initiates the write to that peer. Once it is 90 percent certain that its entire set of discovered peers constitutes the target group, it terminates the DHT Walk immediately.

3. Early Return

In the follow-up phase, the node no longer waits for all 20 target peers to acknowledge the write. As soon as a quorum of 15 peers confirms storage, the API returns success to the user. The remaining five writes are completed asynchronously in the background. This ensures high data availability without forcing the user to wait for the slowest nodes in the set.

What This Means for the Developer Stack

For developers building decentralized applications, this change shifts IPFS from a slow archival storage layer to a viable runtime component.

Previously, to bypass the 20-second write tax, developers had to architect complex workarounds. They built centralized write-through caches, relied on proprietary pinning services, or used pubsub channels to broadcast content identifiers directly to known gateways. These workarounds added architectural complexity and reintroduced single points of failure.

With Kubo v0.39.0, these workarounds are largely unnecessary. Sub-second publishing means you can write data and expect it to be discoverable almost instantly. This is particularly valuable for applications handling dynamic content, real-time state updates, or rapid debugging cycles.

There are, however, clear trade-offs to keep in mind:

• Statistical vs. Deterministic Guarantees: By returning early after 15 successful writes, there is a theoretical window where a highly unstable network could drop the remaining background writes. In practice, because the optimization reduces overall network overhead by 40 percent, the DHT is less congested, making data availability highly resilient.
• Adoption Requirements: To benefit from this, you must run Kubo v0.39.0 or later. The optimization is enabled by default, so upgrading your self-hosted nodes or gateways immediately unlocks the performance gains without configuration changes.

A Shift in Distributed Systems Design

The success of Optimistic Provide is a case study in practical distributed systems engineering. It proves that trying to force absolute, deterministic consensus in a highly dynamic, permissionless network is a recipe for poor performance. By embracing statistical "good enough" thresholds, IPFS achieves Web2-like speeds without sacrificing its decentralized foundation.