Node's Real Resilience Bug Is Policy Order, Not Breakers

Node.js has had a working circuit breaker for a decade. opossum is on version 10 and pulls about 1.3 million downloads a week; cockatiel shipped 4.0 in May and does roughly 2 million. So when a new package called breakwater shows up with a "why another one" section in its README, the reflexive answer is: you didn't need to.

Having read the code, the docs, and the pitch, I think that reflex is half right. Breakwater's breaker isn't the story. Its argument about ordering is, and it's an argument the Node ecosystem has mostly avoided having.

The bug nobody's library catches

Every mid-sized Node service accumulates the same pile: p-retry around one client, an opossum breaker around another, a hand-rolled AbortSignal.timeout somewhere else. Each piece works. The system they form doesn't, because nobody decided how they nest.

Take the two obvious ways to combine retry and a breaker:

retry( circuitBreaker( timeout( fn ) ) )   // A
circuitBreaker( retry( timeout( fn ) ) )   // B

In A, every attempt feeds the breaker's failure window individually, and once the circuit opens the retry loop gets a CircuitOpenError and stops. In B, three real failures against a dying dependency collapse into one recorded failure, the breaker opens far later than it should, and in the meantime retry sleeps through exponential backoff to hammer a host that's already down. Same three policies, opposite behaviour under load. Most teams pick B by accident, because they added retry inside the client method and the breaker outside it.

This isn't news to the JVM or .NET world. resilience4j documents a default aspect order (retry outermost, bulkhead innermost), and Polly v8 made pipeline order an explicit builder decision. Cockatiel ported Polly's wrap(retry, breaker, timeout) composition and gets the nesting right if you read the README carefully. What no Node library did was ship an opinionated default and explain the trade-off at length. Breakwater's resilience() does exactly that:

fallback( staleCache( retry( rateLimit( bulkhead( circuitBreaker( timeout( fn ) ) ) ) ) ) )

Two choices in there are worth noticing. Timeout is innermost so each attempt gets its own budget and a hang becomes a countable failure. And bulkhead and rate limit sit outside the breaker, on the reasoning that a full local queue says nothing about the dependency's health and shouldn't be allowed to trip its circuit — both rejections are marked retryable so the outer retry backs off through a burst instead. That's a defensible departure from resilience4j's default, and the docs argue it rather than assert it.

Where the comparison table is honest, and where it isn't

Breakwater's README ranks itself against opossum and cockatiel. I checked the claims.

On opossum, it's fair. Opossum is a breaker with a timeout option, a fallback, and a capacity semaphore that acts as a crude bulkhead. It has no retry policy at all, no backoff strategies, and Prometheus metrics live in a separate opossum-prometheus package. That's by design — opossum has never pretended to be Polly — but if you want a full pipeline, you're assembling it yourself.

On cockatiel, the original blog post's claim that "maintenance has slowed" is stale. There was a 22-month gap between 3.2.1 and 4.0.0, but 4.0 landed this May with ESM output, a Node 22 floor, halfOpenSampling for the breaker, and fixes to bulkhead queue starvation and abort propagation. The repo was pushed to last week. Cockatiel is alive.

What is true is that cockatiel's observability stops at per-policy onSuccess/onFailure/onBreak callbacks. There's no correlation ID crossing a wrap() pipeline, no aggregated stats, no collector interface. Breakwater's typed events all carry one correlationId, stats() on a breaker returns failure rate plus p50/p95/p99 latency over the same window, and a single MetricsCollector wires every policy in a pipeline. The breakwater/prometheus and breakwater/otel entry points are optional subpaths with prom-client and @opentelemetry/api as peer dependencies, so the core stays at zero runtime deps. If you've ever built the same Prometheus gauges around cockatiel three times, that's the gap being filled.

The Redis breaker is the ambitious part

A local breaker on a 40-pod deployment means 40 pods independently discovering the same outage. Breakwater's breakwater/redis store shares one circuit per name across the fleet, with transitions committed via a Lua script and a fence token so a stale probe result can't clobber a newer recovery period. When Redis itself is unreachable, the breaker degrades to local state, keeps the last agreed fleet-wide state (an open circuit doesn't spring shut), and reports the degradation once rather than per call.

That's the right design; it's also the part I'd trust least at version 1.1.2. The fleet-wide probe election disappears during a Redis outage, so N instances probe N times per cooldown — the docs admit this. And every execution still does a Redis read to decide, which is a network round trip on your hot path. There's a redis-overhead benchmark in the repo; run it against your own latency budget before you ship it.

Should you switch?

Here's the actual state of the project: first publish on July 24, eighteen releases since, 1.0.0 and 1.1.0 on the same day, five GitHub stars, around 400 weekly downloads, one maintainer. The features the launch post listed as "planned" — Redis state, Prometheus/OTel adapters, stale-while-open caching — all shipped within a month. That velocity cuts both ways: it's impressive, and it's exactly the churn profile you don't want under a payments client. The author says it's running in production behind a RabbitMQ client; I couldn't find any independent deployment.

So, concretely:

• Already on cockatiel, happy: stay. Your wrap(retry, breaker, timeout) is already order A. Steal breakwater's composition doc as a code-review checklist.
• On opossum and adding retry by hand: this is the migration that pays. new CircuitBreaker(fn, { timeout: 3000, errorThresholdPercentage: 50, resetTimeout: 30000 }) maps to resilience({ timeout: 3_000, circuitBreaker: { failureThreshold: 0.5, halfOpenAfter: 30_000 }, retry: { attempts: 3 } }) and you get backoff, a correlation ID, and error codes (CIRCUIT_OPEN, TIMEOUT, RETRY_EXHAUSTED) you can branch on instead of string-matching messages.
• Greenfield on Node 22+: try breakwater on a non-critical dependency first, pin the minor, and read docs/versioning.md — it's unusually clear about which interfaces you implement (StateStore, MetricsCollector) and how additions to them are versioned.

Two gotchas from the API: policy.wrap(fn) preserves fn's signature, which means the wrapped function never receives the combined AbortSignal — use execute(({ signal }) => …) when you need cancellation to reach the network call. And this isn't forwarded, so bind methods before wrapping.

The verdict

Breakwater's circuit breaker is table stakes. Its contribution is treating resilience as a pipeline with documented, defended ordering and one observability surface — and doing that with a semver promise and a test suite (21 files, Stryker mutation testing) that most weekend projects skip. Whether it survives past the one-maintainer stage is an open question. But the ordering argument stands on its own, and if your service currently runs retry inside its breaker, you have a bug regardless of which library you use to fix it.