Mutation Testing Finds What 100% Coverage Hides

Line coverage answers one question: did this line execute while some test was running? It says nothing about whether any assertion cared. That gap is old news to testing researchers, but it's worth restating every time a well-run project demonstrates it in public, and breakwater just did.

Breakwater is a small Node.js resilience toolkit (retry, circuit breaker, timeout, policy composition) with 100% line and branch coverage. Its author, Pedro Rogério, pointed StrykerJS at it and got 1,114 mutants, of which 199 survived. An initial mutation score of 80.52%. Among the survivors: a real bug in how named policies propagated into their rate limiter, so the same policy showed up on a metrics dashboard as two separate series. Nobody had reported it. The fix shipped to npm the same week.

The interesting part isn't that mutation testing found a bug. It's the shape of what it found, and what that says about how you should actually run the tool.

What the survivors were

Mutation testing flips operators, deletes statements, swaps < for <=, and reruns your tests. A mutant that survives means your suite can't tell the difference between the code you wrote and a corrupted version of it. Rogério's 199 survivors sorted into five buckets, and only one of them was the production defect.

The most instructive survivor was a test that lied. It carried a comment saying it reproduced a floating-point case where a naive ceil() lands one millisecond short in retryAfterMs. The configuration it used (interval: 8737) no longer triggered that correction path at all. The test passed with or without the code it claimed to protect. Coverage reported the line as exercised, because it was; the assertion just didn't depend on it. He found a value that actually hits the rounding error (interval: 161, since 161 * (1/161) lands at 0.999…) and rewrote the test around it.

That's the failure mode I'd bet is most common in mature suites. Not missing tests, but tests that drifted away from their purpose during refactors and kept passing. Coverage can't detect drift by construction. A mutation run detects it in minutes.

The largest bucket was documented behavior with no test pinning it: circuit breaker event emission, distributed state-store paths, boundary semantics on limits, jitter. Then two pieces of dead code (unreachable guards in timer callbacks), and finally equivalent mutants that Stryker can't distinguish from the original, like removing { once: true } from an abort listener. The end state after triage: 223 tests (up from 157) and a 95.11% score, with the remaining survivors individually justified rather than chased.

This is a code-review tool, not a merge gate

Rogério keeps the mutation run as a manual npm run test:mutation step rather than a CI job, and I think that's right for a library of this size. It also matches the most credible large-scale data on the practice. Google's mutation testing system, described in Petrović and Ivanković's "Practical Mutation Testing at Scale", runs on changed lines during code review, used by more than 24,000 developers across 1,000+ projects. Their early finding was brutal: developers rated 85% of raw mutants as unproductive (trivially equivalent, or exposing something not worth testing). Only after aggressive suppression heuristics and per-line caps did the productive ratio climb to 89%. The lesson they drew is the one breakwater's triage rediscovered: the score matters less than the per-mutant judgement, and the tool earns its keep by surfacing a handful of things a reviewer can act on.

That reframes the "don't chase 100%" advice. Killing an equivalent mutant means writing a test that asserts an implementation detail, which is a regression in suite quality, not an improvement. A mutation score of 80–95% on meaningful code, with every survivor either fixed or annotated, is the finish line. Loiane Groner's recent Angular write-up reached the same conclusion from the other direction: a pipe with 100% coverage scored 62.5% on mutation, and her recommended config sets break at 50 rather than anywhere near the top.

How to run it without it eating your CI budget

The historical objection to mutation testing is cost. Every mutant is a test run, and the technique dates to the 1970s precisely because it was too expensive to use back then. Three things changed that, and they're all in StrykerJS today.

First, coverageAnalysis: "perTest" records which tests cover which mutant during the initial run and executes only those tests per mutant. That's what got breakwater's full run under six minutes on a 1,114-mutant codebase. Without it (the "off" setting), Stryker runs the entire suite for every mutant.

Second, --incremental. Stryker diffs your source and test files against reports/stryker-incremental.json from the last run and re-executes only the mutants whose outcome could have changed. Cache that file as a CI artifact and a typical PR's mutation run shrinks to the mutants in the changed lines, which is exactly the Google model. Know the limitations: it doesn't notice dependency upgrades, snapshot changes, or environment differences, so pair it with a scheduled full run using --force.

Third, Stryker 10.0.0, released August 14, dropped Node 20 support, added an empty-expression mutator, a mutant-filtering layer to keep that new mutator from generating redundant mutants, and saving a partial incremental report when a run dies unexpectedly. That last one matters for CI: a killed job used to mean a wasted run.

A reasonable starting config for a TypeScript project on node:test looks like breakwater's:

{
  "testRunner": "tap",
  "tap": {
    "testFiles": ["tests/**/*.test.ts"],
    "nodeArgs": ["--import", "tsx", "--test", "--test-reporter=tap"]
  },
  "coverageAnalysis": "perTest",
  "timeoutMS": 20000,
  "thresholds": { "high": 80, "low": 60, "break": 50 }
}

Vitest, Jest, Mocha, Jasmine, and Karma runners exist too. Outside JavaScript, PIT has been the production-grade option for the JVM for over a decade, Stryker.NET covers C#, and mutmut and cargo-mutants serve Python and Rust.

Where this goes

The next step is already visible in the research. Meta's engineering team has published on mutation-guided LLM test generation, where mutants become the specification for what a generated test must catch. That's a more useful target than "raise coverage," which an LLM can satisfy by writing tests that assert nothing, precisely the lying-test pattern above. Expect mutation scores, not coverage percentages, to become the metric that AI-authored test suites are judged by.

For now the practical move is smaller. Run Stryker once against your best-covered module, sort survivors by file, and look at the first ten. If none of them is a test that quietly stopped testing anything, you have a better suite than most.