ProofRun Wants Receipts for Your Coding Agent's Claims

Every developer running a coding agent has hit this moment: the agent announces "all tests pass," and you realize you have no idea whether that's a report or a prediction. Maybe it ran pytest two edits ago. Maybe it never ran it at all and is pattern-matching on "the change looks right." The words are identical either way.

ProofRun, a small Go CLI that surfaced on Hacker News this week, attacks that gap with a mechanism so unfashionable it's almost contrarian: no LLM anywhere. It runs your check as a real subprocess, records the real exit code, and binds the result to a fingerprint of your exact code state — git HEAD plus a SHA-256 hash of everything uncommitted, untracked files included. Change one byte and the stored PASS flips to STALE automatically. Results live in a signed local receipt.json, and proofrun status --strict exits non-zero unless every required check is a fresh, observed PASS. Four statuses — PASS, FAIL, STALE, NOT RUN — and no fifth "probably fine."

Let's be clear about scale before going further: this is a five-day-old repo with 12 stars and a Show HN that drew five points and zero comments. You should not adopt it as a trust root this week. But the design is worth your attention, because it's the inner-loop version of an idea that already won once.

Supply-chain attestation, shrunk to your working tree

Bind a claim to a content hash, sign it, and make verification mechanical rather than social — that's not new. It's the entire premise of software supply-chain security: in-toto attestations assert "this step ran on this artifact," SLSA provenance asserts "this binary came from this build on this source," and CI systems increasingly refuse to deploy without the paperwork. What ProofRun does is port that shape from the release pipeline down to the working tree, where the artifacts are your uncommitted diffs and the untrusted party isn't a compromised build server — it's the agent typing in your terminal.

There's a second lens that I think is even more useful: ProofRun is a build-cache key read backwards. Bazel and Nix hash inputs so they can skip work whose result is already known; ProofRun hashes inputs so it can invalidate a claim the moment its inputs drift. Same math, opposite purpose. And that reframing reveals what the real product is. It isn't the HMAC signature — it's STALE. The dominant failure mode with coding agents isn't malice, it's stale belief: the model genuinely did see tests pass, forty tool calls and three refactors ago, and neither it nor you noticed the evidence expired. Humans do the same thing before every "how did this break, tests were green" incident. A status that decays automatically when code changes is cache invalidation applied to trust, and it's the part of this design I'd bet survives.

The honest limits, and one leaky boundary

Credit where due: the README's threat-model section is more candid than most security products'. The signing is tamper-evident, not tamper-proof — the HMAC key sits in .proofrun/secret on the same machine, so any agent with shell access can read it and forge a receipt, or replay a genuinely signed one after reverting code. ProofRun says so plainly, and its GitHub Action accordingly trusts nothing local: it wipes .proofrun/ and re-runs the checks itself.

Which means the local receipt is not cryptographic assurance and shouldn't be sold as one. It's two humbler things: a freshness tracker, and a speed bump that catches an agent naively editing JSON to make red look green. Today's agents fail the naive way, so the speed bump has real value — but the boundary will leak the moment agents get better at inspecting their environment. Real assurance still requires an independent re-execution the agent can't touch, which is to say: CI, same as it ever was.

There's a good dogfooding story here too, with the obvious caveat that it's self-reported. The project is written by Claude Code, with every change adversarially reviewed by a second agent (Codex) before a human merges. That review process reportedly caught ProofRun's own verification being bypassable — a misquoted shell argument let a check run zero tests and still report PASS — which is both a point in the process's favor and a reminder of how sharp the edges are in exactly this kind of tool.

What you'd actually do with it

Adoption is one YAML file and a prompt change. You declare checks as argv arrays in .proofrun.yml (argv-exact on purpose — pytest -k "foo bar" can't be satisfied by a string that merely looks similar), then tell the agent in your CLAUDE.md or AGENTS.md to run tests through proofrun run test -- pytest instead of bare pytest. The gate is proofrun status --strict wired wherever you already gate: a pre-commit hook, a stop-hook in your agent harness, or CI via the Action.

The trade-offs are real. You're maintaining a parallel registry of commands that can drift from your Makefile. Argv-exact matching is brittle across environments. Pre-1.0 churn is live — v0.3 invalidated all earlier receipts. And on any repo with a decent test suite, "hash the diff, re-run everything on any change" is exactly the cost problem Bazel-style caching exists to solve; ProofRun tells you your evidence is stale but does nothing to make refreshing it cheap.

The deeper question is whether this should be a standalone binary at all. Harnesses are already growing verification organs — Claude Code hooks can force a test run before the agent declares victory, and agent-orchestration frameworks are converging on "don't trust the model's summary, check the exit code." Tellingly, an unrelated project with the identical name — rokabytedev/proofrun — appeared in the same window, chasing the same anxiety with the opposite mechanism: screenshots and human sign-off instead of hashes and exit codes. When two strangers independently ship "proof, not claims" tools under the same name, that's a space converging, not a coincidence.

My call: the tool is a maybe, the pattern is a yes. Expect receipts — a machine-checkable record of what ran, against which code state, with what result — to get absorbed into agent harnesses and CI within a year or two, the way provenance got absorbed into build systems. ProofRun's lasting contribution may be nothing more than its four-word insight: an agent's claim about its own work is a cache entry, and cache entries need invalidation.