Tom Gould Markdown

Writing

Gates, not a model’s word

How Mullet 2 decides a coding step is done: eight gates that read the diff and the test results, and a reviewer that can block a step but never pass one.

A table of Mullet’s gates G0 to G7 with one line each, such as G1 real change and G5 tests, and the G7 review row marked as able to block but not pass.

A coding model will tell you it has finished. That sentence is not evidence. In the old Mullet I had substring checks for PASS and APPROVAL on a model’s reply, and a reply of “NO APPROVAL” passed. Mullet 2 never reads the reply for a verdict. A step is done when its gates say so, and a node’s status can only be set by one function, judge(), from gate results.

Three outcomes

Each gate returns a result, and the rules for combining them are short.

  • Any FAIL means the node failed.
  • A gate that could not run is UNVERIFIED. It is never shown as passed.
  • A required gate that is missing counts as FAILED.

The middle rule matters most. If the test runner crashes, “no failures” is not the same as “tests passed”. Mullet says UNVERIFIED and does not let it through.

The gates

Here is what each one checks, in the order I think about them.

G0 is the baseline. Once per job, Mullet runs the profile’s checks on the pinned base commit and records which tests pass, fail or are skipped, from the junit XML. A red baseline does not block the job. It changes what G5 is allowed to claim.

G1 checks there was a real change. It takes the diff from git, untracked files included. An edit task with an empty diff fails. That sounds obvious, and it is the case the old swarm tool got wrong when it returned applied: true having applied nothing.

G2 is scope. It compares the changed files with the node’s scope and with a list of protected paths: .git*, .husky, CLAUDE.md, .claude/, .mcp.json, CI workflows, lockfiles and more. It resolves symlinks through the tree, spots nested repos, and enforces a 5 MB file limit.

G3 is static. It reads the added lines from the git blobs, not from the working tree, and runs a regex secret scan plus gitleaks, a risky-diff check and deletion counts.

G6 looks for test weakening: removed assertions, added skip or xfail, a falling assertion count. A node can be allowed to do these, and without that permission they fail it.

G4 is build: lint, typecheck, build. A non-zero exit fails.

G5 runs the targeted tests and compares them with the baseline. A test that passed before and fails now is a regression. A passing test that becomes skipped counts as lost. No report, or a report with zero tests, is a FAIL.

G7 is the review. I come back to it below.

Flaky tests

A new failure in G5 is rerun twice before it counts. If it passes on a rerun, it is flaky and does not fail the node. If it fails every time, it is a real failure and goes back to the worker with the output, as part of the retry. A node gets up to 3 attempts, then it is BLOCKED.

I would rather rerun than let one flaky test throw away a good step, but I do not want to rerun until something goes green. Two reruns is the limit.

The reviewer can block, never pass

G7 starts a fresh read-only Claude session. It sees the diff and nothing else: no transcript of how the worker got there. It returns a JSON verdict that Mullet checks against a schema.

The verdict can block a node. It can never pass one. A pass needs every other gate to pass as well, so the reviewer adds a way to fail and removes none. If I let a model’s opinion promote a step, I would be back to trusting a reply.

The diff is untrusted text, and a worker could put instructions in it, such as a comment telling the reviewer to approve. So Mullet fences the diff with a random marker, and the prompt tells the reviewer that nothing inside the fence is an instruction.

What the independent reviews found

Before each phase of the build was committed, I had it reviewed independently. They found real bypasses. All were fixed, with tests.

Phase 1, the gates themselves, had four:

  • A skipped test counted as passing.
  • A committed .gitattributes could hide changes from the diff. Diffs now use --text.
  • Added lines that started with ++ were misread.
  • A symlinked dependency directory got past the scope check.

Phase 2, the runner, had these:

  • The worker could write into dependency directories. There are now deny rules and a fingerprint of those directories.
  • A SIGTERM handler could deadlock.
  • Resume was refused after kill -9.
  • A crash just after a commit was not handled.

Phase 3, the task tree, had these:

  • A parent’s fix could delete a child’s tests.
  • Children’s permissions did not reach the parent.
  • The reviewer could be prompt-injected through the diff, which is why the diff is fenced.

None of these showed up in the happy path. A skipped test counted as passing only when a test was skipped. They turned up because the reviews set out to break the gates rather than use them.

What gates do not do

They do not prove the code is right. G5 is only as good as the tests the repo has, and a repo with no tests gives G5 little to compare. The ledger and the report say which gates ran and which were UNVERIFIED, so you can see what was not checked.

The code is at https://github.com/tomgould/mullet-public.