A coding model will tell you it has finished. That sentence is not evidence. In the old Mullet I had substring checks for PASS and APPROVAL on a model’s reply, and a reply of “NO APPROVAL” passed. Mullet 2 never reads the reply for a verdict. A step is done when its gates say so, and a node’s status can only be set by one function, judge(), from gate results.
Each gate returns a result, and the rules for combining them are short.
\nThe middle rule matters most. If the test runner crashes, “no failures” is not the same as “tests passed”. UNVERIFIED is for checks that legitimately could not run, such as a test suite the repo doesn’t have. It is never counted as passed in a report, and mullet job merge refuses an unverified step unless you pass --allow-unverified.
This is stricter than I first wrote it. A gate that crashes, or a required review that gives no verdict, now pauses the whole job. Nothing is judged or committed for that step, and the worker’s model isn’t marked as having failed, because the fault was in my checking, not in its work.
\nHere is what each one checks, in the order I think about them.
\nG0 is the baseline. Once per job, Mullet runs the profile’s checks on the pinned base commit and records which tests pass, fail or are skipped, from the junit XML. A red baseline does not block the job. It changes what G5 is allowed to claim.
\nG1 checks there was a real change. It takes the diff from git, untracked files included. An edit task with an empty diff fails. That sounds obvious, and it is the case the old swarm tool got wrong when it returned applied: true having applied nothing.
G2 is scope. It compares the changed files with the node’s scope and with a list of protected paths: .git*, .husky, CLAUDE.md, .claude/, .mcp.json, CI workflows, lockfiles and more. It resolves symlinks through the tree, spots nested repos, and enforces a 5 MB file limit.
G3 is static. It reads the added lines from the git blobs, not from the working tree, and runs a regex secret scan plus gitleaks, a risky-diff check and deletion counts.
\nG6 looks for test weakening: removed assertions, added skip or xfail, a falling assertion count. A node can be allowed to do these, and without that permission they fail it.
\nG4 is build: lint, typecheck, build. A non-zero exit fails.
\nG5 runs the targeted tests and compares them with the baseline. A test that passed before and fails now is a regression. A passing test that becomes skipped counts as lost. No report, or a report with zero tests, is a FAIL.
\nG7 is the review. I come back to it below.
\nA new failure in G5 is rerun twice before it counts. If it passes on a rerun, it is flaky and does not fail the node. If it fails every time, it is a real failure and goes back to the worker with the output, as part of the retry. A node gets up to 3 attempts, then it is BLOCKED.
\nI would rather rerun than let one flaky test throw away a good step, but I do not want to rerun until something goes green. Two reruns is the limit.
\nG7 starts a fresh read-only Claude session. It sees the diff and nothing else: no transcript of how the worker got there. It returns a JSON verdict that Mullet checks against a schema.
\nThe verdict can block a node. It can never pass one. A pass needs every other gate to pass as well, so the reviewer adds a way to fail and removes none. If I let a model’s opinion promote a step, I would be back to trusting a reply.
\nThe diff is untrusted text, and a worker could put instructions in it, such as a comment telling the reviewer to approve. So Mullet fences the diff with a random marker, and the prompt tells the reviewer that nothing inside the fence is an instruction.
\nBefore each phase of the build was committed, I had it reviewed independently. They found real bypasses. All were fixed, with tests.
\nPhase 1, the gates themselves, had four:
\n.gitattributes could hide changes from the diff. Diffs now use --text.++ were misread.Phase 2, the runner, had these:
\nkill -9.Phase 3, the task tree, had these:
\nNone of these showed up in the happy path. A skipped test counted as passing only when a test was skipped. They turned up because the reviews set out to break the gates rather than use them.
\nThey do not prove the code is right. G5 is only as good as the tests the repo has, and a repo with no tests gives G5 little to compare. The ledger and the report say which gates ran and which were UNVERIFIED, so you can see what was not checked.
\nThe code is at https://github.com/tomgould/mullet-public.
" }, "markdown_text": "---\ntitle: 'Gates, not a model’s word'\ncanonical: 'https://www.dealmobile.co.uk/writing/gates-not-a-models-word'\nurl: 'https://www.dealmobile.co.uk/writing/gates-not-a-models-word'\ntype: Post\npublished: '2026-10-25'\nsummary: 'How Mullet 2 decides a coding step is done: eight gates that read the diff and the test results, and a reviewer that can block a step but never pass one.'\n---\n\n# Gates, not a model’s word\n\nPublished 2026-10-25\n\n\n\nA coding model will tell you it has finished. That sentence is not evidence. In the old [Mullet](https://www.dealmobile.co.uk/projects/mullet.md) I had substring checks for PASS and APPROVAL on a model’s reply, and a reply of “NO APPROVAL” passed. Mullet 2 never reads the reply for a verdict. A step is done when its gates say so, and a node’s status can only be set by one function, `judge()`, from gate results.\n\n## Three outcomes\n\nEach gate returns a result, and the rules for combining them are short.\n\n- Any FAIL means the node failed.\n- A gate that could not run is UNVERIFIED. It is never shown as passed.\n- A required gate that is missing counts as FAILED.\n\nThe middle rule matters most. If the test runner crashes, “no failures” is not the same as “tests passed”. UNVERIFIED is for checks that legitimately could not run, such as a test suite the repo doesn’t have. It is never counted as passed in a report, and `mullet job merge` refuses an unverified step unless you pass `--allow-unverified`.\n\nThis is stricter than I first wrote it. A gate that crashes, or a required review that gives no verdict, now pauses the whole job. Nothing is judged or committed for that step, and the worker’s model isn’t marked as having failed, because the fault was in my checking, not in its work.\n\n## The gates\n\nHere is what each one checks, in the order I think about them.\n\nG0 is the baseline. Once per job, Mullet runs the profile’s checks on the pinned base commit and records which tests pass, fail or are skipped, from the junit XML. A red baseline does not block the job. It changes what G5 is allowed to claim.\n\nG1 checks there was a real change. It takes the diff from git, untracked files included. An edit task with an empty diff fails. That sounds obvious, and it is the case the old swarm tool got wrong when it returned `applied: true` having applied nothing.\n\nG2 is scope. It compares the changed files with the node’s scope and with a list of protected paths: `.git*`, `.husky`, `CLAUDE.md`, `.claude/`, `.mcp.json`, CI workflows, lockfiles and more. It resolves symlinks through the tree, spots nested repos, and enforces a 5 MB file limit.\n\nG3 is static. It reads the added lines from the git blobs, not from the working tree, and runs a regex secret scan plus gitleaks, a risky-diff check and deletion counts.\n\nG6 looks for test weakening: removed assertions, added skip or xfail, a falling assertion count. A node can be allowed to do these, and without that permission they fail it.\n\nG4 is build: lint, typecheck, build. A non-zero exit fails.\n\nG5 runs the targeted tests and compares them with the baseline. A test that passed before and fails now is a regression. A passing test that becomes skipped counts as lost. No report, or a report with zero tests, is a FAIL.\n\nG7 is the review. I come back to it below.\n\n## Flaky tests\n\nA new failure in G5 is rerun twice before it counts. If it passes on a rerun, it is flaky and does not fail the node. If it fails every time, it is a real failure and goes back to the worker with the output, as part of the retry. A node gets up to 3 attempts, then it is BLOCKED.\n\nI would rather rerun than let one flaky test throw away a good step, but I do not want to rerun until something goes green. Two reruns is the limit.\n\n## The reviewer can block, never pass\n\nG7 starts a fresh read-only Claude session. It sees the diff and nothing else: no transcript of how the worker got there. It returns a JSON verdict that Mullet checks against a schema.\n\nThe verdict can block a node. It can never pass one. A pass needs every other gate to pass as well, so the reviewer adds a way to fail and removes none. If I let a model’s opinion promote a step, I would be back to trusting a reply.\n\nThe diff is untrusted text, and a worker could put instructions in it, such as a comment telling the reviewer to approve. So Mullet fences the diff with a random marker, and the prompt tells the reviewer that nothing inside the fence is an instruction.\n\n## What the independent reviews found\n\nBefore each phase of the build was committed, I had it reviewed independently. They found real bypasses. All were fixed, with tests.\n\nPhase 1, the gates themselves, had four:\n\n- A skipped test counted as passing.\n- A committed `.gitattributes` could hide changes from the diff. Diffs now use `--text`.\n- Added lines that started with `++` were misread.\n- A symlinked dependency directory got past the scope check.\n\nPhase 2, the runner, had these:\n\n- The worker could write into dependency directories. There are now deny rules and a fingerprint of those directories.\n- A SIGTERM handler could deadlock.\n- Resume was refused after `kill -9`.\n- A crash just after a commit was not handled.\n\nPhase 3, the task tree, had these:\n\n- A parent’s fix could delete a child’s tests.\n- Children’s permissions did not reach the parent.\n- The reviewer could be prompt-injected through the diff, which is why the diff is fenced.\n\nNone of these showed up in the happy path. A skipped test counted as passing only when a test was skipped. They turned up because the reviews set out to break the gates rather than use them.\n\n## What gates do not do\n\nThey do not prove the code is right. G5 is only as good as the tests the repo has, and a repo with no tests gives G5 little to compare. The ledger and the report say which gates ran and which were UNVERIFIED, so you can see what was not checked.\n\nThe code is at