The reviewer that can only say no

G7 is the last of Mullet’s gates. Once G1 to G6 have passed, a fresh read-only Claude session looks at the step’s goal, scope and acceptance commands, the list of changed files and the diff, and it can read the clone. It never sees the worker’s transcript.

It can block a step. It can never pass one. It only runs once every other gate has passed, and a step needs every gate to pass, so the reviewer adds a way to fail and removes none. Gates, not a model’s word explains that rule and how the diff is fenced so a worker can’t write instructions to the reviewer into it. This post is about what changed in Mullet 2.3.0.

Why the reviewer came first

When I read the ledger’s 101 attempts, G7 was the first failing gate for 23 of the 40 failures. They were real defects that the tests hadn’t caught. Each block cost a whole attempt, and the next attempt got the reviewer’s reasons as free text.

So the reviewer wasn’t wrong too often. Its blocks were expensive and vague, and the work this round was to make each one something the next attempt could act on.

What it checks now

The reviewer blocks a step if the change doesn’t do the whole step (a stub or a partial version), if it does anything it wasn’t asked to, or if it contains anything harmful, such as a hidden network call. 2.3.0 adds three checks:

• New or changed behaviour has a test that would fail without the change. A refactor that keeps behaviour, or a step that changes no code, needs none.
• The edge cases that the step’s goal or acceptance commands name are handled.
• The new code has no secrets and no injection: no keys or passwords written into it, and no outside input passed unchecked to a shell, a query, eval or a path.

The first is the one I wanted most. A worker can make a test suite pass by adding code no test touches, and every other gate will be happy with that.

Block only for a defect you can name

The prompt says it in one line:

Block only for a specific defect you can name, never for style or taste.

A block for taste throws away an attempt and gives the next worker nothing to fix. A block for a named defect can be checked, by the next attempt and by me.

Findings

The verdict now carries findings beside the free-text reasons. A finding has a path, an optional line and the problem. The next attempt’s feedback gets the reasons first, so the feedback’s length cap can’t cut them off, then one line per finding as path:line: problem, each labelled as the reviewer’s words. There are at most 20 findings, of at most 280 characters each, so all of them fit.

A finding only counts if it is about a file the change touched. A finding on any other file is counted in the gate’s details and passed on as a reason marked “(file not changed)”. A path that leads out of the clone, or holds a control character, is dropped. A block whose findings all name unchanged files still blocks.

Paths turned out to be harder than that sounds. An early version dropped good findings because the reviewer named a file by its absolute path in the clone, or with a leading ./. An independent review of this round’s work caught it, and both forms now reduce to the repo-relative path.

The repo’s own standards, from the base commit

Most repos have rules a test can’t check. A repo profile can now name a Markdown file of them, in its stored profile or in the repo’s .mullet/project.yaml:

review_standards: docs/STANDARDS.md

The default is none. The file is read from the job’s base commit, never from the work tree or from a later step’s commit. A worker edits the repo, and if the standards came from the work tree, a worker could rewrite the rules its own reviewer applies. The test for this sets up exactly that: the base commit holds the real rule, then a later commit and a dirty work tree each replace it with a rule that tells the reviewer anything goes. The reviewer only ever sees the base commit’s version.

The standards are shown to the reviewer only, fenced with the call’s random marker as data, and capped at 20,000 characters. Workers don’t pay for them in their prompts. A missing, symlinked, binary or empty file is ignored, with a note in the gate’s details.

A missing verdict

Sometimes a review finishes its session without giving its JSON verdict. Before 2.3.0 that paused the whole job. Now a review that finished on its own is resumed once in the same session and asked only for its verdict:

jobs:
  reask_attempts: 1          # times to ask again, in the same session, for a reply without its structured part (0 turns it off)

A review cut off by its turn limit or its clock is never asked again. It may have stopped halfway through reading, and “answer now” would get a verdict on partial reading. It pauses the job, as before. That rule is there because the first version got it wrong, and the independent review found it.

A verdict that only came on a second asking is marked as such and trusted less. Where the review is forced for safety, on a one-way step or a risky step in a ladder job, a review that only stopped short of blocking when asked again does not let the step through: the job pauses for me. A block on a second asking still fails the attempt, since blocking is the safe answer.

What it doesn’t do

It doesn’t prove the code is right. It reads a diff capped at 120,000 characters, 20,000 for any one file, and reads the clone for the rest. It costs a Claude session per attempt. And it is a model reading a diff, so it will miss things, which is acceptable only because it can’t make anything pass.

The code is at https://github.com/tomgould/mullet-public.