Tom Gould Markdown JSON

We read 101 attempts before changing anything

Before changing Mullet’s planner or reviewer, I counted what had actually failed in its ledger. It wasn’t cheap models writing code that failed tests. It was a reviewer blocking real defects, steps sized wrong, and risky edits in steps labelled mechanical.

A table headed 101 attempts, 40 failed, listing the first gate each failed attempt failed: G7, the review, 23; G3, risky edits in a mechanical step, 6; G1, no files changed, 5; G2, edits outside scope, 4; G5, tests, 2. Below it: workers said done 72 times, and 30 of those failed a gate. A table headed 101 attempts, 40 failed, listing the first gate each failed attempt failed: G7, the review, 23; G3, risky edits in a mechanical step, 6; G1, no files changed, 5; G2, edits outside scope, 4; G5, tests, 2. Below it: workers said done 72 times, and 30 of those failed a gate.

Mullet 2.3.0, released on 26 October 2026, changes the planner, the reviewer and the way workers are stopped. Before I touched any of them, I read the ledger.

Every attempt Mullet makes at a step of a job goes into it: which worker made it, what the worker said at the end, which gates ran and what each one returned. When I started this round it held 101 attempts, and 40 of them had failed. I had a list of things I meant to fix. The ledger reordered it.

What failed first

A failed attempt often fails more than one gate, so I counted the first gate that failed. Gates, not a model’s word says what each gate checks.

First failing gate Attempts What it means
G7, the review 23 the reviewer blocked a defect it could name
G3, static checks 6 risky edits in a step the plan called mechanical
G1, real change 5 the worker changed no files
G2, scope 4 edits outside the files the step listed
G5, tests 2 the tests, compared with the baseline
All 40

That adds up: 23, 6, 5, 4 and 2 make 40.

I expected the tests to be where attempts died, with cheaper models writing code that didn’t work. Two attempts out of 40 first failed on tests. The biggest group by far was the reviewer, and the 23 blocks were real defects that the tests had not caught. The next group, 15 attempts across G1, G2 and G3, were as much about the plan as about the worker. A step called mechanical whose change turned out to be risky. A step whose scope didn’t list the new test file it needed. A step where the worker found nothing to change.

What that changed

The order of work changed first. The reviewer was the most expensive gate, because each block costs a whole attempt. The blocks were right, so making the reviewer quieter was not the answer. Making each block more useful to the next attempt was: findings by file and line, not a paragraph of prose. That is the reviewer that can only say no.

The planner came second. G1, G2 and G3 are the gates where the shape of a step matters most, so the planner now has to say which steps come after which, list any new test files in a step’s scope, and never call a step mechanical if it deletes, kills, sends data or changes a public API or stored format. That is a task plan that knows what depends on what.

The decision records for the planner and reviewer changes quote these same figures as their context. I’d rather a design decision cite a count than a feeling, and decision records for a solo project is about why I keep them at all.

What the workers said

A worker ends its attempt with a short structured reply, and the status in it is one of three words: done, blocked or split. Here is what the 101 replies said.

Reply Count
done 72
blocked 0
split 0
off-protocol 14
no claim at all 15

Of the 72 that said done, 30 then failed a gate. So the other 10 failures came from replies that said something off-protocol or nothing.

A worker saying it has finished counts for nothing in Mullet. Status comes from gate results and exit codes, and this is the number that justifies that rule: 30 of 72 is getting on for half.

The idea I dropped

I had planned to act on blocked and split. A worker that said a step was too big would get it split into smaller steps, and one that said it was blocked would stop the step and ask me. It is a reasonable design, and the reply format already offered both words.

The ledger said it would never run. No worker answered blocked, and none answered split, in 101 tries. Workers say done, or something off-protocol, or nothing. Building and testing a path that no worker takes would have bought nothing, so I dropped it.

What I kept is the record. Each reply is now written to the ledger as a worker_claim event, so if a worker ever does say it’s blocked, I’ll see it in the counts. The claim also goes into the step’s commit message, quoted and marked as not verified. Stop the silent worker, but let a finished one leave shows what that looks like.

Why the ledger made this possible

It records every attempt, not just the ones that passed. A failed attempt is the useful one: it says which gate stopped it, and why. Without that I’d have tuned the planner on a guess.

2.3.0 makes those failures easier to read. mullet ladder report gains a “Why attempts failed” section, with the first failing gate, how many attempts it stopped and its most common reason, for ladder jobs. A single failed attempt can be looked at, and its work copied out as a branch if it was nearly right:

mullet job show <job> --attempt N --diff
mullet job export <job> --attempt N --branch NAME

What I don’t know yet

These counts come from the jobs I’ve run so far, and 101 attempts is a small sample. They are a snapshot of one point in time, not a rate. Whether the planner changes actually reduce the G1, G2 and G3 failures needs new figures from new jobs, and I’ll only know once those have run. The same goes for the reviewer: if the findings work, the next attempt after a block should pass more often, and the ledger will say whether it does.

The code is at https://github.com/tomgould/mullet-public.

More on Mullet

More writing