--- title: 'We read 101 attempts before changing anything' canonical: 'https://www.dealmobile.co.uk/writing/we-read-101-attempts-before-changing-anything' url: 'https://www.dealmobile.co.uk/writing/we-read-101-attempts-before-changing-anything' type: Post published: '2026-10-26' summary: 'Before changing Mullet’s planner or reviewer, I counted what had actually failed in its ledger. It wasn’t cheap models writing code that failed tests. It was a reviewer blocking real defects, steps sized wrong, and risky edits in steps labelled mechanical.' --- # We read 101 attempts before changing anything Published 2026-10-26 ![A table headed 101 attempts, 40 failed, listing the first gate each failed attempt failed: G7, the review, 23; G3, risky edits in a mechanical step, 6; G1, no files changed, 5; G2, edits outside scope, 4; G5, tests, 2. Below it: workers said done 72 times, and 30 of those failed a gate.](https://www.dealmobile.co.uk/sites/default/files/styles/almanac_shot_md/public/content/images/posts/we-read-101-attempts-before-changing-anything/hero.png.webp?itok=QcTvqp6H) Mullet 2.3.0, released on 26 October 2026, changes the planner, the reviewer and the way workers are stopped. Before I touched any of them, I read the ledger. Every attempt Mullet makes at a step of a job goes into it: which worker made it, what the worker said at the end, which gates ran and what each one returned. When I started this round it held 101 attempts, and 40 of them had failed. I had a list of things I meant to fix. The ledger reordered it. ## What failed first A failed attempt often fails more than one gate, so I counted the first gate that failed. [Gates, not a model’s word](https://www.dealmobile.co.uk/writing/gates-not-a-models-word.md) says what each gate checks. | First failing gate | Attempts | What it means | |:--|:--|:--| | G7, the review | 23 | the reviewer blocked a defect it could name | | G3, static checks | 6 | risky edits in a step the plan called mechanical | | G1, real change | 5 | the worker changed no files | | G2, scope | 4 | edits outside the files the step listed | | G5, tests | 2 | the tests, compared with the baseline | | All | 40 | | That adds up: 23, 6, 5, 4 and 2 make 40. I expected the tests to be where attempts died, with cheaper models writing code that didn’t work. Two attempts out of 40 first failed on tests. The biggest group by far was the reviewer, and the 23 blocks were real defects that the tests had not caught. The next group, 15 attempts across G1, G2 and G3, were as much about the plan as about the worker. A step called mechanical whose change turned out to be risky. A step whose scope didn’t list the new test file it needed. A step where the worker found nothing to change. ## What that changed The order of work changed first. The reviewer was the most expensive gate, because each block costs a whole attempt. The blocks were right, so making the reviewer quieter was not the answer. Making each block more useful to the next attempt was: findings by file and line, not a paragraph of prose. That is [the reviewer that can only say no](https://www.dealmobile.co.uk/writing/the-reviewer-that-can-only-say-no.md). The planner came second. G1, G2 and G3 are the gates where the shape of a step matters most, so the planner now has to say which steps come after which, list any new test files in a step’s scope, and never call a step mechanical if it deletes, kills, sends data or changes a public API or stored format. That is [a task plan that knows what depends on what](https://www.dealmobile.co.uk/writing/a-task-plan-that-knows-what-depends-on-what.md). The decision records for the planner and reviewer changes quote these same figures as their context. I’d rather a design decision cite a count than a feeling, and [decision records for a solo project](https://www.dealmobile.co.uk/writing/decision-records-for-a-solo-project.md) is about why I keep them at all. ## What the workers said A worker ends its attempt with a short structured reply, and the status in it is one of three words: done, blocked or split. Here is what the 101 replies said. | Reply | Count | |:--|:--| | done | 72 | | blocked | 0 | | split | 0 | | off-protocol | 14 | | no claim at all | 15 | Of the 72 that said done, 30 then failed a gate. So the other 10 failures came from replies that said something off-protocol or nothing. A worker saying it has finished counts for nothing in Mullet. Status comes from gate results and exit codes, and this is the number that justifies that rule: 30 of 72 is getting on for half. ## The idea I dropped I had planned to act on blocked and split. A worker that said a step was too big would get it split into smaller steps, and one that said it was blocked would stop the step and ask me. It is a reasonable design, and the reply format already offered both words. The ledger said it would never run. No worker answered blocked, and none answered split, in 101 tries. Workers say done, or something off-protocol, or nothing. Building and testing a path that no worker takes would have bought nothing, so I dropped it. What I kept is the record. Each reply is now written to the ledger as a `worker_claim` event, so if a worker ever does say it’s blocked, I’ll see it in the counts. The claim also goes into the step’s commit message, quoted and marked as not verified. [Stop the silent worker, but let a finished one leave](https://www.dealmobile.co.uk/writing/stop-the-silent-worker-but-let-a-finished-one-leave.md) shows what that looks like. ## Why the ledger made this possible It records every attempt, not just the ones that passed. A failed attempt is the useful one: it says which gate stopped it, and why. Without that I’d have tuned the planner on a guess. 2.3.0 makes those failures easier to read. `mullet ladder report` gains a “Why attempts failed” section, with the first failing gate, how many attempts it stopped and its most common reason, for ladder jobs. A single failed attempt can be looked at, and its work copied out as a branch if it was nearly right: ```bash mullet job show --attempt N --diff mullet job export --attempt N --branch NAME ``` ## What I don’t know yet These counts come from the jobs I’ve run so far, and 101 attempts is a small sample. They are a snapshot of one point in time, not a rate. Whether the planner changes actually reduce the G1, G2 and G3 failures needs new figures from new jobs, and I’ll only know once those have run. The same goes for the reviewer: if the findings work, the next attempt after a block should pass more often, and the ledger will say whether it does. The code is at . - **Projects:** [Mullet](https://www.dealmobile.co.uk/projects/mullet.md) - **Tags:** [Python](https://www.dealmobile.co.uk/writing/tag/python.md), [AI](https://www.dealmobile.co.uk/writing/tag/ai.md), [Mullet](https://www.dealmobile.co.uk/writing/tag/mullet.md) --- Index of this site: [llms.txt](https://www.dealmobile.co.uk/llms.txt)