Cheap models only for the work that is safe to be wrong about

The delegation ladder hands each step of a job to the cheapest worker that can pass its checks. The whole idea is in that sentence’s last few words. Cheapest first is a good policy only if a wrong answer from a cheap model costs an attempt and nothing more. So the question the ladder really answers isn’t “which model is cheapest?” It’s “is this step safe to be wrong about?”

A worker’s “done” is worth nothing

In the first 101 attempts in Mullet’s ledger, workers said they were done 72 times, and 30 of those then failed a gate (we read 101 attempts before changing anything). That is why nothing in Mullet reads a worker’s reply for a verdict (gates, not a model’s word).

What makes a wrong attempt cheap is everything around the model. The worker edits a private clone, never my checkout (sandboxing an AI coding worker). Its change has to pass gates that read the diff and run the tests. A failure goes up the ladder with the gate output attached. A wrong attempt costs its tokens and some time, and that’s all.

That only holds when the gates can tell right from wrong for that particular step. So the ladder sorts steps by exactly that.

Three groups

The planner, which is always Claude, marks each step’s kind. The ladder then puts every step into one of three groups (step_group in mullet/jobs/ladder.py):

• Mechanical steps, such as adding a test file, updating the docs or wiring in a flag. Cheap rungs go first.
• Checkable steps: a design step with acceptance commands, a narrow scope (at most four scope globs by default, and no whole-tree glob such as **) and nothing risky in it. Its checks can judge it, so it may start below Claude.
• Claude-first steps: analysis, any other design step, a parent’s fix that integrates its children, and every one-way step.

On top of that sit two rules that can only raise risk. A step whose goal, acceptance or scope mentions code that deletes, kills or sends data is risky. The default words are delete, remove, rm, rmtree, unlink, kill, terminate, signal, wipe, truncate, drop, push, upload and send. And the planner can mark a step as a one-way door: a public API, a removal, a stored format. A plan can raise a step’s risk that way, but it can never lower it, because the words are checked whatever the plan says. By default, analysis steps never go to free or local models at all.

The settings that draw these lines are in the ladder: block. This is from the documented keys in ladder.py:

ladder:
  start:                                   # the cheapest rung each kind of step may use
    mechanical: cheapest                   # free, local, kimi, openrouter or claude
    design: cheapest
    analysis: kimi
  narrow_globs: 4                          # most scope globs a checkable design step may have
  risky_words: [delete, kill, send, ...]   # mark steps that always get the G7 review

Where Claude comes in

Which rung goes first also depends on how much of my Claude week is left, which the ladder post covers. The code is less simple than “Claude last”. When Claude’s week has room, Claude comes before the paid rungs even for mechanical steps, because the subscription is already paid for. When the week is tight, Claude is the last resort for every group, so even claude-first steps start on a cheap paid model.

That is why the review rule doesn’t depend on who wrote the code. A one-way step gets Claude’s independent review (G7) in every job, even one planned with --no-review, and in a ladder job so does every risky step, whichever rung wrote it. The reviewer can block a step but can never pass one, so the review is one more way to fail and never a way round the other gates (the reviewer that can only say no). On those forced reviews, a verdict that doesn’t block, given only after the reviewer had to be asked a second time, isn’t enough: the job pauses for me to look.

Wrong still costs money

A wrong attempt on a paid rung is cheap, not free. The limits are one envelope across all jobs, daily and weekly, plus caps for each provider. Each attempt gets an allowance from whatever room is left, and the key proxy enforces it while the attempt runs: once the allowance is spent it answers HTTP 402, and the step moves up the ladder. Nothing is limited until you set a value, and mullet doctor warns until you do. At the start of each session Mullet also asks me for a budget and a delegation choice (every session starts with one question).

For scale, a weekend of jobs cost about $50, checked against the providers’ own consoles (we checked the bill).

If you’re building the same thing

Don’t start with the model list. Start with the checks. A cheap model is usable for exactly the work your checks can judge, and nowhere else. If a step has no acceptance commands, or touches the whole tree, or deletes things, nothing would catch a bad attempt there except a careful reader, so it goes to the best reader you have, and gets reviewed whoever writes it.

And let risk only go up. Every input to the decision (the planner’s labels, the words in the step, the one-way flag) can make a step more careful, and none of them can make it less.

What I don’t know yet

Whether the lines are drawn in the right places. The ledger showed 6 attempts failing G3 for “risky edits in a mechanical step”: the planner called a step mechanical and the worker’s change was risky. The planner’s prompt has changed since (a task plan that knows what depends on what). Whether that moved the numbers needs new jobs to tell.

The code is at https://github.com/tomgould/mullet-public.