Tom Gould

Writing

Checking a delegate’s edits before trusting them

How Mullet decides what a cheaper model actually changed, and flags the edits worth a second look, without asking another model and without trusting the first one’s word.

Mullet hands coding tasks to cheaper models. When a task means editing files, it goes to opencode, an open-source coding agent, running whichever model Mullet picked, such as DeepSeek on OpenRouter or Kimi. A local model in LM Studio gets a small agent loop of Mullet’s own instead. At the end the model says it has finished, and usually it has. Sometimes it has also changed a string it was never asked to touch, turned curly quotes into straight ones, or made a failing test pass by changing the number it expects. A cheap model saying “done” isn’t evidence, so Mullet checks the work before it hands the result back. This is how.

Done means a clean exit

Mullet runs opencode run with JSON output and reads its events line by line. Text events become the response, and the token counts and costs of each step are added up. An error event ends the attempt at once, and if nothing arrives for 240 seconds, the process is killed and the attempt fails.

The run only counts as complete when opencode exits with code 0. What the model wrote in its last message doesn’t come into it. A failed attempt gets one retry: on whichever of three candidate models has the largest context window if the context overflowed, or one step up the chain otherwise. The local agent loop stops after 30 iterations, and a run that doesn’t finish there goes to opencode with a cloud model.

The files come from git, not from the model

Before the run, Mullet takes git status --short in the project, and afterwards it takes it again. The changed files are the paths in the second list that weren’t in the first:

after_status = _git_status_snapshot(cwd_resolved)
files_changed = (
    sorted(after_status - before_status)
    if before_status is not None and after_status is not None
    else None
)

So a stray edit to a config file shows up in the result even if the model never mentions it. It compares paths, not contents, and that leaves two gaps. A file that already had uncommitted changes isn’t listed if the delegate edits it again, and if the delegate commits its work, the status is clean and nothing is listed. When git isn’t available the list is None rather than empty, so “nothing changed” and “couldn’t tell” stay different.

A classifier with no model in it

When a run completes with changed files, each one goes through a classifier in mullet/core/verification/diff_risk.py. It compares the file at HEAD with the file on disk using Python’s difflib, and looks only at places where old lines were replaced by new ones. Pure additions and deletions aren’t scored, so a brand-new file never raises a flag. Within a replaced block, it pairs the old and new lines in order and checks each pair for:

  • a string or number literal that changed
  • a straight quote within two characters of where a curly quote used to be
  • a new if (, while (, &&, ||, ?? or ternary ?
  • a TypeScript non-null assertion, !, straight after a literal
  • a comment whose text changed
  • a call to the same function with different arguments, once whitespace, casts and type annotations are stripped

The last one is reported as a changed test assertion instead when the line looks like a test, with assert in it or a call such as expect or toBe. The quote check is typical of the rest:

def _check_quote_flattening(old_line: str, new_line: str) -> bool:
    for i, ch in enumerate(old_line):
        if ch not in _CURLY_QUOTES:
            continue
        window = range(max(0, i - 2), min(len(new_line), i + 3))
        for j in window:
            if new_line[j] in _STRAIGHT_QUOTES:
                return True
    return False

Each flag records the file, the line number in the new file, the kind of flag, the old and new lines and a short reason. Changing expect(items).toHaveLength(2) to toHaveLength(1) raises two: a number changed from 2 to 1, and a changed test assertion. Changing "Don’t" to "Don't" in a label raises a changed string and a flattened quote.

Some edits are meant to pass without a flag, and they do, with tests to keep it that way: a change of indentation, a type annotation added to a parameter, as const after a number, a comment added or removed, and a ! after a variable name.

Why flags, and why no model

Every check is a regular expression or a string comparison, so it costs nothing and gives the same answer every time. The obvious alternative was to ask another model to review the diff. That costs money on every edit, and the reviewer can make the same kinds of mistake as the model it’s reviewing.

The other choice was what to do with a flag. The checks are blunt, and plenty of flagged edits are exactly what was asked for: “fix the typo in the error message” has to change a string. So nothing is blocked or rolled back. The flags go back in the result as risk_flags, next to files_changed, and the calling agent decides which lines to read before it trusts the run. The README’s advice is to review the changes with git diff and commit them yourself.

The classifier is also wrapped so that it can’t break a run. If it throws an exception, Mullet logs a warning and returns an empty list of flags. That keeps edits flowing, but it means an empty list can mean either that nothing looked risky or that the check failed, and only the log says which.

What it misses

The checks were written with JavaScript and TypeScript in mind. Comments are found by // and /* */, so a changed # comment in Python goes unnoticed, and so does new logic written with Python’s and, or or if x:. A Python assert whose comparison changes, say from == to >=, raises nothing at all.

Pairing lines in order is crude too. If a delegate adds a new first argument to log('a', 'b'), the classifier compares 'a' with the new argument and reports a changed string that didn’t change. A delegate that rewrites a whole function, rather than editing a few lines, can produce pairs of lines that have nothing to do with each other.

None of that makes the flags useless. They’re a short list of places to look, produced for free, where the alternative is reading every diff in full or taking a one-line “done” on trust. A changed test expectation is the edit I’d least want to miss, because it makes a failing test pass without fixing anything, and that check works in any language whose assertions are calls with literals in them.

The code is on GitHub.