Picking the cheapest model that can do the job

Mullet sends each coding task to the cheapest model that can handle it. “Cheapest” is a lookup in a price table. “Can handle it” is the hard part, and it’s where most of the routing code goes. This post follows a task from the first estimate to the model, and on to what happens when that model has gone.

Delegation is switched off on my machine for now, for the reasons on the product page, so this describes the code in the public copy. Its model names and prices date from October 2026, and they go stale quickly.

An estimate first

Before anything is picked, a preflight step in mullet/core/router/preflight.py turns the task into four facts: how many tokens it will read, how many it’s expected to write, a complexity tier (LOW, STANDARD, ADVANCED or CRITICAL), and whether it needs step-by-step reasoning.

The input size is counted locally. The other three come from a model, with a prompt that wants nothing back but JSON:

_META_PROMPT = """Analyze this engineering task. Reply with ONLY a JSON object, no prose, no markdown:
{{"expected_output_tokens": <integer 50-32000>, "complexity": "LOW|STANDARD|ADVANCED|CRITICAL", "needs_reasoning": <true|false>, "confidence": <0.0-1.0>}}

Task:
<<<{task}>>>"""

It asks a local model first, if one is running, then Gemini Flash, unless the task is in economy mode. The reply is checked rather than trusted: the output estimate is clamped to between 50 and 32,000 tokens, an unknown tier becomes STANDARD, and a reply with no JSON in it counts as no reply. If neither model answers, a heuristic takes over. Short, plain messages skip all of this and go straight to LOW, and every estimate is cached on disk for seven days.

The heuristic

The heuristic is the scoring the product page describes. Keywords, file paths, subtasks, the domain and how much of the system a task touches add points, and the total falls into a tier at 30, 76 and 90. “Redesign the auth architecture across all services, migrate production data and debug the token race condition” scores 85: 20 for architecture words, 15 for debugging, 20 for risk words, 20 for security and 10 for touching a module. That puts it in ADVANCED, five points short of CRITICAL.

It’s crude. “Add a retry with exponential backoff to the HTTP client” scores 0, because none of its words are on any list. That’s why it comes last, and why its estimates carry a confidence of 0.5, where a model’s answer carries the model’s own figure.

The cheapest model above a floor

Which provider gets the task depends on the budget and the active preset. Once that’s settled, routing_matrix.py picks the model. Each provider used to have a hand-written ladder: reasoning tasks to this model, critical tasks to that one, everything else to a cheap default. The ladders are still there, but a value picker in value_picker.py now runs first, and a ladder only decides when the picker comes back empty.

The picker needs to know how good each model is. quality.py gives each one three scores out of 100, for coding, reasoning and general work. The coding score averages SWE-Bench Pro and Terminal-Bench results where both exist, and each score records its source and whether it was measured, inherited from a sibling model or estimated.

Each tier sets a floor. For coding it’s 10 at LOW, 18 at STANDARD, 54 at ADVANCED and 65 at CRITICAL; the bottom two are low on purpose, because those tiers want the cheapest model that will do. The picker keeps the models whose context window fits the task, with a 10% margin because Mullet’s token count and the provider’s rarely agree, and only reasoning models if the task needs one. Then it drops anything below the floor and sorts the rest by price:

def _sort_key(item: tuple[str, int]) -> tuple[bool, float, str]:
    model_id, _ctx = item
    cost = _blended_cost(model_id)
    is_zero_cost = cost == 0.0
    # zero/missing pricing sorts LAST (worst), not first -- an unpriced
    # model must never win the cost tie-break just because _pricing_for's
    # fail-open default looks like "free". Still selectable if it's the
    # only survivor.
    return (is_zero_cost, cost, model_id)

The cost is the average of the input and output prices. If nothing clears the floor, the picker tries the next tier up; if that fails too, it returns nothing and the ladder decides.

With the October prices and scores, a coding task without reasoning goes to DeepSeek V4 Flash on OpenRouter at LOW, STANDARD and ADVANCED alike. It scores 59 for coding, at $0.09 per million input tokens and $0.18 per million output. OpenRouter’s ladder still names DeepSeek V4 Pro for advanced work, but it never gets asked: Flash clears the floor of 54 and costs about a fifth as much. A task that needs reasoning skips the picker on OpenRouter and, in the default mode, goes to V4 Pro, the only model on the list tuned for reasoning.

Kimi shows the other path. Its two models have only estimated scores, 35 and 25, so at ADVANCED neither clears the floor, nor CRITICAL’s above it. The picker returns nothing, and Kimi’s ladder sends the task to K2.7 Code.

When the model has gone

In July, LOW work on OpenRouter got a step of its own: two free models, qwen/qwen3-coder:free and tencent/hy3:free, tried before the paid list. They had a list of their own because of the sort above, where a free model would always come last. By 20 October OpenRouter had withdrawn both. A request for the first came back as a 404, “This model is unavailable for free. The paid version is available now - use this slug instead: qwen/qwen3-coder”, and until the fix, that error failed the task.

The fix has three parts. First, model_availability.py decides whether an error is about the model rather than the request. It matches OpenRouter’s wording for a withdrawn or unknown model, and any 404 that mentions the model, but never a rate limit or a server error, which are retried elsewhere:

if isinstance(status, int) and (status == 429 or status >= 500):
    return False
message = str(exc)
if _UNAVAILABLE_PATTERNS.search(message):
    return True
return status == 404 and "model" in message.lower()

Second, the model is written to ~/.mullet/unavailable_models.json with the time, and the routers leave recorded models out of their candidate lists. Entries expire after a day, so a model that comes back is tried again without anyone clearing a cache.

Third, the request doesn’t fail. OpenRouterProvider catches the error, records the model, picks a replacement and sends the same request again:

tried: list[str] = []
while True:
    try:
        return self._create_once(
            messages, tools, max_tokens, temperature, stream, reasoning_config,
        )
    except Exception as exc:
        if not self._switch_from_unavailable_model(exc, tried):
            raise

The replacement comes from the same value picker, at the tier the MCP router picked. A withdrawn free model falls back to the other free models first, then to the paid list; a paid model goes straight to the paid list. It stops after three replacements for one request, and the provider keeps the new model for later calls. If OpenRouter’s error suggests a model, the suggestion is logged, but the choice still comes from the picker. Streaming requests do the same when the stream opens.

The free list is empty now. On 20 October none of the free models OpenRouter listed was a free version of a model with a sourced quality score, so LOW work goes to the paid list until one is.

What it doesn’t do

The quality scores date from research in July, and a comment in the code calls the floors a first-pass calibration. Kimi’s scores are estimates, which is why the picker can’t place it above STANDARD. Prices live in a YAML file that scripts/refresh_pricing.py updates from OpenRouter’s model list when it’s run.

A missing score cuts both ways. A model with no entry in quality.py passes every floor, which keeps routing working before it’s been scored, but means it counts as good enough for any tier until it is.

The list of unavailable models lives on one machine and lasts a day, so the first task after a withdrawal still makes one failed request.

Using it

mullet explain "redesign the auth architecture across all services, migrate production data and debug the token race condition"
mullet routing simulate --changed-only

mullet explain shows the heuristic’s working without calling a model. mullet routing simulate replays past routing decisions through the current policy and preset, and --changed-only lists only the tasks that would now go somewhere else, so it shows what a change to a floor or a price would move. MULLET_DELEGATION=off in .env turns delegation off and leaves search working, which is how it’s set on my machine.

The code is on GitHub.