Tom Gould Markdown

Writing

A 30-minute test suite in four and a half

Cutting every test off from the real world let Mullet’s 16,860 tests run in parallel, and flushed out a real Kimi call and a fixture deleting a real file.

Two bars: the old Mullet test run of about 30 minutes in series, and the new run of about 4.5 minutes across 16 workers.

Mullet’s test suite used to take about 30 minutes. It was about 17,900 tests run in series. Now it is about 16,860 tests, run on 16 workers with pytest-xdist, in about 4.5 minutes.

The speed-up is not the interesting part. Parallel runs only work when tests cannot affect each other or the machine they run on. Making that true was the real work, and it found things I did not know were wrong.

Why it was slow, and why that mattered

Thirty minutes means you do not run the suite before committing. You run the file you changed and hope. Mullet 2’s gates compare test results against a baseline, so a suite you cannot run quickly is a suite you cannot lean on. Four and a half minutes is short enough to run before a commit that touches several modules.

My working rule is still to run only the files I changed while I work, and the full suite before committing across modules. The command is:

python3 -m pytest tests -q -n 16 --timeout 300

Cutting tests off from the real world

Parallel tests need isolation, and isolation has four parts here. Every test gets:

  • No network beyond localhost. A guard on socket.connect fails any test that tries to reach out.
  • Provider keys removed from the environment.
  • A throwaway HOME.
  • Mock providers. Tests that need a real model are marked live and excluded by default.

The guard is blunt on purpose. It does not try to guess which calls are harmless. A test that connects to anything but localhost fails and tells you where.

The throwaway HOME matters as much as the network guard. Anything that writes to ~ writes to a temporary directory that is deleted afterwards. If a test used to write to my real home, it now cannot.

What the isolation found

I expected a few failures. I did not expect these.

A test had been calling the real Kimi API. A test that makes a real call passes when the real service answers. With the network guard and no keys, it failed.

An old fixture deleted my real learned-constraints file. Under a throwaway HOME, the path it deleted no longer resolved to my file. I might never have found that by reading the suite, because a fixture that deletes something is easy to skim past.

Some tests were writing to my real HUD, the status bar Mullet draws in Claude Code. That explains something that had puzzled me for a while: I kept seeing “Kimi agents” in my status bar when nothing was delegating. The tests had been drawing them.

All three were the same fault. The tests were reaching out of their own sandbox, and nothing had stopped them. They only showed up when something did.

The same idea in the product

This is the same principle as the sandbox around the worker, described in Sandboxing an AI coding worker. Code you did not look at closely should not be able to reach things you did not intend. For the tests, the “worker” was mine.

It also fits the gates in Gates, not a model’s word. A G5 comparison against a baseline is only worth anything if the baseline is repeatable. A suite that depends on the network, the clock or the contents of my home directory is not.

What I did not do

The count fell from about 17,900 to about 16,860 because Mullet 2 deleted the old orchestration modes, and their tests went with them. The product page I wrote before this had “17,700 tests”, which was already out of date.

I also have not tried to make the suite faster than 4.5 minutes. It is fast enough for now.

What changed in how I work

Before, a green full run was an event. Now it is a step I take before a commit that touches several modules. The run is cut off from the network, the keys and my home directory, so a pass on my laptop means the same as a pass anywhere else. That was not true when some tests quietly used my real files and my real provider.

Doing the same to yours

If you have a suite you do not trust to run in parallel, the order I would try is:

  1. Put a guard on socket.connect that fails anything outside localhost.
  2. Strip provider keys and credentials from the test environment.
  3. Point HOME at a temporary directory.
  4. Run it. Read every failure, because each one is a test reaching somewhere it should not.
  5. Then add -n workers.

Do the fourth step before the fifth. Parallelism will make the failures harder to read.

The code is at https://github.com/tomgould/mullet-public.