Mullet’s test suite used to take about 30 minutes. Now it is about 17,700 tests, run on 8 workers with pytest-xdist, in about 5 minutes. (By the end of that day it was about 18,000, and still about 5 minutes.)
\nThe speed-up is not the interesting part. Parallel runs only work when tests cannot affect each other or the machine they run on. Making that true was the real work, and it found things I did not know were wrong.
\nThirty minutes means you do not run the suite before committing. You run the file you changed and hope. Mullet 2’s gates compare test results against a baseline, so a suite you cannot run quickly is a suite you cannot lean on. Five minutes is short enough to run before a commit that touches several modules.
\nMy working rule is still to run only the files I changed while I work, and the full suite before committing across modules. The command is:
\npython3 -m pytest tests -q -n 8 --timeout 300\nParallel tests need isolation, and isolation has four parts here. Every test gets:
\nsocket.connect fails any test that tries to reach out.HOME.live and excluded by default.The guard is blunt on purpose. It does not try to guess which calls are harmless. A test that connects to anything but localhost fails and tells you where.
\nThe throwaway HOME matters as much as the network guard. Anything that writes to ~ writes to a temporary directory that is deleted afterwards. If a test used to write to my real home, it now cannot.
I expected a few failures. I did not expect these.
\nA test had been calling the real Kimi API. A test that makes a real call passes when the real service answers. With the network guard and no keys, it failed.
\nAn old fixture deleted my real learned-constraints file. Under a throwaway HOME, the path it deleted no longer resolved to my file. I might never have found that by reading the suite, because a fixture that deletes something is easy to skim past.
Some tests were writing to my real HUD, the status bar Mullet draws in Claude Code. That explains something that had puzzled me for a while: I kept seeing “Kimi agents” in my status bar when nothing was delegating. The tests had been drawing them.
\nAll three were the same fault. The tests were reaching out of their own sandbox, and nothing had stopped them. They only showed up when something did.
\nThis is the same principle as the sandbox around the worker, described in Sandboxing an AI coding worker. Code you did not look at closely should not be able to reach things you did not intend. For the tests, the “worker” was mine.
\nIt also fits the gates in Gates, not a model’s word. A G5 comparison against a baseline is only worth anything if the baseline is repeatable. A suite that depends on the network, the clock or the contents of my home directory is not.
\nI also have not tried to make the suite faster than about 5 minutes. It is fast enough for now.
\nBefore, a green full run was an event. Now it is a step I take before a commit that touches several modules. The run is cut off from the network, the keys and my home directory, so a pass on my laptop means the same as a pass anywhere else. That was not true when some tests quietly used my real files and my real provider.
\nIf you have a suite you do not trust to run in parallel, the order I would try is:
\nsocket.connect that fails anything outside localhost.HOME at a temporary directory.-n workers.Do the fourth step before the fifth. Parallelism will make the failures harder to read.
\nThe code is at https://github.com/tomgould/mullet-public.
" }, "markdown_text": "---\ntitle: 'A 30-minute test suite in five minutes'\ncanonical: 'https://www.dealmobile.co.uk/writing/a-30-minute-test-suite-in-five'\nurl: 'https://www.dealmobile.co.uk/writing/a-30-minute-test-suite-in-five'\ntype: Post\npublished: '2026-10-25'\nsummary: 'Cutting every test off from the real world let Mullet’s tests (about 17,700 when I wrote this) run in parallel, and flushed out a real Kimi call and a fixture deleting a real file.'\n---\n\n# A 30-minute test suite in five minutes\n\nPublished 2026-10-25\n\n\n\n[Mullet](https://www.dealmobile.co.uk/projects/mullet.md)’s test suite used to take about 30 minutes. Now it is about 17,700 tests, run on 8 workers with pytest-xdist, in about 5 minutes. (By the end of that day it was about 18,000, and still about 5 minutes.)\n\nThe speed-up is not the interesting part. Parallel runs only work when tests cannot affect each other or the machine they run on. Making that true was the real work, and it found things I did not know were wrong.\n\n## Why it was slow, and why that mattered\n\nThirty minutes means you do not run the suite before committing. You run the file you changed and hope. Mullet 2’s gates compare test results against a baseline, so a suite you cannot run quickly is a suite you cannot lean on. Five minutes is short enough to run before a commit that touches several modules.\n\nMy working rule is still to run only the files I changed while I work, and the full suite before committing across modules. The command is:\n\n```\npython3 -m pytest tests -q -n 8 --timeout 300\n\n```\n\n## Cutting tests off from the real world\n\nParallel tests need isolation, and isolation has four parts here. Every test gets:\n\n- No network beyond localhost. A guard on `socket.connect` fails any test that tries to reach out.\n- Provider keys removed from the environment.\n- A throwaway `HOME`.\n- Mock providers. Tests that need a real model are marked `live` and excluded by default.\n\nThe guard is blunt on purpose. It does not try to guess which calls are harmless. A test that connects to anything but localhost fails and tells you where.\n\nThe throwaway `HOME` matters as much as the network guard. Anything that writes to `~` writes to a temporary directory that is deleted afterwards. If a test used to write to my real home, it now cannot.\n\n## What the isolation found\n\nI expected a few failures. I did not expect these.\n\nA test had been calling the real Kimi API. A test that makes a real call passes when the real service answers. With the network guard and no keys, it failed.\n\nAn old fixture deleted my real learned-constraints file. Under a throwaway `HOME`, the path it deleted no longer resolved to my file. I might never have found that by reading the suite, because a fixture that deletes something is easy to skim past.\n\nSome tests were writing to my real HUD, the status bar Mullet draws in Claude Code. That explains something that had puzzled me for a while: I kept seeing “Kimi agents” in my status bar when nothing was delegating. The tests had been drawing them.\n\nAll three were the same fault. The tests were reaching out of their own sandbox, and nothing had stopped them. They only showed up when something did.\n\n## The same idea in the product\n\nThis is the same principle as the sandbox around the worker, described in [Sandboxing an AI coding worker](https://www.dealmobile.co.uk/writing/sandboxing-an-ai-coding-worker.md). Code you did not look at closely should not be able to reach things you did not intend. For the tests, the “worker” was mine.\n\nIt also fits the gates in [Gates, not a model’s word](https://www.dealmobile.co.uk/writing/gates-not-a-models-word.md). A G5 comparison against a baseline is only worth anything if the baseline is repeatable. A suite that depends on the network, the clock or the contents of my home directory is not.\n\n## What I did not do\n\nI also have not tried to make the suite faster than about 5 minutes. It is fast enough for now.\n\n## What changed in how I work\n\nBefore, a green full run was an event. Now it is a step I take before a commit that touches several modules. The run is cut off from the network, the keys and my home directory, so a pass on my laptop means the same as a pass anywhere else. That was not true when some tests quietly used my real files and my real provider.\n\n## Doing the same to yours\n\nIf you have a suite you do not trust to run in parallel, the order I would try is:\n\n1. Put a guard on `socket.connect` that fails anything outside localhost.\n2. Strip provider keys and credentials from the test environment.\n3. Point `HOME` at a temporary directory.\n4. Run it. Read every failure, because each one is a test reaching somewhere it should not.\n5. Then add `-n` workers.\n\nDo the fourth step before the fifth. Parallelism will make the failures harder to read.\n\nThe code is at