Every command a Mullet job runs on my machine runs in a sandbox: the repo’s setup, its checks, the gates and the opencode worker. Until 2.3.0 that sandbox was bubblewrap, written straight into the module that called it. Sandboxing an AI coding worker describes what it does. This post is about moving it behind an interface without changing any of that, so a second backend could be added safely.
\nBubblewrap needs working user namespaces. Some Linux machines don’t have them, and macOS has no bubblewrap at all. I wanted a container backend for the first case and room for a macOS one later. Either way, a new backend has to keep exactly the guarantees bubblewrap gives, and the code that calls the sandbox shouldn’t need to know which one it is talking to.
\nmullet/jobs/sandbox_backends/ now holds a SandboxBackend protocol: a name and seven methods, check, build_argv, spawn, run, run_shell, nested_argv and nested_check. BubblewrapBackend was the first implementation. The old module, mullet.jobs.sandbox, keeps every name its callers use, picks one backend per process and hands each call to it. Gates and executors need no change to run under a new backend.
The backend is a setting:
\nsandbox:\n backend: bubblewrap # bubblewrap (default), docker, podman or auto\nauto takes the first backend that works on the machine. seatbelt is a reserved name: choosing it refuses all work, with a message that says so.
A sandbox is a bad place for a subtle change. If a mount moves or a flag goes missing, no test has to fail. The walls are just lower. Ordinary tests check what I thought to check, and a refactor is exactly when I’m not thinking about the parts I didn’t touch.
\nSo before the change went in, I compared the command lines the old code and the new code built for the same inputs: 15,552 of them, with zero differences. That is what lets me say the refactor changed nothing for anyone already using bubblewrap.
\nThe comparison itself isn’t in the repo. What stays is a test, test_bubblewrap_argv_is_exactly_the_documented_line, that pins the exact bubblewrap command line for a known spec, with network and without. Another checks that the old module no longer offers build_argv at all, so nothing outside a backend can build a sandbox command line its own way.
Before, the guarantees were only described, in the sandbox module’s docstring. Now they are tests too. Each test_contract_* test in tests/test_sandbox_backends.py takes a backend fixture, runs real commands under that backend and checks what happened:
| Contract test | \nWhat it holds the backend to | \n
|---|---|
| writes land only in writable paths | \nnothing writable outside the clone and the paths the job names | \n
| root and exposed paths are read-only | \nthe host’s files can’t be changed | \n
| no network | \na worker, check or gate reaches nothing | \n
| environment is an allowlist | \nonly the variables Mullet chose get in | \n
| process environments hold no parent secret | \nMullet’s own keys never appear inside | \n
| timeout kills the whole tree | \nno process the command started outlives it | \n
| the key proxy socket is reachable | \nthe opencode worker can still reach its provider through the proxy | \n
| spawn confines a long-lived worker | \na worker gets the same walls as a check | \n
A few more check that build_argv never returns the bare command, and that a refused spec leaves no log behind.
A backend joins the suite by adding its name to CONTRACT_BACKENDS. On a machine where it can’t run, its contract tests skip, unless the MULLET_REQUIRE_SANDBOX environment variable names it, and then a skip is a failure. CI sets that for Docker, so the container backend can’t quietly stop being tested.
There is no fallback to running unsandboxed. An unknown, reserved or broken backend makes check() fail, and every entry point then raises SandboxUnavailable. auto raises when no backend works. Nothing runs instead. A value Mullet doesn’t recognise in the config is reported by mullet config show and mullet doctor, and bubblewrap is kept, which is still a sandbox. mullet doctor also says which backend is active and why.
That rule shaped the interface more than anything else. nested_argv starts a second sandbox inside the first, one for each of the opencode worker’s shell commands, with no network at all. A backend either does that or raises; there is no best-effort version. It mattered as soon as the container backend arrived, because a container can’t nest another sandbox without loosening the engine’s default security profiles, so opencode is refused under it. A container backend stricter than the usual Docker recipe explains why.
The design, with what each caller needs from a sandbox, is decision record 0007 in the repo’s docs/adr/ (decision records for a solo project says why there is such a folder).
Check a refactor by difference, not only by tests. Have the old code and the new produce their outputs for the same inputs and compare every one. It catches the change you didn’t know to test for, and “zero differences” is a much stronger statement than “the tests pass”.
\nAnd write the guarantees as tests that take the implementation as a parameter. The container backend was written against those tests, and it is only allowed to run jobs because it passes them. The same will go for macOS: until a backend passes the contract there, macOS jobs stay refused.
\nThis is in Mullet 2.3.0. The code is at https://github.com/tomgould/mullet-public.
" }, "markdown_text": "---\ntitle: 'Putting the sandbox behind an interface'\ncanonical: 'https://www.dealmobile.co.uk/writing/putting-the-sandbox-behind-an-interface'\nurl: 'https://www.dealmobile.co.uk/writing/putting-the-sandbox-behind-an-interface'\ntype: Post\npublished: '2026-10-26'\nsummary: 'Mullet’s bubblewrap sandbox now sits behind a backend interface, with contract tests every backend must pass. Before the change went in, 15,552 command lines from the old code and the new were compared, and none differed.'\n---\n\n# Putting the sandbox behind an interface\n\nPublished 2026-10-26\n\n\n\nEvery command a Mullet job runs on my machine runs in a sandbox: the repo’s setup, its checks, the gates and the opencode worker. Until 2.3.0 that sandbox was bubblewrap, written straight into the module that called it. [Sandboxing an AI coding worker](https://www.dealmobile.co.uk/writing/sandboxing-an-ai-coding-worker.md) describes what it does. This post is about moving it behind an interface without changing any of that, so a second backend could be added safely.\n\n## Why bother\n\nBubblewrap needs working user namespaces. Some Linux machines don’t have them, and macOS has no bubblewrap at all. I wanted a container backend for the first case and room for a macOS one later. Either way, a new backend has to keep exactly the guarantees bubblewrap gives, and the code that calls the sandbox shouldn’t need to know which one it is talking to.\n\n## The interface\n\n`mullet/jobs/sandbox_backends/` now holds a `SandboxBackend` protocol: a `name` and seven methods, `check`, `build_argv`, `spawn`, `run`, `run_shell`, `nested_argv` and `nested_check`. `BubblewrapBackend` was the first implementation. The old module, `mullet.jobs.sandbox`, keeps every name its callers use, picks one backend per process and hands each call to it. Gates and executors need no change to run under a new backend.\n\nThe backend is a setting:\n\n```yaml\nsandbox:\n backend: bubblewrap # bubblewrap (default), docker, podman or auto\n\n```\n\n`auto` takes the first backend that works on the machine. `seatbelt` is a reserved name: choosing it refuses all work, with a message that says so.\n\n## A refactor checked by difference\n\nA sandbox is a bad place for a subtle change. If a mount moves or a flag goes missing, no test has to fail. The walls are just lower. Ordinary tests check what I thought to check, and a refactor is exactly when I’m not thinking about the parts I didn’t touch.\n\nSo before the change went in, I compared the command lines the old code and the new code built for the same inputs: 15,552 of them, with zero differences. That is what lets me say the refactor changed nothing for anyone already using bubblewrap.\n\nThe comparison itself isn’t in the repo. What stays is a test, `test_bubblewrap_argv_is_exactly_the_documented_line`, that pins the exact bubblewrap command line for a known spec, with network and without. Another checks that the old module no longer offers `build_argv` at all, so nothing outside a backend can build a sandbox command line its own way.\n\n## The contract\n\nBefore, the guarantees were only described, in the sandbox module’s docstring. Now they are tests too. Each `test_contract_*` test in `tests/test_sandbox_backends.py` takes a `backend` fixture, runs real commands under that backend and checks what happened:\n\n| Contract test | What it holds the backend to |\n|:--|:--|\n| writes land only in writable paths | nothing writable outside the clone and the paths the job names |\n| root and exposed paths are read-only | the host’s files can’t be changed |\n| no network | a worker, check or gate reaches nothing |\n| environment is an allowlist | only the variables Mullet chose get in |\n| process environments hold no parent secret | Mullet’s own keys never appear inside |\n| timeout kills the whole tree | no process the command started outlives it |\n| the key proxy socket is reachable | the opencode worker can still reach its provider through the proxy |\n| spawn confines a long-lived worker | a worker gets the same walls as a check |\n\nA few more check that `build_argv` never returns the bare command, and that a refused spec leaves no log behind.\n\nA backend joins the suite by adding its name to `CONTRACT_BACKENDS`. On a machine where it can’t run, its contract tests skip, unless the `MULLET_REQUIRE_SANDBOX` environment variable names it, and then a skip is a failure. CI sets that for Docker, so the container backend can’t quietly stop being tested.\n\n## The rule I kept\n\nThere is no fallback to running unsandboxed. An unknown, reserved or broken backend makes `check()` fail, and every entry point then raises `SandboxUnavailable`. `auto` raises when no backend works. Nothing runs instead. A value Mullet doesn’t recognise in the config is reported by `mullet config show` and `mullet doctor`, and bubblewrap is kept, which is still a sandbox. `mullet doctor` also says which backend is active and why.\n\nThat rule shaped the interface more than anything else. `nested_argv` starts a second sandbox inside the first, one for each of the opencode worker’s shell commands, with no network at all. A backend either does that or raises; there is no best-effort version. It mattered as soon as the container backend arrived, because a container can’t nest another sandbox without loosening the engine’s default security profiles, so opencode is refused under it. [A container backend stricter than the usual Docker recipe](https://www.dealmobile.co.uk/writing/a-container-backend-stricter-than-the-usual-docker-recipe.md) explains why.\n\nThe design, with what each caller needs from a sandbox, is decision record 0007 in the repo’s `docs/adr/` ([decision records for a solo project](https://www.dealmobile.co.uk/writing/decision-records-for-a-solo-project.md) says why there is such a folder).\n\n## What I’d do again\n\nCheck a refactor by difference, not only by tests. Have the old code and the new produce their outputs for the same inputs and compare every one. It catches the change you didn’t know to test for, and “zero differences” is a much stronger statement than “the tests pass”.\n\nAnd write the guarantees as tests that take the implementation as a parameter. The container backend was written against those tests, and it is only allowed to run jobs because it passes them. The same will go for macOS: until a backend passes the contract there, macOS jobs stay refused.\n\nThis is in Mullet 2.3.0. The code is at