Putting the sandbox behind an interface

Every command a Mullet job runs on my machine runs in a sandbox: the repo’s setup, its checks, the gates and the opencode worker. Until 2.3.0 that sandbox was bubblewrap, written straight into the module that called it. Sandboxing an AI coding worker describes what it does. This post is about moving it behind an interface without changing any of that, so a second backend could be added safely.

Why bother

Bubblewrap needs working user namespaces. Some Linux machines don’t have them, and macOS has no bubblewrap at all. I wanted a container backend for the first case and room for a macOS one later. Either way, a new backend has to keep exactly the guarantees bubblewrap gives, and the code that calls the sandbox shouldn’t need to know which one it is talking to.

The interface

mullet/jobs/sandbox_backends/ now holds a SandboxBackend protocol: a name and seven methods, check, build_argv, spawn, run, run_shell, nested_argv and nested_check. BubblewrapBackend was the first implementation. The old module, mullet.jobs.sandbox, keeps every name its callers use, picks one backend per process and hands each call to it. Gates and executors need no change to run under a new backend.

The backend is a setting:

sandbox:
  backend: bubblewrap   # bubblewrap (default), docker, podman or auto

auto takes the first backend that works on the machine. seatbelt is a reserved name: choosing it refuses all work, with a message that says so.

A refactor checked by difference

A sandbox is a bad place for a subtle change. If a mount moves or a flag goes missing, no test has to fail. The walls are just lower. Ordinary tests check what I thought to check, and a refactor is exactly when I’m not thinking about the parts I didn’t touch.

So before the change went in, I compared the command lines the old code and the new code built for the same inputs: 15,552 of them, with zero differences. That is what lets me say the refactor changed nothing for anyone already using bubblewrap.

The comparison itself isn’t in the repo. What stays is a test, test_bubblewrap_argv_is_exactly_the_documented_line, that pins the exact bubblewrap command line for a known spec, with network and without. Another checks that the old module no longer offers build_argv at all, so nothing outside a backend can build a sandbox command line its own way.

The contract

Before, the guarantees were only described, in the sandbox module’s docstring. Now they are tests too. Each test_contract_* test in tests/test_sandbox_backends.py takes a backend fixture, runs real commands under that backend and checks what happened:

Contract test	What it holds the backend to
writes land only in writable paths	nothing writable outside the clone and the paths the job names
root and exposed paths are read-only	the host’s files can’t be changed
no network	a worker, check or gate reaches nothing
environment is an allowlist	only the variables Mullet chose get in
process environments hold no parent secret	Mullet’s own keys never appear inside
timeout kills the whole tree	no process the command started outlives it
the key proxy socket is reachable	the opencode worker can still reach its provider through the proxy
spawn confines a long-lived worker	a worker gets the same walls as a check

A few more check that build_argv never returns the bare command, and that a refused spec leaves no log behind.

A backend joins the suite by adding its name to CONTRACT_BACKENDS. On a machine where it can’t run, its contract tests skip, unless the MULLET_REQUIRE_SANDBOX environment variable names it, and then a skip is a failure. CI sets that for Docker, so the container backend can’t quietly stop being tested.

The rule I kept

There is no fallback to running unsandboxed. An unknown, reserved or broken backend makes check() fail, and every entry point then raises SandboxUnavailable. auto raises when no backend works. Nothing runs instead. A value Mullet doesn’t recognise in the config is reported by mullet config show and mullet doctor, and bubblewrap is kept, which is still a sandbox. mullet doctor also says which backend is active and why.

That rule shaped the interface more than anything else. nested_argv starts a second sandbox inside the first, one for each of the opencode worker’s shell commands, with no network at all. A backend either does that or raises; there is no best-effort version. It mattered as soon as the container backend arrived, because a container can’t nest another sandbox without loosening the engine’s default security profiles, so opencode is refused under it. A container backend stricter than the usual Docker recipe explains why.

The design, with what each caller needs from a sandbox, is decision record 0007 in the repo’s docs/adr/ (decision records for a solo project says why there is such a folder).

What I’d do again

Check a refactor by difference, not only by tests. Have the old code and the new produce their outputs for the same inputs and compare every one. It catches the change you didn’t know to test for, and “zero differences” is a much stronger statement than “the tests pass”.

And write the guarantees as tests that take the implementation as a parameter. The container backend was written against those tests, and it is only allowed to run jobs because it passes them. The same will go for macOS: until a backend passes the contract there, macOS jobs stay refused.

This is in Mullet 2.3.0. The code is at https://github.com/tomgould/mullet-public.