Tom Gould Markdown JSON

A container backend stricter than the usual Docker recipe

Mullet 2.3.0 can run a job’s commands in Docker: one throwaway container per command, no network, no capabilities, secrets on stdin and a lifeline that ends the container if Mullet dies. Optional, Linux only, with bubblewrap still the default.

One throwaway container per command. A table of the flags Mullet’s container backend passes and what each means: --network none, no network outside setup; --read-only, a read-only root; --cap-drop ALL, no capabilities; no-new-privileges; --user with your own uid and gid; --memory and --pids-limit, 8g and 4096 by default; --pull never, a local image only; and the environment, on stdin, never on a command line. One throwaway container per command. A table of the flags Mullet’s container backend passes and what each means: --network none, no network outside setup; --read-only, a read-only root; --cap-drop ALL, no capabilities; no-new-privileges; --user with your own uid and gid; --memory and --pids-limit, 8g and 4096 by default; --pull never, a local image only; and the environment, on stdin, never on a command line.

Mullet 2.3.0 adds a second sandbox backend: Docker, with Podman in the code. It is optional and Linux only, and bubblewrap stays the default. It is for Linux machines where bubblewrap can’t run because user namespaces aren’t available. It runs every command a job makes (setup, checks and gates) in a throwaway container, one per command, and it had to pass the same contract tests as bubblewrap before it was allowed near a job (putting the sandbox behind an interface).

Where the usual recipe falls short

The common way of running untrusted code in a container has three gaps that matter for unattended work. Decision record 0011 in the repo starts from them:

  • An environment passed with -e KEY=value shows in ps and in docker inspect.
  • A pulled image is whatever the registry serves that day.
  • Killing the docker run client with SIGKILL leaves its container running. I checked that on Docker 29.9.2.

The third is the one I cared about most. If Mullet crashes at 3am, nothing it started should keep going.

The flags

This is the start of the command line the backend builds, from mullet/jobs/sandbox_backends/container.py:

argv = [self._engine(), "run", "--rm", "--interactive", "--sig-proxy=true",
        "--name", name, "--label", LABEL, *owner, "--pull", "never", "--log-driver", "none",
        "--user", f"{uid}:{gid}", *self._engine_options(),
        "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--read-only",
        "--ipc", "private", "--pids-limit", str(s.pids_limit),
        "--memory", s.memory, "--memory-swap", s.memory]
if not spec.network:
    argv += ["--network", "none"]

In words: no network, except for setup commands, which get the engine’s default network (the internet, not the host’s loopback). A read-only root. Every capability dropped, and no new privileges. Your own uid and gid, so files in the clone stay yours; rootless Podman adds --userns keep-id for the same reason. A memory limit with swap included, and a limit on processes and threads. /tmp, /run and $HOME are empty tmpfs mounts. Only the clone and the few paths the job names are bind-mounted, at their own paths, writable ones read-write and the rest read-only.

A writable path that is a symlink is refused, and so is any mount with a symlinked component inside a writable path, because a command could plant one between runs and the daemon would follow it. That is checked when the command line is built, and again on the exact bind strings just before docker run.

The image is never pulled. mullet sandbox build-image builds it locally from the Dockerfile Mullet ships, whose base image is pinned by digest. check() requires the image’s labels, so an unrelated image that happens to carry the tag isn’t used. The labels are not a signature, and a deliberately forged image would pass. The decision record says so.

Secrets on stdin

No environment value goes on a command line or into the container’s configuration. The container’s PID 1 reads the environment from stdin as one JSON line, then starts the command with stdin on /dev/null. docker inspect has nothing worth reading.

A lifeline the command can’t cut

The rest of stdin is a lifeline. Mullet holds the write end of the pipe. If Mullet dies, however it dies, the pipe closes, and PID 1 kills every process in the container and exits, which ends it.

The first version didn’t hold. An independent review found that a command inside the container could kill the process watching stdin, so a hard Mullet crash left its container running. The fix was to make the watcher PID 1 itself. Each container now starts Mullet’s own init, pid1.py, from an empty environment, in place of Docker’s --init. The kernel never delivers SIGKILL or SIGSTOP to a PID namespace’s init from inside that namespace, and PID 1’s handlers for other signals only forward them or wake its loop. It also makes itself non-dumpable, so the command, though it runs as the same uid, can’t trace it or read its memory.

For the case where the lifeline never reaches PID 1, such as a daemon restarted mid-run, there is a backstop. Every container carries owner labels: the Mullet process, its start, the boot, the machine and the user. Runner start and resume, mullet job stop and mullet doctor remove containers whose owner has died. A second review found that the owner’s start time came from the wall clock, so a clock step could make a live job look dead and have its container removed. Owners are now named by their start in clock ticks since boot, compared exactly, and only your own mullet-sbx-* containers are ever removed.

Checking the checks

check() refuses a daemon without seccomp, then starts a real probe container that must report Seccomp: 2, a zero CapEff and NoNewPrivs: 1 from /proc/self/status. The contract tests run against Docker in a CI job that builds the image and sets MULLET_REQUIRE_SANDBOX=docker, so a skip there is a failure.

Security tests are easy to write so they pass whatever the code does. So I checked them by mutation: twelve deliberate weakenings of the protections, and a test caught each one.

Turning it on

mullet sandbox build-image

Then, in the config file:

sandbox:
  backend: bubblewrap   # bubblewrap (default), docker, podman or auto
  image: mullet-sandbox:<version>   # docker and podman only
  memory: 8g            # docker and podman only: the container's memory limit
  pids_limit: 4096      # docker and podman only: the most processes and threads

Set backend to docker. mullet doctor says which backend is active, or why it refused.

The limits

  • The opencode worker is refused under containers. It runs each of its shell commands in a second sandbox with no network inside the first, and that needs new user and network namespaces, which Docker’s and Podman’s default seccomp and AppArmor profiles refuse. Allowing them means loosening both, and Mullet doesn’t. Use bubblewrap for opencode jobs. The Claude worker uses Claude Code’s own sandbox either way.
  • Podman is untested. The backend is there and passes the unit tests, but I have no machine with Podman, so it has never run a real container here and I don’t claim it works.
  • macOS is untested, and jobs there stay refused until the contract tests pass on it.
  • Commands see the image’s userland, not the host’s. A repo that needs other toolchains needs its own image, built FROM mullet-sandbox and named in sandbox.image.
  • Each command costs a container start, about 0.2 seconds with a warm engine.

The code is at https://github.com/tomgould/mullet-public.

More on Mullet

More writing