Writing down an upload protocol and testing servers against it

TurboPush has two halves: a TypeScript client that sends files in chunks from the browser, and a PHP endpoint that checks them and puts them back together. The README says either half can be paired with your own code. That’s only true if the conversation between them is written down somewhere other than inside a PHP class. So version 2.1 added PROTOCOL.md, a specification of the protocol, and a conformance suite that checks any server against it over HTTP. How the endpoint itself handles chunks is in Making large uploads resumable.

The protocol on paper

The protocol is small. Every operation is a POST to one URL. A multipart body is a chunk, and a JSON body with an action of finalize, status or abort is one of the other three operations. Every response is the same JSON envelope, {"success": ..., "message": ..., "data": ...}, and each status code has one meaning: 404 for an upload the server doesn’t hold, 409 for a conflict with the upload’s state, 413 for a limit, 415 for a refused file type.

PROTOCOL.md is written so that someone could build a server in another language without reading the PHP. It uses RFC 2119’s MUST and SHOULD, and calls the PHP class “the reference implementation” wherever it describes what that class happens to do. The detail is in the places where a reasonable server could go either way. For example:

• An upload ID must match ^[A-Za-z0-9_-]{1,200}$ in full, and anything else is a 400 “without trimming, decoding or rewriting it”, because servers use the ID as a directory name.
• A number in a multipart field is ASCII digits only. +2000, 2000, 0x10 and 1e3 are all 400s.
• An empty data may be sent as [], because that’s how PHP encodes an empty array, so clients must accept {} and [].
• A JSON number such as 10.0 may be rejected even though it’s whole, because parsers in some languages can’t tell it from 10. The suite doesn’t test that case, and clients must send plain integers.

That last kind of note matters. A rule only the PHP version could meet would make the document a description of one server, not a protocol.

Markers that link the two

Every requirement the suite verifies carries the ID of the check that verifies it:

- Sending a chunk again MUST succeed and MUST NOT count its bytes twice [check: `upload.duplicate-chunk`].

A requirement without a marker isn’t checked over HTTP, usually because it needs the server’s internals or a long wait, such as deleting abandoned uploads after 24 hours. So the gaps are visible in the specification itself, and a new check isn’t finished until its marker is in place.

The suite

The suite is plain Node.js 18 or later, with no dependencies. It talks HTTP through node:http rather than fetch, and builds its multipart bodies by hand, so every header and every byte is under its control: an Origin header, a field called file[], a NUL byte in a value.

A check is an object with an ID, a severity, a description and a run() function. This is the one behind that first marker:

{
    id: 'upload.duplicate-chunk',
    severity: 'MUST',
    description: 'Resending a chunk is harmless and is not counted twice',
    async run(ctx) {
        const fileId = ctx.id();
        const fileName = ctx.name('dup', 'bin');
        const data = randomBytes(2000);
        await ctx.sendChunks({ fileId, fileName, data, order: [0, 0, 1, 1, 0] });
        const status = expectOk(await ctx.status(fileId), 'status');
        expect(status.receivedBytes === 2000,
            `status: receivedBytes is ${JSON.stringify(status.receivedBytes)} after resends, expected 2000`);
        const response = await ctx.finalize({ fileId, fileName, fileSize: 2000, totalChunks: 2 });
        expectStored(ctx, expectOk(response, 'finalize'), data);
    },
},

There are 74 checks, 66 of them MUST and 8 SHOULD, in six files: the upload lifecycle, request validation, security, concurrency, CORS, and sweeps. A failed MUST fails the run and a failed SHOULD is a warning. A check that needs something you didn’t supply, such as the server’s limits or an origin on its allowlist, is skipped rather than failed.

The concurrency checks are the ones I’d least like to do by hand. Sixteen chunks of one file arrive at once, in reverse order. Six copies of the same chunk, each with different bytes, arrive together, and all six must succeed and leave exactly one complete version. Eight finalize requests for one upload race each other, and all eight must return the same file name, with one file stored. Four uploads of the same name finalize together and must get four distinct files.

Looking inside, when it can

The suite treats the server as a black box, but if it can read the server’s upload directory (--upload-dir), it checks more. Stored files are compared byte for byte with what was sent, and afterwards the directory mustn’t contain any file the server didn’t report.

The path traversal check uses that too. It uploads files named ../../<marker>.txt, C:\<marker>.txt, ....//....//<marker>.txt, \\server\share\<marker>.txt, ..%2F..%2F<marker>.txt and a few more, each with a random marker. A server may refuse each one with a 4xx, or store it under a single plain name inside the upload directory. Then the suite looks for the marker in the system temp directory, the filesystem root, the current directory and the two directories above the upload directory, in case a file got out.

Some checks run last, over every response the others received. Every response must be the JSON envelope, with success matching the status code. No error may contain a server path: the sweep looks for absolute paths such as /var/... and C:\... and for the temp and upload directories, ignoring any path the request itself sent, such as a file name the server echoed back.

What it found

The suite’s first target was my own endpoint, and it found two bugs there.

The first was PHP’s switch, which compares loosely. A request with {"action": true} matched case 'finalize', as did 0 on PHP 7, and with the other fields filled in it really finalized the upload. The fix is to treat anything that isn’t a string as no action at all:

// switch compares loosely, so a non-string such as true (or 0 on
// PHP 7) would otherwise match 'finalize'
$action = is_string($data['action']) ? $data['action'] : '';
switch ($action) {
    case 'finalize':
        return $this->finalizeUpload($data);

The check behind it, action.non-string, sends true, 0, 1, ["finalize"] and {"name": "finalize"}, and expects a 400 for each.

The second was the content type. JSON actions must arrive as application/json, because browsers send plain text, form and multipart requests to other sites without asking first, and a JSON content type forces them to ask. The endpoint checked that the header started with application/json, so application/jsonx got through. It now compares the media type exactly, ignoring case and parameters such as charset. A neighbouring check, csrf.simple-requests, sends an abort as plain text, as a form and as multipart, and then asks for the upload’s status to make sure it’s still there.

Running it on every PHP version

conformance/php.mjs runs the suite against the reference endpoint. It starts PHP’s built-in server on a free port with fresh temporary storage and four workers, because the built-in server otherwise handles one request at a time and the concurrency checks would prove nothing. PHP runs with every error reported and displayed, so any warning or deprecation breaks that response’s JSON and fails the envelope check. The suite runs twice, the second time against an endpoint with a CORS allowlist.

GitHub Actions runs it on every PHP version from 7.4 to 8.5. On PHP 8.1 on my machine, the first pass sends 579 requests in 1.4 seconds: 70 checks pass and 4 are skipped, because they need an allowlisted origin. Those four pass in the second run.

What it can’t tell you

Anything that can only be seen inside the server is out of reach. The suite doesn’t wait 24 hours to see abandoned uploads deleted, and it can’t see that the PHP endpoint’s locks are unreliable on NFS, which the specification says plainly instead. Passing means a server behaves correctly over HTTP in the ways the checks probe. It doesn’t prove the server is right.

Using it

npm run conformance:php        # against the PHP endpoint, as CI does
node conformance/run.mjs --list

node conformance/run.mjs --url http://localhost:8000/upload.php \
  --max-file-size 8388608 --max-chunk-size 262144 --max-chunks 64 \
  --upload-dir /srv/uploads --only fileid.,filename.

Point it at a test deployment, not a live one: it leaves small files whose names start with tpc- in the upload directory. The exit code is 1 if any MUST check fails. The code, PROTOCOL.md and the suite are on GitHub.