Tom Gould Markdown JSON

Our release notes were wrong, and how I found out

Mullet 2.3.0’s notes made two claims the code did not support. Checking each claim against the repo while I wrote about the release caught both, and 2.3.1 corrected them.

A table headed notes said and code said. Decision records: 0001 to 0010 against 0001 to 0011. Container labels: missing in 2.3.0 against present at the 2.3.0 tag. A line below says both were corrected in 2.3.1. A table headed notes said and code said. Decision records: 0001 to 0010 against 0001 to 0011. Container labels: missing in 2.3.0 against present at the 2.3.0 tag. A line below says both were corrected in 2.3.1.

A release note is a claim about code, and a claim can be wrong. Two of the claims in Mullet 2.3.0’s notes were.

What the notes said

  • The decision records, the short context, decision and consequences files beside the code, ran from 0001 to 0010.
  • Containers started by 2.3.0 lack the owner labels that the cleanup of orphaned containers relies on, so they are never cleaned up and have to be removed by hand.

What the code said

There were eleven decision records, not ten. The eleventh is the one for the container sandbox, which was the main new feature of the release, and the notes had left it out.

The second claim was written before the last fixes went in. The v2.3.0 tag already had the labels, so the warning described a build that never shipped. I only found that by looking at what the tag contained, rather than at what I remembered writing.

How it was found

I was writing about the release, and for each claim in a post I looked for the line of code or the file that backed it. That is slow, and it turned up more than I expected. These notes were the largest finding. A smaller one was that the Python plan() call returned the planner’s notes but dropped the questions it could not answer from the code, so a script that approved straight away never saw them.

None of it was hidden by a test. A test checks what the code does. Nothing checks that the words describing the code are true.

What changed

Mullet 2.3.1 corrected both notes, and plan() now returns the open questions. The sandbox image tag follows the version, so after an upgrade mullet sandbox build-image has to be run again. That is the part of the old warning that still holds. One older line about rebuilding an image from 2.3.0 is still in the 2.3.0 notes, and it is the second thing I would fix.

I don’t know whether anyone acted on the wrong notes in the short time they were up. If you did rebuild an image you did not need to, that cost a few minutes and nothing else.

The habit I am keeping

When a post or a changelog says what the code does, I find the file that proves it before it goes out. For anything with a number in it, I find the number. The posts I wrote this week have a few lines I could not back that way, and they say so, as we checked the bill did about the ledger.

The notes for every release are in the changelog.

More on Mullet

More writing