Cloning every repository without a token in the URL

Most of GitHub Auditor’s reports start from a local clone of every repository in an account. auditer checkout makes those clones and keeps them up to date, for as many accounts as the config lists, and a script in the repository is ready to run it from cron each night. Counting developer activity from git covers what the reports do with the clones. This post is about getting them there.

Private repositories need a token. The quick way to give git one is to put it in the URL, as in https://<token>@github.com/owner/repo.git. Git then saves that URL in .git/config in plain text, and anything that prints the remote, git remote -v for one, prints the token with it. Here’s how the tool avoids that, and what else checkout does along the way.

Asking git for the password

Git can get a password from a program. If the GIT_ASKPASS environment variable names one, git runs it whenever it needs a credential, and takes whatever it prints as the answer. _run_git() in src/auditer/git_ops.py writes a two-line shell script that prints the token, points GIT_ASKPASS at it for a single git command, and deletes it when the command ends:

def _make_askpass_script(token: str) -> str:
    path = tempfile.mktemp(suffix=".sh", prefix="git_askpass_")
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o700)
    try:
        os.write(fd, f"#!/bin/sh\necho {token}\n".encode())
    finally:
        os.close(fd)
    return path

A few details matter. The file is created with mode 0o700, so from the moment it exists only its owner can read or run it. Creating it with the usual permissions and tightening them afterwards would leave a gap, and a test checks that the mode is exactly 0o700. tempfile.mktemp() only picks a name, so on its own it would be open to a race. O_EXCL closes that: the open fails if anything already exists at that path, so a file planted there in the meantime is never used.

The environment carries the script’s path, not the token. GIT_TERMINAL_PROMPT=0 is set alongside it, so git never falls back to asking on the terminal, and a run from cron can’t hang waiting for someone to type. The script is removed in a finally block, so it goes even when the command fails or times out. If removing it fails, that’s logged as a warning rather than crashing the run, and there’s a test for that too.

The URL still has a user name

The clone URL gets a user name, but no password: https://[email protected]/owner/repo.git. With the user name already in the URL, git only asks for the password, and the script answers with the token.

Bitbucket Cloud works the same way with a different user name. For git over HTTPS with an API token, it wants the fixed name x-bitbucket-api-token-auth, not the account’s email address. Its REST API takes the same token with the email address as the user name, so the tool uses one convention for the API and the other for git.

Adding the user name has two traps, and _build_auth_url() handles both. Bitbucket’s clone URLs already contain a user name, so it has to be replaced, not added to. And a user name with an @ in it would end the user part of the URL early, so it’s percent-encoded:

parts = urlsplit(clone_url)
netloc = f"{quote(auth_username, safe='')}@{parts.hostname}"
if parts.port:
    netloc += f":{parts.port}"
return urlunsplit((parts.scheme, netloc, parts.path, parts.query, parts.fragment))

After a clone, the remote URL is set back to the plain clone URL. For an update, the URL with the user name is set just before the fetch and put back in a finally block. So .git/config holds the plain URL except while a clone or fetch is running, and even then it holds a user name, never the token.

Keeping it out of the output

When a clone or fetch fails, git’s error output is shown. Before it is, sanitise_git_output() in src/auditer/utils/security.py masks anything that matches one of GitHub’s token formats (ghp_, github_pat_, gho_, ghs_, ghu_ and ghr_), and any user name and password in a github.com URL. The same function runs over every log line, through a logging formatter in src/auditer/utils/logging.py:

class SanitisedFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        original = super().format(record)
        return sanitise_git_output(original)

auditer config --show prints each token as four asterisks and its last four characters. The token itself lives in config.yaml, which git ignores.

A clone is a mirror

An update runs git fetch --all --prune, then git reset --hard to the remote’s default branch. These clones are read-only copies of what’s on GitHub, and the reset makes sure each one matches. But a reset throws away uncommitted work. So before updating, checkout runs git status --porcelain, skips any clone with changes in it, and lists the skipped ones at the end. --no-skip-dirty turns that off. A repository with no commits has no default branch on the remote yet, so the reset is skipped for it.

With --tags, the fetch adds --tags --force. A comment in the code explains why: a tag moved on the remote should replace the local one, not make git refuse the whole fetch. With --branches, every local branch is moved to match its remote branch. --full turns on both.

The activity report fetches the clones too, so that it covers work pushed since the last checkout. That uses a separate function, fetch_repo(), which never touches the working tree or local branches, because someone might be working in that clone. It has a five-minute timeout for each repository.

Reporting what changed

For every branch a fetch updates, git prints a line with the old and new commits. Two dots between them mean the branch moved forward. Three dots mean a forced update, so _parse_fetch_output() looks for which one it got:

match = _HASH_RANGE_RE.search(stripped)
if match:
    old_hash = match.group(1)
    new_hash = match.group(2)
    forced = "..." in match.group(0)

For each updated branch, git log old..new lists the commits that were added, with their authors. For a force push, git rev-list --count new..old counts the commits that disappeared. The summary shows the branch as force-pushed, with the commits added and removed and a link to GitHub’s compare view. New branches and tags are listed with links too.

Repositories deleted from GitHub show up as well. After the run, checkout compares the local folders with the full list from the API, taken before any filters for visibility, forks, exclusions or projects, so a repository that’s only filtered out isn’t reported as deleted. Any folder left over is listed as an orphan and left on disk.

Repository names come from the API and become folder names, so before a clone each name is checked against a pattern of safe characters and refused if it contains ...

Using it

auditer checkout --account my-org --dry-run
auditer checkout --all --full

The first shows what would be cloned or updated without doing it. For the nightly run, scripts/daily-checkout.sh runs the second from the repository root and appends to a log file:

0 4 * * * /path/to/github-auditor/scripts/daily-checkout.sh

The code is on GitHub.