Switching the Default Python on FreeBSD Ports

I learned this the slow way: 3.14 to 3.15 on one host, 3.11 to 3.12 on several others, each run teaching me about a failure the previous one had not shown. Every failure is a consequence of how the ports tree models Python, so any of them can hit anyone who runs DEFAULT_VERSIONS+=python=3.x.

This post explains why the change is harder than it looks, lists the failures I actually met, and ends with the shell script that came out of it. It is a plain sh script around portmaster, and it is built to be re-run until the machine is clean.

Why a new default Python is not a version bump

The ports tree does not treat Python modules as one package that gets newer. Each module is built once per interpreter, using a flavor: devel/py-setuptools with FLAVOR=py311 produces py311-setuptools, and with FLAVOR=py312 it produces py312-setuptools. The origin is the same, the package names differ, and nothing in MOVED connects them. To pkg, going from 3.11 to 3.12 means removing about a hundred packages and installing about a hundred different ones.

Upgrade tools are built around the other case, where one package name gets a newer version. pkg upgrade and portmaster -a never swap py311-foo for py312-foo. portmaster -o can replace a single package with a new origin, one at a time, in an order you have to work out yourself.

The interpreter's version also ends up in places that no package name shows:

• Scripts get a shebang such as #!/usr/local/bin/python3.11 when the port is built.
• Compiled code and Python files live in /usr/local/lib/python3.11/site-packages.
• lang/python and lang/python3 are meta ports that own the python and python3 symlinks, and the pythonpackage carries the old version in its own version string.
• Some packages link against libpython3.11.so.

That leaves five kinds of package that have to be found and handled. The standard tools see only the first.

There is a further trap: the build needs the interpreter it is replacing. Modules need setuptools, flit, installer and wheel in the new flavor before anything else builds. Tools such as meson and glib need a working Python before their dependents can be configured. Delete things in the wrong order and the machine cannot rebuild itself.

Field notes: what actually broke

These are the failures I met, in the rough order they showed up. Each one is now handled by the script, or noted as something to check by hand.

Bootstrap packages with a dead shebang

glib20 failed to configure with Could not execute g-ir-scanner --version. The ports glib-bootstrap and gobject-introspection-bootstrap live under /usr/local/*-bootstrap, outside /usr/local/bin, and their scripts still named the removed interpreter. They only matter while building, but once broken they stop the build of everything above them.

The fix is to delete those packages and build them again before anything else. One detail cost me time: make stage does not build missing build dependencies, it fails with g-ir-scanner - not found, so the rebuild has to go through make install.

Flavored packages that fight over the same files

devel/scons installs /usr/local/bin/scons for every flavor, so scons-py312 cannot be installed while scons-py311 is present. mod_wsgi and uwsgi behave the same way. The script removes the old-flavor package of the same origin right before it installs the new one.

An origin that differs only by case

pkg knew devel/py-automat, the ports tree has devel/py-Automat. Anything that compares origins as plain strings misses the module and reports it as never migrated. All comparisons in the script are case-insensitive, and the new origin is found by a case-insensitive directory lookup.

Local edits to the ports tree

Several local Makefile tweaks (py-jaraco.*, py-SecretStorage, py-pynacl) assumed normalized wheel names, while the tree's py-setuptools 63.1.0 does not normalize them, so the wheel glob in python.mk matched nothing. Resetting the files to upstream fixed all of them. If your tree is a git checkout, run git status in it before the migration.

A build that fails because the module is installed

py-google-re2 could not be built while py312-scikit-build-core was installed, because of a circular import through pathspec. The script keeps a small table of such conflicts: it removes the blocking module, builds the port, and puts the module back. Failing to put it back is recorded as a failure.

The vulnerability check

make refuses to build a port that pkg audit lists, so a migration can stall on a package that has nothing to do with Python. The script exports DISABLE_VULNERABILITIES=yes and says so loudly; KEEP_VULN_CHECK=1 turns that off. Run pkg audit -F yourself afterwards.

The meta port that stayed behind

On two hosts python-3.11_3,2 stayed registered after the migration, and I only noticed because portmaster offered to upgrade it. My script had skipped the meta port because /usr/local/bin/python already pointed at 3.12. The symlink was right, the package was not, and deleting the old package removes /usr/local/bin/python with it. The script now compares the package version as well and rebuilds lang/python when it still carries the old one.

Scripts that grep -r cannot see

On one host, pythontex, latexminted and a dozen other TeX Live scripts still pointed at python3.11 after the run completed with no warning. They are symlinks into /usr/local/share, and grep -r does not follow symlinks while it walks a tree. The script now builds its file list with find -L. This was the only case where the final check itself was blind, so it is the one I would test first in my own setup.

Files that no package owns

Three kinds of leftover remained in /usr/local/lib/python3.11:

• Files owned by packages that are not modules: xcb-proto installs xcbgen there, and at-spi2-core installs gi files. They have no py311 in their names, so only a scan of the old tree finds them.
• Orphans from pip install outside pkg (autopep8, paramiko and others). Nothing migrates them. Reinstall them from ports, or put them in a virtual environment.
• Leftover __pycache__ directories and other debris.

The approach: inventory first, then a fixed order

The script is built on two rules. Everything that has to be rebuilt is written down before anything is deleted, and every step can be repeated until the machine is clean. The work directory /var/tmp/pyupgrade-<old>-<new> holds the inventory and the list of finished origins, so a second run skips what is done and picks up where the last one stopped.

1. Checks and a way back. The script refuses to start in a state it does not recognize, takes a boot environment with bectl when it is available, and backs up make.conf. DRYRUN=1 prints every command without running it.
2. Inventory. It records all installed py<old>-* modules with their origins and flavors, then every other package that needs a rebuild: users of libpython, packages that depend on a module, owners of scripts with the old shebang, and owners of files in the old interpreter's tree.
3. DEFAULT_VERSIONS. It edits the python and python3 lines in /etc/make.conf one knob at a time, so python= never matches python3=.
4. The interpreter. portmaster -o lang/python<new> lang/python<old> installs the new Python, then the pythonand python3 meta ports are rebuilt if the symlink or the package version is still old.
5. Bootstrap packages. Stale *-bootstrap packages are deleted and rebuilt, before any build that could depend on them.
6. Modules. Old py<old>-* packages are removed with pkg delete -f, and the new origins are built, flavor-aware, with the conflict table applied.
7. Dependents. Every other package from the inventory is rebuilt one at a time with make clean stage deinstall install clean. I avoid portmaster -f here because it forces a rebuild of every dependency of each port as well.
8. Final checks. pkg check -d -n -a, modules without a new counterpart, leftover libpython users, scripts that still name the old interpreter, leftover py<old> packages, and the list of failed ports. A non-empty failure list makes the exit status 2.

The order matters in two places. The new interpreter must exist before any old module is removed, otherwise nothing can be built. Bootstrap packages must be fixed before the modules, because rebuilding anything that needs glib fails while its bootstrap packages carry a dead shebang.

The script never runs pkg autoremove. Whether something is no longer needed is a decision I want to make myself, from the output of pkg autoremove -n.

The script

upgrade_python.sh takes the current and the target version. Start with a dry run, which prints every command and changes nothing, then run it for real inside tmux or screen, because a migration compiles for hours:

DRYRUN=1 ./upgrade_python.sh 3.11 3.12
./upgrade_python.sh 3.11 3.12 2>&1 | tee /tmp/python.log

If a port fails, fix the cause and run the same command again. A dry run cannot be used for a resumed migration, because the old packages are already gone by then.

It assumes portmaster and a ports tree in /usr/ports, or wherever PORTSDIR points, so it is not a poudriere recipe. A poudriere build happens in a clean jail, which would also have avoided the py-google-re2 conflict described above, but I have not tried a migration that way.

It was written and corrected against my own machines. It has not been run anywhere else, so read it before you trust it, and take a boot environment or a ZFS snapshot first. On one of my hosts bectl could not create a boot environment because the root dataset was the pool itself, so I used zfs snapshot there.

#!/bin/sh
#
# FreeBSD Python major version upgrade helper (portmaster based).
#
# Usage:   ./upgrade_python.sh <current_version> <target_version>
# Example: ./upgrade_python.sh 3.14 3.15
#
# Environment:
#   DRYRUN=1   print the commands instead of running them
#   MAKE_ARGS  extra arguments for the make(1) calls, e.g. MAKE_ARGS="BATCH=yes"
#              to accept default OPTIONS instead of showing config dialogs
#   KEEP_VULN_CHECK=1
#              keep the vulnerability check of the ports framework. By default
#              DISABLE_VULNERABILITIES=yes is exported so that vulnerable ports
#              do not stop the run.
#
# Flow:
#   0. Sanity checks, boot environment snapshot, backup of make.conf
#   1. Collect everything that has to be rebuilt (BEFORE touching anything)
#   2. Update DEFAULT_VERSIONS in /etc/make.conf
#   3. Switch the interpreter itself (portmaster -o)
#   4. Delete old pyXY-* modules and rebuild them as pyZZ-* (flavor-aware)
#   5. Rebuild non-python packages still linked against the old libpython
#   6. Final consistency checks
#
# The script is resumable: finished origins are recorded in a work directory
# and skipped on the next run with the same arguments.

set -u

MAKE_CONF="/etc/make.conf"

# ---------------------------------------------------------------- helpers

die() {
    echo "[-] ERROR: $*" >&2
    exit 1
}

info() {
    echo "[+] $*"
}

warn() {
    echo "[!] $*" >&2
}

# Run a command, or only print it when DRYRUN=1.
run() {
    if [ "${DRYRUN:-0}" = "1" ]; then
        # printf, not echo: some echo implementations eat backslashes (\1 in sed)
        printf 'DRYRUN: %s\n' "$*"
        return 0
    fi
    "$@"
}

# List text files below the given paths whose shebang names the old interpreter.
# "grep -r" does not follow symlinks while walking a tree, and many scripts in
# /usr/local/bin are symlinks (texlive: pythontex, latexminted, ... point into
# /usr/local/share/texmf-dist), so they would never be found. find -L follows
# them; -I skips binaries.
stale_shebang_files() {
    find -L "$@" -type f -print0 2>/dev/null \
        | xargs -0 -r grep -lIE "^#!.*python${CUR_VER}" 2>/dev/null
}

# Convert "name-version" to an origin; pass real origins through unchanged.
to_origin() {
    case "$1" in
        */*) echo "$1" ;;
        *)   pkg query '%o' "$1" 2>/dev/null | head -n 1 ;;
    esac
}

# Rebuild exactly ONE port, without touching its dependencies.
#
# Do not use "portmaster -f" for this: -f forces every port that portmaster
# walks, i.e. the whole dependency tree of the target. In a real 3.11 -> 3.12
# run it recompiled python312, pkg, libiconv, libffi, muon, samurai, ... again
# for every single target (about two hours of wasted compile time).
#
# The new version is built and staged first, so the old package stays installed
# until the replacement is ready; only the short deinstall + install is exposed.
rebuild_one() {
    _origin="$1"
    _flavor=""
    case "${_origin}" in
        *@*) _flavor="${_origin#*@}"; _origin="${_origin%@*}" ;;
    esac
    _dir="${PORTSDIR:-/usr/ports}/${_origin}"
    [ -d "${_dir}" ] || { warn "no such port directory: ${_dir}"; return 1; }

    if [ -n "${_flavor}" ]; then
        set -- "FLAVOR=${_flavor}"
    else
        set --
    fi

    # shellcheck disable=SC2086  # MAKE_ARGS is meant to be word-split
    run make -C "${_dir}" ${MAKE_ARGS:-} "$@" clean || return 1
    # shellcheck disable=SC2086
    run make -C "${_dir}" ${MAKE_ARGS:-} "$@" stage || return 1

    # A flavored port whose old flavor is a differently named package (scons-py311
    # -> scons-py312) is not replaced by "deinstall": that only looks at the new
    # name, and pkg then refuses the install because both packages own the same
    # files (/usr/local/bin/scons). Remove the old-flavor package first, now that
    # the replacement has been built and staged.
    if [ -n "${_flavor}" ]; then
        for _old in $(pkg query '%o %n' 2>/dev/null | awk -v o="${_origin}" '$1 == o { print $2 }'); do
            _ofl=$(pkg query '%At %Av' "${_old}" 2>/dev/null | awk '$1 == "flavor" { print $2; exit }')
            case "${_ofl}" in
                *py${CUR_NODOT}*)
                    info "Removing old-flavor package ${_old} (${_ofl}) before installing ${_origin}@${_flavor}"
                    run pkg delete -f -y "${_old}" || warn "could not delete ${_old}" ;;
            esac
        done
    fi

    # shellcheck disable=SC2086
    run make -C "${_dir}" ${MAKE_ARGS:-} "$@" deinstall install clean
}

# Print an origin as it exists in the ports tree. The origin recorded in the pkg
# database can differ from the directory name only by case (an old package had
# devel/py-automat, the tree has devel/py-Automat), and the directory is then
# not found on a case sensitive file system. Flavors ("@py312") are kept.
resolve_origin() {
    _ro="$1"
    _rf=""
    case "${_ro}" in
        *@*) _rf="@${_ro#*@}"; _ro="${_ro%@*}" ;;
    esac
    if [ ! -d "${PORTSDIR}/${_ro}" ]; then
        _hit=$(find "${PORTSDIR}/${_ro%/*}" -maxdepth 1 -type d -iname "${_ro#*/}" 2>/dev/null | head -n 1)
        [ -n "${_hit}" ] && _ro="${_ro%/*}/${_hit##*/}"
    fi
    echo "${_ro}${_rf}"
}

# Print an origin, with "@<new flavor>" appended when the installed package was
# built with a flavor of the OLD python version.
#
# pkg reports origins without a flavor, but some ports are flavored without a
# py<ver>- name prefix (scons-py311 is devel/scons@py311). Rebuilding the bare
# origin would use the default flavor, not the one that replaces the old package.
# The flavor comes from the package's "flavor" annotation. Needs ORIGIN_NAMES_FILE
# (lines of "<origin> <package name>") and CUR_NODOT/TGT_NODOT to be set.
add_flavor() {
    _o="$1"
    _fl=""
    _pn=$(awk -v o="${_o}" '$1 == o { print $2; exit }' "${ORIGIN_NAMES_FILE}" 2>/dev/null)
    if [ -n "${_pn}" ]; then
        _fl=$(pkg query '%At %Av' "${_pn}" 2>/dev/null | awk '$1 == "flavor" { print $2; exit }')
    fi
    case "${_fl}" in
        *py${CUR_NODOT}*)
            echo "${_o}@$(echo "${_fl}" | sed "s/py${CUR_NODOT}/py${TGT_NODOT}/g")" ;;
        *)
            echo "${_o}" ;;
    esac
}

# Known build-time conflicts: ports that fail to build while a certain python
# module is installed. Prints the origin of the module that has to be out of the
# way. Add new cases here as they turn up.
#
#  - py-google-re2: its work directory contains a "re2" package that shadows the
#    real one. py-scikit-build-core registers a setuptools plugin that imports
#    pathspec, which does "import re2", finds the half-built source package and
#    dies with "cannot import name '_re2' ... circular import". A poudriere build
#    never sees this because the jail only holds the declared dependencies.
conflict_for() {
    case "${1%@*}" in
        devel/py-google-re2|devel/py-google-re2-stubs) echo "devel/py-scikit-build-core" ;;
    esac
}

# Build one python module with portmaster. If it is on the conflict list, the
# conflicting module is deleted first and put back afterwards (also when the
# build fails). The restore uses rebuild_one, which compiles nothing heavy for the
# pure-python modules this is meant for.
build_module() {
    _bm_origin="$1"
    _bm_conf=$(conflict_for "${_bm_origin}")
    _bm_removed=""
    if [ -n "${_bm_conf}" ]; then
        _bm_pkg=$(pkg query -g '%o %n' "py${TGT_NODOT}-*" 2>/dev/null \
            | awk -v o="${_bm_conf}" '$1 == o { print $2; exit }')
        if [ -n "${_bm_pkg}" ]; then
            info "Temporarily removing ${_bm_pkg}: it breaks the build of ${_bm_origin}"
            run pkg delete -f -y "${_bm_pkg}" && _bm_removed="${_bm_conf}"
        fi
    fi
    _bm_rc=0
    run portmaster --no-confirm "${_bm_origin}" || _bm_rc=1
    if [ -n "${_bm_removed}" ]; then
        info "Restoring ${_bm_removed}"
        if ! rebuild_one "${_bm_removed}@py${TGT_NODOT}"; then
            warn "could not restore ${_bm_removed}; reinstall it by hand"
            echo "${_bm_removed}@py${TGT_NODOT}" >> "${FAILED_FILE}"
        fi
    fi
    return "${_bm_rc}"
}

# ------------------------------------------------------------- arguments

if [ $# -ne 2 ]; then
    echo "Usage: $0 <current_version> <target_version>"
    echo "Example: $0 3.14 3.15"
    exit 1
fi

CUR_VER="$1"
TGT_VER="$2"

case "${CUR_VER}" in
    [0-9].[0-9]|[0-9].[0-9][0-9]) ;;
    *) die "invalid current version '${CUR_VER}' (expected e.g. 3.14)" ;;
esac
case "${TGT_VER}" in
    [0-9].[0-9]|[0-9].[0-9][0-9]) ;;
    *) die "invalid target version '${TGT_VER}' (expected e.g. 3.15)" ;;
esac
[ "${CUR_VER}" != "${TGT_VER}" ] || die "current and target versions are the same"

# 3.14 -> 314: used in package names, flavors and origins
CUR_NODOT=$(echo "${CUR_VER}" | tr -d '.')
TGT_NODOT=$(echo "${TGT_VER}" | tr -d '.')

OLD_LIB="libpython${CUR_VER}.so.1.0"

# ---------------------------------------------------------- sanity checks

[ "$(id -u)" -eq 0 ] || die "must be run as root"
command -v pkg >/dev/null 2>&1 || die "pkg not found"
command -v portmaster >/dev/null 2>&1 || die "portmaster not found (ports-mgmt/portmaster)"

# The old interpreter is removed by step 3, so a resumed run will not find it.
# That is fine as long as the new interpreter is installed and the work directory
# of the first run (with the saved module and dependents lists) still exists.
if ! pkg info -e "python${CUR_NODOT}"; then
    if pkg info -e "python${TGT_NODOT}" \
        && [ -f "/var/tmp/pyupgrade-${CUR_NODOT}-${TGT_NODOT}/.started" ]; then
        info "python${CUR_NODOT} is gone but python${TGT_NODOT} is installed: resuming"
        if [ "${DRYRUN:-0}" = "1" ]; then
            die "DRYRUN cannot be used when resuming: the old packages are gone, so nothing could be collected"
        fi
    else
        die "package python${CUR_NODOT} is not installed"
    fi
fi

# The target interpreter must exist in the ports tree and be a valid python version
PORTSDIR="${PORTSDIR:-/usr/ports}"
[ -d "${PORTSDIR}/lang/python${TGT_NODOT}" ] \
    || die "${PORTSDIR}/lang/python${TGT_NODOT} not found - update the ports tree first"
grep -qF "${TGT_VER}" "${PORTSDIR}/Mk/Uses/python.mk" 2>/dev/null \
    || warn "python.mk does not mention ${TGT_VER}; DEFAULT_VERSIONS may be rejected"
info "Python ${CUR_VER} is installed, migrating to ${TGT_VER}"

# The ports framework refuses to build a port that pkg audit lists as vulnerable.
# Many of the ports rebuilt here are, and the whole run then stops at the first
# one. The variable is exported, so portmaster and every make call below see it.
# make only tests whether it is DEFINED, so "=no" would not turn the check back
# on; KEEP_VULN_CHECK=1 leaves it unset instead.
if [ "${KEEP_VULN_CHECK:-0}" = "1" ]; then
    unset DISABLE_VULNERABILITIES
    info "Vulnerability check stays enabled (KEEP_VULN_CHECK=1)"
else
    DISABLE_VULNERABILITIES=yes
    export DISABLE_VULNERABILITIES
    warn "DISABLE_VULNERABILITIES=yes: ports listed by pkg audit will still be built"
fi

WORKDIR="/var/tmp/pyupgrade-${CUR_NODOT}-${TGT_NODOT}"
# Keep dry-run state separate so it can never make a real run skip work
if [ "${DRYRUN:-0}" = "1" ]; then
    WORKDIR="${WORKDIR}-dryrun"
    rm -rf "${WORKDIR}"
fi
mkdir -p "${WORKDIR}" || die "cannot create ${WORKDIR}"
MODULES_FILE="${WORKDIR}/modules.txt"        # old package names (pyOLD-*)
ORIGINS_FILE="${WORKDIR}/new_origins.txt"    # origins for the new flavor
DEPENDENTS_FILE="${WORKDIR}/dependents.txt"  # non-python packages linked to old lib
DONE_FILE="${WORKDIR}/done.txt"              # origins already rebuilt
FAILED_FILE="${WORKDIR}/failed.txt"
touch "${DONE_FILE}"
: > "${FAILED_FILE}"

# ------------------------------------------- snapshot and backups (first run)

if [ ! -f "${WORKDIR}/.started" ]; then
    if command -v bectl >/dev/null 2>&1; then
        BE_NAME="pre-python${CUR_NODOT}-to-${TGT_NODOT}-$(date +%Y%m%d%H%M)"
        info "Creating boot environment ${BE_NAME}"
        run bectl create "${BE_NAME}" || warn "bectl create failed (not on ZFS?)"
    else
        warn "bectl not available - no boot environment snapshot was taken"
    fi
    run cp -p "${MAKE_CONF}" "${MAKE_CONF}.pre-python${TGT_NODOT}" 2>/dev/null || true
    touch "${WORKDIR}/.started"
fi

# -------------------------------------------- 1. collect what to rebuild

if [ ! -s "${MODULES_FILE}" ]; then
    info "Collecting installed py${CUR_NODOT}-* packages"
    pkg query -g '%n-%v %o' "py${CUR_NODOT}-*" > "${WORKDIR}/modules_raw.txt" 2>/dev/null || true
    [ -s "${WORKDIR}/modules_raw.txt" ] || warn "no py${CUR_NODOT}-* packages found - is that expected?"

    # Old package names, used for pkg delete
    awk '{print $1}' "${WORKDIR}/modules_raw.txt" | sort -u > "${MODULES_FILE}"

    # New origins: keep the flavor, just swap pyOLD for pyNEW
    awk '{print $2}' "${WORKDIR}/modules_raw.txt" \
        | sed "s/py${CUR_NODOT}/py${TGT_NODOT}/g" | sort -u \
        | while read -r _o; do resolve_origin "${_o}"; done | sort -u > "${ORIGINS_FILE}"
fi

if [ ! -f "${DEPENDENTS_FILE}" ]; then
    info "Collecting non-python packages linked against ${OLD_LIB}"
    : > "${DEPENDENTS_FILE}"
    for p in $(pkg shlib -qR "${OLD_LIB}" 2>/dev/null); do
        # Skip the python modules themselves, they are handled separately
        if grep -qxF "$(pkg query '%n-%v' "${p}" 2>/dev/null | head -n 1)" "${MODULES_FILE}" 2>/dev/null; then
            continue
        fi
        origin=$(to_origin "${p}")
        [ -n "${origin}" ] || continue
        case "${origin}" in
            lang/python${CUR_NODOT}|lang/python${CUR_NODOT}@*) continue ;;
        esac
        echo "${origin}" >> "${DEPENDENTS_FILE}"
    done
    # Non-python packages that merely depend on the python modules (glib, gtk-doc,
    # itstool, ...). pkg delete -f leaves their dependency records dangling, so
    # they have to be rebuilt as well.
    pkg query -g '%rn %ro' "py${CUR_NODOT}-*" 2>/dev/null \
        | awk -v pfx="py${CUR_NODOT}-" 'index($1, pfx) != 1 {print $2}' >> "${DEPENDENTS_FILE}"
    # Scripts with the old interpreter hard-coded in the shebang (scons, rrsync,
    # graphviz, ...). Their owners need a rebuild even without a py<ver> dependency.
    stale_shebang_files /usr/local/bin /usr/local/sbin \
        | xargs -r pkg which -qo 2>/dev/null >> "${DEPENDENTS_FILE}"
    # The shebang scan also finds two things that must NOT be rebuilt here:
    #  - the old interpreter itself (its own pydoc/idle/2to3 scripts carry the old
    #    shebang). "Rebuilding" lang/python${CUR_NODOT} would install the old Python
    #    again next to the new one.
    #  - python modules that ship scripts (poetry, keyring, breathe, ...). They are
    #    rebuilt in step 4 and would otherwise be queued a second time.
    awk -v cur="lang/python${CUR_NODOT}" '
        FILENAME == ARGV[1] { o = $0; sub(/@.*/, "", o); skip[o] = 1; next }
        { o = $0; sub(/@.*/, "", o); if (o != cur && !(o in skip)) print }
    ' "${ORIGINS_FILE}" "${DEPENDENTS_FILE}" > "${DEPENDENTS_FILE}.tmp" \
        && mv "${DEPENDENTS_FILE}.tmp" "${DEPENDENTS_FILE}"
    # Re-attach the python flavor for ports like devel/scons (see add_flavor).
    ORIGIN_NAMES_FILE="${WORKDIR}/origin_names.txt"
    pkg query '%o %n' > "${ORIGIN_NAMES_FILE}" 2>/dev/null || : > "${ORIGIN_NAMES_FILE}"
    : > "${DEPENDENTS_FILE}.tmp"
    while read -r _dep; do
        [ -n "${_dep}" ] || continue
        case "${_dep}" in
            *@*) echo "${_dep}" >> "${DEPENDENTS_FILE}.tmp" ;;
            *)   add_flavor "${_dep}" >> "${DEPENDENTS_FILE}.tmp" ;;
        esac
    done < "${DEPENDENTS_FILE}"
    mv "${DEPENDENTS_FILE}.tmp" "${DEPENDENTS_FILE}"
    sort -u "${DEPENDENTS_FILE}" -o "${DEPENDENTS_FILE}"
fi

# Rescan on every run, including resumed ones: dependents.txt is reused as it
# is, so an owner of stale-shebang scripts that an earlier run missed would
# otherwise never be queued. Anything already queued, already rebuilt, being a
# module (step 4) or the old interpreter itself is left out.
touch "${DONE_FILE}"
#
# A second source of owners: non-module packages that install files into the old
# interpreter's own tree (xcb-proto -> xcbgen, at-spi2-core -> gi overrides, ...).
# They have no py<ver> name, do not link libpython and ship no script, so neither
# earlier scan sees them, yet they stay unusable for the new interpreter.
for _o in $( { stale_shebang_files /usr/local/bin /usr/local/sbin | xargs -r pkg which -qo 2>/dev/null
               if [ -d "/usr/local/lib/python${CUR_VER}" ]; then
                   find "/usr/local/lib/python${CUR_VER}" -type f -print0 2>/dev/null | xargs -0 -r pkg which -qo 2>/dev/null
               fi; } | sort -u); do
    case "${_o}" in lang/python${CUR_NODOT}|lang/python${CUR_NODOT}@*) continue ;; esac
    sed 's/@.*//' "${ORIGINS_FILE}" 2>/dev/null | grep -qixF "${_o%@*}" && continue
    grep -qxF "${_o}" "${DEPENDENTS_FILE}" && continue
    grep -qxF "${_o}" "${DONE_FILE}" && continue
    info "Queueing ${_o}: scripts with the old interpreter found on rescan"
    echo "${_o}" >> "${DEPENDENTS_FILE}"
done

info "Modules to migrate:    $(wc -l < "${MODULES_FILE}" | tr -d ' ')"
info "Other dependents:      $(wc -l < "${DEPENDENTS_FILE}" | tr -d ' ')"
info "Work files are in ${WORKDIR}"

# ---------------------------------------- 2. DEFAULT_VERSIONS in make.conf

info "Updating ${MAKE_CONF}"
# make.conf may carry both the "python" and the "python3" knob, often on separate
# lines, e.g.
#   DEFAULT_VERSIONS+=python3=3.14
#   DEFAULT_VERSIONS+=python=3.14
# A plain "python=" pattern does not match "python3=", so each knob is handled on
# its own. Only live (uncommented) DEFAULT_VERSIONS lines are rewritten, so old
# commented-out examples are left alone. The character before the knob name must
# be a blank or "=" so that e.g. "python3=" is never taken for "python=".
for knob in python python3; do
    if grep -Eq "^[[:space:]]*DEFAULT_VERSIONS.*[[:space:]=]${knob}=[0-9]" "${MAKE_CONF}" 2>/dev/null; then
        run sed -i '' -E "/^[[:space:]]*DEFAULT_VERSIONS/s/([[:space:]=])${knob}=[0-9][0-9.]*/\1${knob}=${TGT_VER}/g" "${MAKE_CONF}"
    elif [ "${knob}" = "python" ]; then
        # No python knob at all: add one. python3 is only ever updated, never
        # added, so a make.conf that does not use it stays as it is.
        if [ "${DRYRUN:-0}" = "1" ]; then
            echo "DRYRUN: append 'DEFAULT_VERSIONS+= python=${TGT_VER}' to ${MAKE_CONF}"
        else
            echo "DEFAULT_VERSIONS+= python=${TGT_VER}" >> "${MAKE_CONF}"
        fi
    fi
done
if [ "${DRYRUN:-0}" != "1" ]; then
    info "DEFAULT_VERSIONS python lines now in ${MAKE_CONF}:"
    grep -nE "^[[:space:]]*DEFAULT_VERSIONS.*[[:space:]=]python3?=" "${MAKE_CONF}" 2>/dev/null || true
fi

# ----------------------------------------------- 3. switch the interpreter

if pkg info -e "python${TGT_NODOT}"; then
    info "python${TGT_NODOT} is already installed, skipping interpreter switch"
else
    info "Switching interpreter: lang/python${CUR_NODOT} -> lang/python${TGT_NODOT}"
    run portmaster --no-confirm -o "lang/python${TGT_NODOT}" "lang/python${CUR_NODOT}" \
        || die "interpreter switch failed; fix it and re-run (the script is resumable)"
fi

# The meta ports (python3 first, then python) provide the /usr/local/bin/python3
# and /usr/local/bin/python symlinks. They keep pointing at the old interpreter
# until they are rebuilt, so rebuild them right after the interpreter switch.
# They compile nothing, so this must be instant (see rebuild_one for why not
# "portmaster -f": it re-installed python312 and its whole dependency chain).
for meta in python3 python; do
    if pkg info -e "${meta}"; then
        # Already done on an earlier run: following the symlink chain (python ->
        # python3 -> python3.X) ends at the new interpreter AND the installed
        # package no longer carries the old version. The symlink alone is not
        # enough: another package can leave it pointing at the new interpreter
        # while "python-3.11" is still what pkg has registered for lang/python.
        # (lang/python3 has a version of its own, e.g. 3_4, so only lang/python
        # can fail the second test.)
        _mver=$(pkg query '%v' "${meta}" 2>/dev/null | head -n 1)
        _mold=0
        case "${_mver}" in "${CUR_VER}"*) _mold=1 ;; esac
        if [ "${_mold}" -eq 0 ] \
            && [ "$(readlink -f "/usr/local/bin/${meta}" 2>/dev/null)" = "/usr/local/bin/python${TGT_VER}" ]; then
            info "Meta port lang/${meta} already points to python${TGT_VER}, skipping"
            continue
        fi
        info "Rebuilding meta port lang/${meta}"
        rebuild_one "lang/${meta}" \
            || warn "rebuild of lang/${meta} failed - fix the python/python3 symlinks by hand"
    fi
done

# ------------------ 3b. remove bootstrap packages with a stale interpreter
#
# Build-only packages such as glib-bootstrap and gobject-introspection-bootstrap
# live under /usr/local/<name>-bootstrap, outside /usr/local/bin, so the shebang
# scans elsewhere in this script do not see them. Their scripts (g-ir-scanner,
# glib-mkenums, ...) still point at the old interpreter, and the glib20 configure
# step then fails with "Could not execute g-ir-scanner --version". They are only
# needed while building, so delete them (-f: ignore their dependents) and build
# fresh ones right away. They have to exist before any rebuild_one call: its
# separate "make stage" step does not build missing build dependencies, it fails
# with "depends on file ... g-ir-scanner - not found" (seen on glib20).
# "make install" does build them, so that is what is used here.
# shellcheck disable=SC2086  # the glob is meant to expand
STALE_BOOTSTRAP=$(stale_shebang_files /usr/local/*-bootstrap \
    | xargs -r pkg which -q 2>/dev/null | sort -u)
if [ -n "${STALE_BOOTSTRAP}" ]; then
    # Remember name, origin and flavor of each one before it is deleted.
    BOOTSTRAP_REBUILD=""
    for _bp in ${STALE_BOOTSTRAP}; do
        _bn=$(pkg query '%n' "${_bp}" 2>/dev/null | head -n 1)
        _bo=$(pkg query '%o' "${_bp}" 2>/dev/null | head -n 1)
        _bf=$(pkg query '%At %Av' "${_bp}" 2>/dev/null | awk '$1 == "flavor" { print $2; exit }')
        [ -n "${_bn}" ] && [ -n "${_bo}" ] && BOOTSTRAP_REBUILD="${BOOTSTRAP_REBUILD} ${_bn}:${_bo}${_bf:+@${_bf}}"
    done
    info "Removing bootstrap packages with a stale interpreter: "$(echo ${STALE_BOOTSTRAP})
    # shellcheck disable=SC2086
    run pkg delete -f -y ${STALE_BOOTSTRAP} || warn "pkg delete of bootstrap packages reported problems"
    for _entry in ${BOOTSTRAP_REBUILD}; do
        _bn="${_entry%%:*}"
        _spec="${_entry#*:}"
        _bo="${_spec%@*}"
        _bf=""
        case "${_spec}" in *@*) _bf="${_spec#*@}" ;; esac
        # One may already have been pulled in as a dependency of another.
        if [ "${DRYRUN:-0}" != "1" ] && pkg info -e "${_bn}"; then
            continue
        fi
        info "Rebuilding bootstrap package ${_bn} (${_spec})"
        if [ -n "${_bf}" ]; then set -- "FLAVOR=${_bf}"; else set --; fi
        # shellcheck disable=SC2086
        run make -C "${PORTSDIR}/${_bo}" ${MAKE_ARGS:-} "$@" clean install clean \
            || warn "rebuild of ${_bn} failed; glib20 will not build without it"
    done
fi

# ---------------------------------- 4. delete old modules, rebuild new ones

if [ -s "${MODULES_FILE}" ]; then
    # Only delete modules that are still installed (safe on re-runs)
    TO_DELETE=""
    for m in $(cat "${MODULES_FILE}"); do
        if pkg info -e "${m}"; then
            TO_DELETE="${TO_DELETE} ${m}"
        fi
    done
    if [ -n "${TO_DELETE}" ]; then
        info "Removing old py${CUR_NODOT} modules (pkg delete -f)"
        # shellcheck disable=SC2086
        run pkg delete -f -y ${TO_DELETE} || warn "pkg delete reported problems"
    fi

    info "Rebuilding modules for Python ${TGT_VER}"
    for origin in $(cat "${ORIGINS_FILE}"); do
        if grep -qxF "${origin}" "${DONE_FILE}"; then
            continue
        fi
        # Many modules are already installed by the time their turn comes, because
        # an earlier module pulled them in as a dependency. portmaster rebuilds a
        # port that is named on the command line even if it is up to date, so
        # skip those here (in a real run this was 67 of 143 modules).
        if pkg query -g '%o' "py${TGT_NODOT}-*" 2>/dev/null | grep -qixF "${origin%@*}"; then
            info "Already installed for py${TGT_NODOT}, skipping ${origin}"
            echo "${origin}" >> "${DONE_FILE}"
            continue
        fi
        info "Building ${origin}"
        if build_module "${origin}"; then
            echo "${origin}" >> "${DONE_FILE}"
        else
            warn "FAILED: ${origin}"
            echo "${origin}" >> "${FAILED_FILE}"
        fi
    done
fi

# ------------------------ 5. other packages linked against the old libpython

if [ -s "${DEPENDENTS_FILE}" ]; then
    info "Rebuilding packages that were linked against ${OLD_LIB}"
    for origin in $(cat "${DEPENDENTS_FILE}"); do
        if grep -qxF "${origin}" "${DONE_FILE}"; then
            continue
        fi
        # A resumed run reuses an existing dependents.txt that may predate the
        # filter above, so repeat the two exclusions here.
        case "${origin%@*}" in
            lang/python${CUR_NODOT})
                info "Skipping ${origin}: that is the old interpreter itself"
                continue ;;
        esac
        if sed 's/@.*//' "${ORIGINS_FILE}" 2>/dev/null | grep -qixF "${origin%@*}"; then
            continue    # python module, handled in step 4
        fi
        info "Rebuilding ${origin}"
        # Only this one port: "portmaster -f" would also force-rebuild every
        # dependency of it (python312 and friends) for each of these origins.
        if rebuild_one "${origin}"; then
            echo "${origin}" >> "${DONE_FILE}"
        else
            warn "FAILED: ${origin}"
            echo "${origin}" >> "${FAILED_FILE}"
        fi
    done
fi

# ------------------------------------------------------- 6. final checks

info "Checking package database consistency"
run pkg check -d -n -a || warn "pkg check -d reported missing dependencies"

if [ "${DRYRUN:-0}" != "1" ] && [ -s "${ORIGINS_FILE}" ]; then
    info "Checking that every migrated module has a py${TGT_NODOT} counterpart"
    pkg query -g '%o' "py${TGT_NODOT}-*" 2>/dev/null | sort -u > "${WORKDIR}/installed_new_origins.txt"
    # Compare in lower case: an origin may differ from the installed one only by case.
    tr 'A-Z' 'a-z' < "${ORIGINS_FILE}" | sort -u > "${WORKDIR}/origins_lc.tmp"
    tr 'A-Z' 'a-z' < "${WORKDIR}/installed_new_origins.txt" | sort -u > "${WORKDIR}/installed_lc.tmp"
    MISSING=$(comm -23 "${WORKDIR}/origins_lc.tmp" "${WORKDIR}/installed_lc.tmp")
    if [ -n "${MISSING}" ]; then
        warn "Modules without a py${TGT_NODOT} counterpart:"
        echo "${MISSING}" >&2
    fi
fi

LEFTOVER=$(pkg shlib -qR "${OLD_LIB}" 2>/dev/null || true)
if [ -n "${LEFTOVER}" ]; then
    warn "Packages still linked against ${OLD_LIB}:"
    echo "${LEFTOVER}" >&2
fi

STALE_SCRIPTS=$(stale_shebang_files /usr/local/bin /usr/local/sbin /usr/local/*-bootstrap || true)
if [ -n "${STALE_SCRIPTS}" ]; then
    warn "Scripts still using the old interpreter:"
    echo "${STALE_SCRIPTS}" >&2
fi

LEFTOVER_PY=$(pkg query -g '%n-%v' "py${CUR_NODOT}-*" 2>/dev/null || true)
if [ -n "${LEFTOVER_PY}" ]; then
    warn "Old py${CUR_NODOT} packages still installed:"
    echo "${LEFTOVER_PY}" >&2
fi

if [ -s "${FAILED_FILE}" ]; then
    warn "The following ports failed, fix them and re-run the script:"
    cat "${FAILED_FILE}" >&2
    exit 2
fi

info "Done. Python ${CUR_VER} -> ${TGT_VER} migration finished."
info "Review 'pkg autoremove -n' by hand; it is intentionally not run here."

Lessons and a checklist

The main lesson is that a clean pkg check proves less than it seems to. Packages can be consistent with each other and still run on an interpreter that no longer exists. Every check in the script exists because some earlier check passed on a machine that was still broken.

Before you start:

• Take a boot environment or a ZFS snapshot, and check that it really worked. bectl is not always usable.
• Run git status in /usr/ports and put local changes to Python ports back to upstream, or know exactly why you keep them.
• Run pkg audit -F and decide whether you accept building vulnerable ports, because the ports framework will otherwise stop at the first one.
• Do a dry run, and read the list of modules and dependents it prints.

After the script reports success:

• pkg check -d -n -a. The -n matters: -d -a without it tries to repair dependencies, which is not what you want at this point.
• find -L /usr/local/bin /usr/local/sbin -type f -print0 | xargs -0 grep -lIE '^#!.*python3\.11', with your old version in place of 3.11. Empty output is the goal.
• find /usr/local/lib/python3.11 -type f | xargs pkg which -q | sort -u. Anything listed still belongs to a package that was never rebuilt.
• Look in the old site-packages directory for files that no package owns, and decide for each whether it was installed with pip.
• pkg autoremove -n, read by hand, and then restart the services that run Python code.

Test your own checks. My final check printed nothing on a machine that still had fifteen scripts pointing at a removed interpreter, because it could not see symlinks. A check that has never failed on purpose is not evidence of anything.

A flavor change is a different package, and nothing in the ports tree links the old one to the new one, so for now the migration stays manual. A script mainly makes it repeatable.