Upgrading

This chapter collects the procedures that go beyond a normal ansible-playbook setup.yml run.

Upgrading the operating system

The servers run Debian and get their container stack from different sources depending on the release: Debian 12 (bookworm) hosts take podman from the OpenSUSE Kubic build service because Debian 12’s podman 4.3 is too old, while Debian 13 (trixie) hosts use Debian’s own podman 5.4. The podman role picks the source from the release it finds on the host, and on a trixie host it also replaces any Kubic packages that are still installed. The procedure below therefore ends with a normal playbook run.

Kubic is the reason this upgrade needs care. Its packages are built for Debian testing, against a newer glibc than bookworm ships, so a routine apt upgrade on bookworm can install a conmon or netavark that does not start; the role holds every Kubic package on bookworm for that reason. Its signing key also expires from time to time, after which apt silently keeps using the old index. Moving to Debian’s own packages ends both problems.

Everything is run as root on the server unless stated otherwise. Expect the services to be unavailable from the moment the containers are stopped until the playbook run at the end has finished, which is typically well under an hour.

Preparation

1. Take a backup and a snapshot. Run the Borg backup as usual and create a snapshot of the server in the Hetzner console. The snapshot is the fastest way back if the upgrade goes wrong.

2. Run the playbook once on bookworm so that the podman role has put a hold on every Kubic package, then bring bookworm up to date and reboot. The upgrade has to start from a fully updated system, and the reboot proves that the machine comes back on its own before anything bigger is attempted.

   apt-mark showhold
   apt update
   apt full-upgrade
   apt autoremove --purge
   reboot
   

   showhold must list podman, crun, conmon, netavark, aardvark-dns, containers-common and slirp4netns, and full-upgrade must not touch any of them. Warnings about an expired Kubic signing key are harmless here; the repository is dropped in step 6.

3. Check the packaging state and clean up leftovers.

   dpkg --audit
   find /etc -name '*.dpkg-*' -o -name '*.ucf-*'
   apt list '?obsolete'
   df -h /
   

   Obsolete kernel images can be purged right away. Keep at least a few gigabytes free on /.

4. Install tmux and run the rest of the procedure inside a tmux session, so that a dropped SSH connection does not interrupt apt half-way.

   apt install tmux
   tmux new -s upgrade
   

Upgrade

5. Stop the containers. Podman, conmon and crun are replaced during the upgrade, and stopped containers give a clean state to come back to.

   systemctl stop 'ae-*.service'
   podman ps
   

6. Remove the Kubic packages, then switch the apt sources to trixie and drop the Kubic repository. Removing (not purging) the packages keeps /etc/containers and the container storage under /var/lib/containers in place; only the binaries go, and the containers are stopped anyway. Debian recommends removing foreign packages before a release upgrade, and it spares apt from working around their epochs and their dependency on libassuan0, which trixie no longer has. The Hetzner mirror files live next to the main sources list.

   apt-mark unhold podman crun conmon netavark aardvark-dns containers-common slirp4netns
   apt remove podman crun conmon netavark aardvark-dns containers-common slirp4netns
   rm /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list
   sed -i 's/bookworm/trixie/g' /etc/apt/sources.list /etc/apt/sources.list.d/hetzner-*.list
   grep -r trixie /etc/apt/sources.list /etc/apt/sources.list.d/
   apt update
   

7. Run the upgrade in the two steps Debian recommends, first without new packages, then the full upgrade. Answer configuration file prompts with the default (keep the local version) unless you know better; the playbook rewrites every file it manages anyway.

   apt upgrade --without-new-pkgs
   apt full-upgrade
   

   There is no podman on the machine at this point; the playbook installs Debian’s in step 9. Should a Kubic package have survived step 6 after all, the playbook replaces it too.

8. Reboot into the new kernel.

   reboot
   

   trixie mounts /tmp as tmpfs from the first reboot after the upgrade, limited to half the memory. Run systemctl mask tmp.mount before rebooting if /tmp should stay on disk.

Afterwards

9. Let the playbook install Debian’s container stack, then reboot once more so that the containers come up the normal way, and only then run the whole playbook. Run the playbook from the controller, first the podman role alone so that the package installation can be watched.

   ansible-playbook -i hosts.ini setup.yml --tags podman
   

   The machine booted in step 8 without podman, so quadlet generated no units at boot and nothing started. The role’s daemon-reload makes the units appear, but the roles only start a container whose unit file has changed, and the database roles fail when they cannot reach postgres. A reboot lets quadlet start everything in the usual order; it also proves that an unattended boot works on the new release.

   reboot
   

   After the reboot, podman ps on the server must list the containers again. Then run the whole playbook from the controller.

   ansible-playbook -i hosts.ini setup.yml
   

10. Verify on the server.

    cat /etc/debian_version
    podman version
    podman info --format '{{.Host.NetworkBackend}} {{.Host.DatabaseBackend}} {{.Host.OCIRuntime.Name}}'
    dpkg-query -W -f='${Package} ${db:Status-Status} ${Version}\n' | grep installed | grep debian9999
    systemctl list-units 'ae-*' --state=failed
    podman ps
    ip -br link
    systemctl status ae-nextcloud-data.mount mnt-backup.mount
    

    The grep for debian9999 must come back empty (a line marked config-files is only the leftover configuration of a removed package, step 11 purges it), the network backend must still be netavark, the database backend sqlite, and eth0 must still carry that name (it is pinned by /etc/udev/rules.d/70-persistent-net.rules on Hetzner images). Then log in to the web services and send a test mail.

    A failed ae-postfix.service with address already in use on port 25 means the upgrade installed exim4 as a recommended package. The postfix role purges it on the next playbook run, or do it by hand:

    apt purge exim4-base exim4-config exim4-daemon-light
    systemctl start ae-postfix.service
    

11. Clean up.

    apt purge '?obsolete'
    apt autoremove --purge
    apt purge '?config-files'
    apt modernize-sources
    

Notes

• The Terraform definition in tf/main.tf still names debian-12 as the image. Changing that field on an existing hcloud_server makes Terraform replace the server, so only change it together with a lifecycle { ignore_changes = [image] } block, and only for new servers.

• If the machine does not come back after the reboot, use the Hetzner console and the snapshot from step 1.