Upgrading ========= This chapter collects the procedures that go beyond a normal ``ansible-playbook setup.yml`` run. Upgrading the operating system ------------------------------ The servers run Debian and get their container stack from different sources depending on the release: Debian 12 (bookworm) hosts take podman from the OpenSUSE Kubic build service because Debian 12's podman 4.3 is too old, while Debian 13 (trixie) hosts use Debian's own podman 5.4. The ``podman`` role picks the source from the release it finds on the host, and on a trixie host it also replaces any Kubic packages that are still installed. The procedure below therefore ends with a normal playbook run. Kubic is the reason this upgrade needs care. Its packages are built for Debian testing, against a newer glibc than bookworm ships, so a routine ``apt upgrade`` on bookworm can install a conmon or netavark that does not start; the role holds every Kubic package on bookworm for that reason. Its signing key also expires from time to time, after which apt silently keeps using the old index. Moving to Debian's own packages ends both problems. Everything is run as root on the server unless stated otherwise. Expect the services to be unavailable from the moment the containers are stopped until the playbook run at the end has finished, which is typically well under an hour. Preparation ~~~~~~~~~~~ 1. Take a backup and a snapshot. Run the Borg backup as usual and create a snapshot of the server in the Hetzner console. The snapshot is the fastest way back if the upgrade goes wrong. 2. Run the playbook once on bookworm so that the ``podman`` role has put a hold on every Kubic package, then bring bookworm up to date and reboot. The upgrade has to start from a fully updated system, and the reboot proves that the machine comes back on its own before anything bigger is attempted. .. code-block:: sh apt-mark showhold apt update apt full-upgrade apt autoremove --purge reboot ``showhold`` must list podman, crun, conmon, netavark, aardvark-dns, containers-common and slirp4netns, and ``full-upgrade`` must not touch any of them. Warnings about an expired Kubic signing key are harmless here; the repository is dropped in step 6. 3. Check the packaging state and clean up leftovers. .. code-block:: sh dpkg --audit find /etc -name '*.dpkg-*' -o -name '*.ucf-*' apt list '?obsolete' df -h / Obsolete kernel images can be purged right away. Keep at least a few gigabytes free on ``/``. 4. Install ``tmux`` and run the rest of the procedure inside a tmux session, so that a dropped SSH connection does not interrupt apt half-way. .. code-block:: sh apt install tmux tmux new -s upgrade Upgrade ~~~~~~~ 5. Stop the containers. Podman, conmon and crun are replaced during the upgrade, and stopped containers give a clean state to come back to. .. code-block:: sh systemctl stop 'ae-*.service' podman ps 6. Remove the Kubic packages, then switch the apt sources to trixie and drop the Kubic repository. Removing (not purging) the packages keeps ``/etc/containers`` and the container storage under ``/var/lib/containers`` in place; only the binaries go, and the containers are stopped anyway. Debian recommends removing foreign packages before a release upgrade, and it spares apt from working around their epochs and their dependency on ``libassuan0``, which trixie no longer has. The Hetzner mirror files live next to the main sources list. .. code-block:: sh apt-mark unhold podman crun conmon netavark aardvark-dns containers-common slirp4netns apt remove podman crun conmon netavark aardvark-dns containers-common slirp4netns rm /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list sed -i 's/bookworm/trixie/g' /etc/apt/sources.list /etc/apt/sources.list.d/hetzner-*.list grep -r trixie /etc/apt/sources.list /etc/apt/sources.list.d/ apt update 7. Run the upgrade in the two steps Debian recommends, first without new packages, then the full upgrade. Answer configuration file prompts with the default (keep the local version) unless you know better; the playbook rewrites every file it manages anyway. .. code-block:: sh apt upgrade --without-new-pkgs apt full-upgrade There is no podman on the machine at this point; the playbook installs Debian's in step 9. Should a Kubic package have survived step 6 after all, the playbook replaces it too. 8. Reboot into the new kernel. .. code-block:: sh reboot trixie mounts ``/tmp`` as tmpfs from the first reboot after the upgrade, limited to half the memory. Run ``systemctl mask tmp.mount`` before rebooting if ``/tmp`` should stay on disk. Afterwards ~~~~~~~~~~ 9. Let the playbook install Debian's container stack, then reboot once more so that the containers come up the normal way, and only then run the whole playbook. Run the playbook from the controller, first the ``podman`` role alone so that the package installation can be watched. .. code-block:: sh ansible-playbook -i hosts.ini setup.yml --tags podman The machine booted in step 8 without podman, so quadlet generated no units at boot and nothing started. The role's daemon-reload makes the units appear, but the roles only start a container whose unit file has changed, and the database roles fail when they cannot reach postgres. A reboot lets quadlet start everything in the usual order; it also proves that an unattended boot works on the new release. .. code-block:: sh reboot After the reboot, ``podman ps`` on the server must list the containers again. Then run the whole playbook from the controller. .. code-block:: sh ansible-playbook -i hosts.ini setup.yml 10. Verify on the server. .. code-block:: sh cat /etc/debian_version podman version podman info --format '{{.Host.NetworkBackend}} {{.Host.DatabaseBackend}} {{.Host.OCIRuntime.Name}}' dpkg-query -W -f='${Package} ${db:Status-Status} ${Version}\n' | grep installed | grep debian9999 systemctl list-units 'ae-*' --state=failed podman ps ip -br link systemctl status ae-nextcloud-data.mount mnt-backup.mount The ``grep`` for ``debian9999`` must come back empty (a line marked ``config-files`` is only the leftover configuration of a removed package, step 11 purges it), the network backend must still be ``netavark``, the database backend ``sqlite``, and ``eth0`` must still carry that name (it is pinned by ``/etc/udev/rules.d/70-persistent-net.rules`` on Hetzner images). Then log in to the web services and send a test mail. A failed ``ae-postfix.service`` with ``address already in use`` on port 25 means the upgrade installed exim4 as a recommended package. The ``postfix`` role purges it on the next playbook run, or do it by hand: .. code-block:: sh apt purge exim4-base exim4-config exim4-daemon-light systemctl start ae-postfix.service 11. Clean up. .. code-block:: sh apt purge '?obsolete' apt autoremove --purge apt purge '?config-files' apt modernize-sources Notes ~~~~~ - The Terraform definition in ``tf/main.tf`` still names ``debian-12`` as the image. Changing that field on an existing ``hcloud_server`` makes Terraform replace the server, so only change it together with a ``lifecycle { ignore_changes = [image] }`` block, and only for new servers. - If the machine does not come back after the reboot, use the Hetzner console and the snapshot from step 1.