Terraform ========= The ``tf/`` directory describes the infrastructure a deployment runs on: the server, its firewalls, the SSH keys that may log in, the DNS zone with all of its records, and the reverse DNS entries the mail server needs. Ansible takes over from there and configures the machine. This chapter is about the parts of that which are not automatic: keeping the state file, and connecting a configuration to a deployment that is already running. The state file -------------- Terraform remembers which real object belongs to which resource in a state file, ``tf/terraform.tfstate``. That file is ignored by git and exists only on the machine of whoever runs ``tofu apply``. There is no remote backend and no second copy. The consequence is worth spelling out: without the state, Terraform does not know that the server exists. It does not adopt it, it does not warn about it, it builds a second one next to it -- and a second zone, with a second set of records, next to the zone that is answering queries today. Keep a copy of the state next to the Ansible secrets in ``ansible//``, which the README already asks you to store safely; the two belong together and are needed by the same person on the same bad day. The state contains the values of sensitive variables, including the API token, so treat it exactly like those secrets. .. note:: A remote backend -- an S3 compatible bucket, or a Terraform state backend of any other kind -- is the better long-term answer, because it keeps the state off a single laptop and lets a second person apply. Adding one is a ``backend`` block in ``tf/providers.tf`` and a ``tofu init -migrate-state``. The API token ------------- Hetzner shut the standalone DNS API down in May 2026; ``dns.hetzner.com`` now redirects to the Console and the zones were migrated into the Cloud project. DNS is therefore managed through the Cloud API like everything else, by the official ``hetznercloud/hcloud`` provider, and the ``hetznerdns_token`` variable is gone from ``tf/variables.tf``. An old DNS token does not authenticate against the Cloud API. Create a fresh token in https://console.hetzner.cloud/, in the project that holds these resources, under "Security" and then "API tokens", with **read and write** permissions. ``tf/secrets.auto.tfvars`` then holds a single entry: .. code-block:: terraform hcloud_token = "..." Initialising the working directory ---------------------------------- .. code-block:: sh cd tf tofu init The checked-in ``.terraform.lock.hcl`` records which provider versions were resolved and their checksums, so everybody gets the same ones. Pass ``-upgrade`` only when the constraints in ``tf/providers.tf`` have moved, and commit the refreshed lock file afterwards. .. note:: The lock file records checksums per platform. If you work on a different operating system or architecture than whoever last ran ``init -upgrade``, ``tofu providers lock -platform=...`` adds the missing entries instead of replacing the ones that are there. The ssh-keys directory ---------------------- ``tf/ssh-keys/`` holds one ``.pub`` per key that may log in, and ``main.tf`` turns each file into an ``hcloud_ssh_key`` and puts all of them on the server. The directory is gitignored, so a fresh checkout does not have it -- restore it alongside the state file and the Ansible secrets. This matters more than it looks, because ``fileset`` on a directory that does not exist returns an empty set rather than an error. Silently, an empty ``ssh-keys/`` means a fresh apply builds a server with no key on it at all (Hetzner then mails a root password, and nothing else gets you in), and an apply against imported state destroys every SSH key that was just imported. ``main.tf`` therefore carries a precondition that stops any plan while the directory is empty. If you see .. code-block:: text No public keys in tf/ssh-keys/. that is what happened; put the keys back rather than removing the check. Adopting an existing deployment ------------------------------- If the deployment predates the state file -- or the state was lost -- every object that already exists has to be imported before the first apply. Nothing about this is destructive: an import only writes to the state, and it refuses to overwrite a resource that is already recorded. ``tf/import-existing.sh`` writes the commands for you. It asks the Cloud API what exists, matches each object against the resources this configuration declares, and **prints** the ``tofu import`` lines. It executes nothing, and every request it makes is a read: .. code-block:: sh cd tf HCLOUD_TOKEN="..." ./import-existing.sh example.com > import.sh Read ``import.sh`` before you run it. Three parts of the output deserve attention. Records the configuration does not declare -- anything created by hand in the Console -- are listed at the end under "not declared", with their names, and are *not* imported. Record sets that hold more values than the configuration declares get their own section, described below. And SOA and NS records are skipped entirely: Hetzner manages those together with the zone. A record set holds every value ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The Cloud API models DNS as one *record set* per name and type, not as individual records. Every TXT value at the apex therefore lives in the same record set as the SPF record, and every address of a name lives in the same ``A`` record set. Each resource in ``dns.tf`` declares exactly one value. Importing a record set that holds more than one hands Terraform the whole set, and the next apply removes everything the configuration does not repeat -- a domain verification token for Google or Microsoft, a second address, an old key. This is the one way this workflow can break a working zone without looking dangerous: it shows up in the plan as an *in-place update* of a record set, not as a destroy, so the "no destroys" rule below does not catch it. The script prints the live values under every import line and repeats the oversubscribed ones in a section of their own. For the apex TXT set, list the extra values in ``config.auto.tfvars`` -- .. code-block:: terraform apex_txt_extra = ["google-site-verification=..."] -- and for any other set either add the values to its resource in ``dns.tf`` or move them out of the zone before applying. DKIM comes from Ansible ~~~~~~~~~~~~~~~~~~~~~~~ ``tf/dkim.tf`` is written by the rspamd role (``ansible/roles/rspamd/tasks/generate-dkim.yml``) from the keys it generates on the server, and it is gitignored like the state. A fresh checkout does not have it, so the DKIM record sets are not declared and cannot be imported; the script lists them under "not declared" and says so. Run the rspamd role first if you want them managed, then import them in a second pass. Until then the live DKIM records simply stay unmanaged -- unmanaged means untouched, so mail keeps being verified either way. A deployment that publishes a name under a different label -- ``kalender`` rather than ``calendar`` -- has to say so in two places, and *before* the import. In ``tf/config.auto.tfvars`` through ``service_hosts``, so that the configuration declares the name at all (see `Per-deployment hostnames`_), and again to the script, which does not read the tfvars: .. code-block:: sh SERVICE_HOSTS="seam=kalender" HCLOUD_TOKEN="..." \ ./import-existing.sh example.com Skip either and the renamed record ends up under "not declared" instead of being matched, and the first apply creates a second record under the default name pointing at the same server. Then run the result: .. code-block:: sh sh import.sh Import ids ~~~~~~~~~~ The script only assembles these; the forms are useful to know when a single resource has to be imported by hand later. Zone The zone name (or its numeric id), e.g. ``example.com``. Record set ``//``, where the name is relative to the zone and the apex is called ``@``, e.g. ``example.com/mail/A``. Reverse DNS ``s--``, e.g. ``s-132022102-203.0.113.10``. The ``s`` says the address belongs to a server; the IPv6 entry uses the same shape with the full address. Server, firewall, SSH key The numeric id from the Cloud API. One worked example of each shape: .. code-block:: sh tofu import 'hcloud_zone.zone' 'example.com' tofu import 'hcloud_zone_rrset.host_a["postfix"]' 'example.com/mail/A' tofu import 'hcloud_rdns.mail_ptr_v4' 's-132022102-203.0.113.10' tofu import 'hcloud_server.compact-1' '132022102' tofu import 'hcloud_firewall.http_https' '1042786' tofu import 'hcloud_ssh_key.ssh_keys["laptop"]' '20539371' The single quotes matter: the addresses of the ``for_each`` resources contain double quotes that the shell would otherwise eat. The first plan -------------- Now run a plan, and read it before applying: .. code-block:: sh tofu plan .. warning:: The plan must contain **no destroys and no replacement of** ``hcloud_server.compact-1``. Terraform replaces a server for a change to an immutable field, and replacing it means deleting the machine with everything on it. If the plan touches the server at all, stop and find out why. There is one specific way that used to happen, and it is worth knowing about because the trigger is invisible. ``ssh_keys`` on ``hcloud_server`` forces a new resource, and the Cloud API does not report a running server's keys back, so the provider leaves the attribute empty on an import. The first plan then compared "no keys" in state against a list of keys in the configuration and proposed to replace the machine. ``main.tf`` now carries .. code-block:: terraform lifecycle { ignore_changes = [ssh_keys, public_net] prevent_destroy = true } The first line removes the false diff, and costs nothing: Hetzner only injects these keys when the server is built, so changing the list on a running server has no effect anyway, and a fresh create still uses the full list because ``ignore_changes`` applies only to updates. The second turns any plan that wants to destroy the server into an error instead of a diff to be caught by eye. Lift it deliberately, in the file, when a rebuild really is intended. ``public_net`` is in the same list for a less obvious reason. The provider does not store that block on a read or an import either, so after an import the block in ``main.tf`` shows up as an in-place update of the server -- which looks harmless. It is not: the provider's update code for ``public_net`` starts by **powering the server off**, hard, before it compares anything, and powers it back on at the end. With both address families enabled and no explicit primary IP ids it changes nothing in between, so the whole effect is an outage and an unclean shutdown of every container. If a plan ever shows ``+ public_net { ... }`` on the server, do not apply it; the entry above is what prevents it. For a deployment that predates this change, a correct plan does show some work: * Creates for names that were never in DNS: ``www``, which the redirect vhost needs a certificate for, and the mail discovery names ``mta-sts``, ``autoconfig`` and ``autodiscover``. * Creates for the ``_dmarc`` and ``_mta-sts`` TXT records. * In-place changes on the zone, whose default TTL goes from 60 to 3600 and whose ``delete_protection`` is turned on. * A create for ``local_file.hosts_cfg``, which rewrites ``ansible/hosts.ini`` from ``tf/templates/hosts.ini.tpl``. That file is generated, so anything hand-edited into it is lost; per-host settings belong in ``ansible/host_vars/.yml``, which takes precedence over inventory variables anyway. * ``ttl`` going from a number to ``null`` on imported record sets. Records declare no TTL of their own on purpose, so that the zone default is the only place to change one; ``null`` means "inherit", not "no TTL". This is the intended direction and safe to apply. The order of the first run -------------------------- DNS first, then Ansible. Caddy asks Let's Encrypt for a certificate for every vhost it is configured with, and a vhost whose name does not resolve to this server fails that challenge and is retried, against a rate limit, forever. ``www``, ``mta-sts``, ``autoconfig`` and ``autodiscover`` are all new names, so running the playbook before the apply means four vhosts in exactly that state. 1. ``tofu apply`` -- the records exist and resolve. 2. ``ansible-playbook -i hosts.ini setup.yml`` -- Caddy gets its certificates, and the ``mail-discovery`` role starts serving the MTA-STS policy at ``https://mta-sts./.well-known/mta-sts.txt``. 3. ``tofu apply`` again, if the rspamd role has just written a new ``tf/dkim.tf``. Publishing ``_mta-sts`` before the policy is served is not itself harmful -- a sender that cannot fetch the policy carries on without one -- but the policy starts in ``testing`` mode for a reason, and ``mta_sts_id`` in ``tf/variables.tf`` has to keep matching ``ae_mail_discovery_mta_sts_id`` in the role. Senders re-fetch only when the id changes, so bump both together or the policy change is invisible. Cleaning up afterwards ---------------------- Two name expressions in the previous ``dns.tf`` were wrong in a way that could have created records under nonsensical names -- a literal ``www@``, and a ``calendar`` with the domain infix run together into ``calendar``. Neither is declared any more, so Terraform will not delete them. Look for them in the Console and remove them if they are there. The same holds for every record made by hand. Terraform ignores what it does not declare, so such a record is invisible to ``plan``, survives every apply, and keeps answering queries. Either bring it under management through ``service_hosts``, so the next import or apply owns it, or delete it -- but do not leave it as a surprise for the next person reading the plan. Per-deployment hostnames ------------------------ Not every deployment runs every role, and not every deployment wants the same names. ``dns.tf`` carries a default set of subdomain labels, keyed by the Ansible role that serves them, and the ``service_hosts`` variable is merged over it. An entry with a new label renames a name, an entry with a new key adds one, and an entry with a ``null`` value drops one for a deployment that does not run that role. In ``tf/config.auto.tfvars``: .. code-block:: terraform domain_name = "example.com" service_hosts = { seam = "kalender" # publish the calendar under a German name mattermost = null # this deployment does not run Mattermost } A renamed name has to match what Ansible publishes -- ``seam = "kalender"`` belongs together with the ``ae_seam_domain`` override in ``host_vars`` -- or Caddy answers on a name that DNS never points at, and never gets a certificate for it. Mail policy records ------------------- ``dns.tf`` publishes what a receiving server checks about mail from this domain: the MX, SPF for the apex (``mx -all``) and for the mail host itself (``a -all``, which is what a bounce with an empty sender is checked against), the DKIM keys from ``dkim.tf``, and the DMARC, MTA-STS and TLS reporting records. Three of those are driven by variables in ``tf/config.auto.tfvars``: .. code-block:: terraform dmarc_policy = "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com" tlsrpt_rua = "mailto:tlsrpt@example.com" mta_sts_id = "1" ``dmarc_policy`` defaults to ``p=none``, which asks receivers to do nothing. Enforcing is safe when every legitimate sender for the domain goes through this server -- which is the case for the roles here: users and service accounts authenticate over submission, Mailman rewrites the From header of list posts to the list address (``dmarc_mitigate_action = munge_from``, unconditionally), and rspamd signs everything that leaves. Start with ``p=quarantine`` and a ``rua`` address, read the reports for a few weeks, and move to ``p=reject`` when nothing legitimate shows up as failing. The report addresses are aliases the ``lldap`` role creates and delivers to the ``admin`` group, so run the playbook before the apply that publishes them; see :ref:`mail-aliases-and-shared-mailboxes` in the administration chapter. ``mta_sts_id`` has to change together with the policy body the ``mail-discovery`` role serves (``ae_mail_discovery_mta_sts_id``, ``ae_mail_discovery_mta_sts_mode``): senders cache the policy under the id and only fetch it again when the id in DNS differs.