Terraform¶

The tf/ directory describes the infrastructure a deployment runs on: the server, its firewalls, the SSH keys that may log in, the DNS zone with all of its records, and the reverse DNS entries the mail server needs. Ansible takes over from there and configures the machine.

This chapter is about the parts of that which are not automatic: keeping the state file, and connecting a configuration to a deployment that is already running.

The state file¶

Terraform remembers which real object belongs to which resource in a state file, tf/terraform.tfstate. That file is ignored by git and exists only on the machine of whoever runs tofu apply. There is no remote backend and no second copy.

The consequence is worth spelling out: without the state, Terraform does not know that the server exists. It does not adopt it, it does not warn about it, it builds a second one next to it – and a second zone, with a second set of records, next to the zone that is answering queries today.

Keep a copy of the state next to the Ansible secrets in ansible/<inventory_hostname>/, which the README already asks you to store safely; the two belong together and are needed by the same person on the same bad day. The state contains the values of sensitive variables, including the API token, so treat it exactly like those secrets.

Note

A remote backend – an S3 compatible bucket, or a Terraform state backend of any other kind – is the better long-term answer, because it keeps the state off a single laptop and lets a second person apply. Adding one is a backend block in tf/providers.tf and a tofu init -migrate-state.

The API token¶

Hetzner shut the standalone DNS API down in May 2026; dns.hetzner.com now redirects to the Console and the zones were migrated into the Cloud project. DNS is therefore managed through the Cloud API like everything else, by the official hetznercloud/hcloud provider, and the hetznerdns_token variable is gone from tf/variables.tf.

An old DNS token does not authenticate against the Cloud API. Create a fresh token in https://console.hetzner.cloud/, in the project that holds these resources, under “Security” and then “API tokens”, with read and write permissions. tf/secrets.auto.tfvars then holds a single entry:

hcloud_token = "..."

Initialising the working directory¶

cd tf
tofu init

The checked-in .terraform.lock.hcl records which provider versions were resolved and their checksums, so everybody gets the same ones. Pass -upgrade only when the constraints in tf/providers.tf have moved, and commit the refreshed lock file afterwards.

Note

The lock file records checksums per platform. If you work on a different operating system or architecture than whoever last ran init -upgrade, tofu providers lock -platform=... adds the missing entries instead of replacing the ones that are there.

The ssh-keys directory¶

tf/ssh-keys/ holds one <name>.pub per key that may log in, and main.tf turns each file into an hcloud_ssh_key and puts all of them on the server. The directory is gitignored, so a fresh checkout does not have it – restore it alongside the state file and the Ansible secrets.

This matters more than it looks, because fileset on a directory that does not exist returns an empty set rather than an error. Silently, an empty ssh-keys/ means a fresh apply builds a server with no key on it at all (Hetzner then mails a root password, and nothing else gets you in), and an apply against imported state destroys every SSH key that was just imported. main.tf therefore carries a precondition that stops any plan while the directory is empty. If you see

No public keys in tf/ssh-keys/.

that is what happened; put the keys back rather than removing the check.

Adopting an existing deployment¶

If the deployment predates the state file – or the state was lost – every object that already exists has to be imported before the first apply. Nothing about this is destructive: an import only writes to the state, and it refuses to overwrite a resource that is already recorded.

tf/import-existing.sh writes the commands for you. It asks the Cloud API what exists, matches each object against the resources this configuration declares, and prints the tofu import lines. It executes nothing, and every request it makes is a read:

cd tf
HCLOUD_TOKEN="..." ./import-existing.sh example.com > import.sh

Read import.sh before you run it. Three parts of the output deserve attention. Records the configuration does not declare – anything created by hand in the Console – are listed at the end under “not declared”, with their names, and are not imported. Record sets that hold more values than the configuration declares get their own section, described below. And SOA and NS records are skipped entirely: Hetzner manages those together with the zone.

A record set holds every value¶

The Cloud API models DNS as one record set per name and type, not as individual records. Every TXT value at the apex therefore lives in the same record set as the SPF record, and every address of a name lives in the same A record set.

Each resource in dns.tf declares exactly one value. Importing a record set that holds more than one hands Terraform the whole set, and the next apply removes everything the configuration does not repeat – a domain verification token for Google or Microsoft, a second address, an old key. This is the one way this workflow can break a working zone without looking dangerous: it shows up in the plan as an in-place update of a record set, not as a destroy, so the “no destroys” rule below does not catch it.

The script prints the live values under every import line and repeats the oversubscribed ones in a section of their own. For the apex TXT set, list the extra values in config.auto.tfvars –

apex_txt_extra = ["google-site-verification=..."]

– and for any other set either add the values to its resource in dns.tf or move them out of the zone before applying.

DKIM comes from Ansible¶

tf/dkim.tf is written by the rspamd role (ansible/roles/rspamd/tasks/generate-dkim.yml) from the keys it generates on the server, and it is gitignored like the state. A fresh checkout does not have it, so the DKIM record sets are not declared and cannot be imported; the script lists them under “not declared” and says so. Run the rspamd role first if you want them managed, then import them in a second pass. Until then the live DKIM records simply stay unmanaged – unmanaged means untouched, so mail keeps being verified either way.

A deployment that publishes a name under a different label – kalender rather than calendar – has to say so in two places, and before the import. In tf/config.auto.tfvars through service_hosts, so that the configuration declares the name at all (see Per-deployment hostnames), and again to the script, which does not read the tfvars:

SERVICE_HOSTS="seam=kalender" HCLOUD_TOKEN="..." \
    ./import-existing.sh example.com

Skip either and the renamed record ends up under “not declared” instead of being matched, and the first apply creates a second record under the default name pointing at the same server.

Then run the result:

sh import.sh

Import ids¶

The script only assembles these; the forms are useful to know when a single resource has to be imported by hand later.

Zone: The zone name (or its numeric id), e.g. example.com.
Record set: <zone>/<name>/<type>, where the name is relative to the zone and the apex is called @, e.g. example.com/mail/A.
Reverse DNS: s-<server id>-<ip address>, e.g. s-132022102-203.0.113.10. The s says the address belongs to a server; the IPv6 entry uses the same shape with the full address.
Server, firewall, SSH key: The numeric id from the Cloud API.

One worked example of each shape:

tofu import 'hcloud_zone.zone' 'example.com'
tofu import 'hcloud_zone_rrset.host_a["postfix"]' 'example.com/mail/A'
tofu import 'hcloud_rdns.mail_ptr_v4' 's-132022102-203.0.113.10'
tofu import 'hcloud_server.compact-1' '132022102'
tofu import 'hcloud_firewall.http_https' '1042786'
tofu import 'hcloud_ssh_key.ssh_keys["laptop"]' '20539371'

The single quotes matter: the addresses of the for_each resources contain double quotes that the shell would otherwise eat.

The first plan¶

Now run a plan, and read it before applying:

tofu plan

Warning

The plan must contain no destroys and no replacement of hcloud_server.compact-1. Terraform replaces a server for a change to an immutable field, and replacing it means deleting the machine with everything on it. If the plan touches the server at all, stop and find out why.

There is one specific way that used to happen, and it is worth knowing about because the trigger is invisible. ssh_keys on hcloud_server forces a new resource, and the Cloud API does not report a running server’s keys back, so the provider leaves the attribute empty on an import. The first plan then compared “no keys” in state against a list of keys in the configuration and proposed to replace the machine. main.tf now carries

lifecycle {
  ignore_changes  = [ssh_keys, public_net]
  prevent_destroy = true
}

The first line removes the false diff, and costs nothing: Hetzner only injects these keys when the server is built, so changing the list on a running server has no effect anyway, and a fresh create still uses the full list because ignore_changes applies only to updates. The second turns any plan that wants to destroy the server into an error instead of a diff to be caught by eye. Lift it deliberately, in the file, when a rebuild really is intended.

public_net is in the same list for a less obvious reason. The provider does not store that block on a read or an import either, so after an import the block in main.tf shows up as an in-place update of the server – which looks harmless. It is not: the provider’s update code for public_net starts by powering the server off, hard, before it compares anything, and powers it back on at the end. With both address families enabled and no explicit primary IP ids it changes nothing in between, so the whole effect is an outage and an unclean shutdown of every container. If a plan ever shows + public_net { ... } on the server, do not apply it; the entry above is what prevents it.

For a deployment that predates this change, a correct plan does show some work:

Creates for names that were never in DNS: www, which the redirect vhost needs a certificate for, and the mail discovery names mta-sts, autoconfig and autodiscover.
Creates for the _dmarc and _mta-sts TXT records.
In-place changes on the zone, whose default TTL goes from 60 to 3600 and whose delete_protection is turned on.
A create for local_file.hosts_cfg, which rewrites ansible/hosts.ini from tf/templates/hosts.ini.tpl. That file is generated, so anything hand-edited into it is lost; per-host settings belong in ansible/host_vars/<host>.yml, which takes precedence over inventory variables anyway.
ttl going from a number to null on imported record sets. Records declare no TTL of their own on purpose, so that the zone default is the only place to change one; null means “inherit”, not “no TTL”. This is the intended direction and safe to apply.

The order of the first run¶

DNS first, then Ansible. Caddy asks Let’s Encrypt for a certificate for every vhost it is configured with, and a vhost whose name does not resolve to this server fails that challenge and is retried, against a rate limit, forever. www, mta-sts, autoconfig and autodiscover are all new names, so running the playbook before the apply means four vhosts in exactly that state.

tofu apply – the records exist and resolve.
ansible-playbook -i hosts.ini setup.yml – Caddy gets its certificates, and the mail-discovery role starts serving the MTA-STS policy at https://mta-sts.<domain>/.well-known/mta-sts.txt.
tofu apply again, if the rspamd role has just written a new tf/dkim.tf.

Publishing _mta-sts before the policy is served is not itself harmful – a sender that cannot fetch the policy carries on without one – but the policy starts in testing mode for a reason, and mta_sts_id in tf/variables.tf has to keep matching ae_mail_discovery_mta_sts_id in the role. Senders re-fetch only when the id changes, so bump both together or the policy change is invisible.

Cleaning up afterwards¶

Two name expressions in the previous dns.tf were wrong in a way that could have created records under nonsensical names – a literal www@, and a calendar with the domain infix run together into calendar<infix>. Neither is declared any more, so Terraform will not delete them. Look for them in the Console and remove them if they are there.

The same holds for every record made by hand. Terraform ignores what it does not declare, so such a record is invisible to plan, survives every apply, and keeps answering queries. Either bring it under management through service_hosts, so the next import or apply owns it, or delete it – but do not leave it as a surprise for the next person reading the plan.

Per-deployment hostnames¶

Not every deployment runs every role, and not every deployment wants the same names. dns.tf carries a default set of subdomain labels, keyed by the Ansible role that serves them, and the service_hosts variable is merged over it. An entry with a new label renames a name, an entry with a new key adds one, and an entry with a null value drops one for a deployment that does not run that role.

In tf/config.auto.tfvars:

domain_name = "example.com"

service_hosts = {
  seam       = "kalender"  # publish the calendar under a German name
  mattermost = null        # this deployment does not run Mattermost
}

A renamed name has to match what Ansible publishes – seam = "kalender" belongs together with the ae_seam_domain override in host_vars – or Caddy answers on a name that DNS never points at, and never gets a certificate for it.

Mail policy records¶

dns.tf publishes what a receiving server checks about mail from this domain: the MX, SPF for the apex (mx -all) and for the mail host itself (a -all, which is what a bounce with an empty sender is checked against), the DKIM keys from dkim.tf, and the DMARC, MTA-STS and TLS reporting records. Three of those are driven by variables in tf/config.auto.tfvars:

dmarc_policy = "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com"
tlsrpt_rua   = "mailto:tlsrpt@example.com"
mta_sts_id   = "1"

dmarc_policy defaults to p=none, which asks receivers to do nothing. Enforcing is safe when every legitimate sender for the domain goes through this server – which is the case for the roles here: users and service accounts authenticate over submission, Mailman rewrites the From header of list posts to the list address (dmarc_mitigate_action = munge_from, unconditionally), and rspamd signs everything that leaves. Start with p=quarantine and a rua address, read the reports for a few weeks, and move to p=reject when nothing legitimate shows up as failing. The report addresses are aliases the lldap role creates and delivers to the admin group, so run the playbook before the apply that publishes them; see Mail Aliases and Shared Mailboxes in the administration chapter.

mta_sts_id has to change together with the policy body the mail-discovery role serves (ae_mail_discovery_mta_sts_id, ae_mail_discovery_mta_sts_mode): senders cache the policy under the id and only fetch it again when the id in DNS differs.