nixpkgs' alertmanager uses DynamicUser, so the `alertmanager` user only exists while the service is running. The matrix-alertmanager-urlfile secret is owned by that user; at boot sops-install-secrets cannot resolve it and rejects the whole manifest, leaving /run/secrets empty (no ssh host key, yggdrasil key, tor key, ...). Deploys masked it because the dynamic user was resolvable via nss-systemd at switch time. Create a static alertmanager user/group and disable DynamicUser.
Prometheus
Monitoring over the Yggdrasil network. Requires the yggdrasil clan service on every machine in the instance.
Roles:
server: runs Prometheus and Alertmanager, scraping everynodesmachine by its Yggdrasil address. Optionalmatrix-alertmanagerforwards alerts to Matrix rooms; the bot token and webhook secret are generated as clan vars.nodes: enables theservices.prometheus.exporterslisted inexporters(withopenFirewall) so the server can harvest them.