Per-gateway download/upload/ping from the hourly librespeed runs, with
each link's 7-day median overlaid so the WANSpeedDegraded alert
condition is visible at a glance. Label-driven — new sites appear
automatically once scraped.
iperf3 serves throughput tests from every VLAN and the mesh; an hourly
librespeed run feeds speedtest_* metrics through node_exporter's textfile
collector, and vmalert flags download rates below half the link's own
7-day median so ISP degradation surfaces without per-site thresholds.
cnx.router.proxy serves <service>.<site><n>.cnx.network with a real Let's
Encrypt wildcard obtained via a gateway-scoped TSIG key against ns1; Blocky
resolves the names to the router's LAN address, so they exist only
internally. First user: Omada UI on gw-cnx-1 (omada.cnx1.cnx.network).
sshd carries the admin SSH keys and now also CA-signed host certificates
(TOFU-less verification via the shared openssh-ca); the users service
generates per-machine root passwords (share = false, no prompts).
Reusable cnx.router.* module for the Topton 1U boxes replacing OPNsense:
PPPoE WAN (optionally VLAN-tagged, AIS: 10, secret credentials incl.
username), VLAN-filtering bridge, nftables NAT/firewall with MSS clamp,
Kea DHCP with per-VLAN lease time, Blocky DNS, DHCPv6-PD, CrowdSec with
the ZeroTier mesh whitelisted, optional Omada controller, ZFS disk.
Fleet baseline rides along: admins sops group is now derived for every
machine in clan.nix (secrets encrypt to it from the first vars generate)
and time sync is chrony everywhere instead of systemd-timesyncd.
Grafana 13 externalised its core data sources; nixpkgs no longer bundles
the Elasticsearch datasource, so every DMARC panel failed with "Plugin not
registered". Load the official Grafana-signed elasticsearch plugin via
declarativePlugins, pin the dmarc-ag/dmarc-fo datasource UIDs (Grafana 11+
resolves datasource template variables by UID, not name), and clear stale
uid-less records left by an earlier deploy.
Bump clan-core and nixos-mailserver to 26.05 (NixOS 26.05) and adapt to
the breaking changes surfaced by nix flake check:
- mesh-hosts/clan.nix: read the new shared, instance-scoped zerotier vars
(zerotier-ip-<machine>-zerotier, zerotier-network-zerotier); admit
external members via the controller's native allowedIds.
- monitoring/server.nix: Grafana lost its built-in secret_key default;
mint one via a clan generator and pass it with $__file{}.
- dns/authoritative.nix: services.resolved.extraConfig removed -> settings.
- mail.nix: SNM cert API change (x509.useACMEHost + acme extraDomainNames)
and accounts/dkim option renames.
- docs: mesh runbook updated for the new var paths and allowedIds.
Berwn
kurogeek