Add mx1 to secret vars/shared/mail-dmarc-cred/passphrase

Add mx1 to secret vars/shared/mail-dmarc-cred/hash
Update vars via generator mail-dmarc-cred for machine control
2026-06-21 03:28:02 +07:00 · 2026-06-21 03:28:01 +07:00 · 2026-06-21 03:27:56 +07:00 · 2026-06-21 03:27:23 +07:00 · 2026-06-21 03:17:51 +07:00 · 2026-06-21 03:08:56 +07:00
162 changed files with 8625 additions and 60 deletions
@@ -37,17 +37,33 @@ deploy ns1 and control.
      regenerable over time and control is the backup server, so this needs a
      second client→server pair (e.g. control→ns2) rather than the same topology

-## 3. Blackbox DNS probing
+## 3. Blackbox DNS probing (done — pending deploy)

- [ ] `blackbox_exporter` on control doing real DNS + DNSSEC-validation queries
+`blackbox_exporter` on control (loopback `:9115`), probing each nameserver's
+public v4+v6 address for every zone: an SOA query (zone served?) and a DNSKEY
+query (still signed?). Blackbox has no DO-bit option, so signing is checked by
+asking for DNSKEY directly and asserting the RRset is present. Probe defs live
+in `modules/monitoring/blackbox-probes.nix`, shared by the exporter
+(`blackbox.nix`) and the VM scrape jobs (`server.nix`). Verified live against
+ns1/ns2: SOA + DNSKEY succeed on both servers over v4 and v6.
+
+- [x] `blackbox_exporter` on control doing real DNS + DNSSEC-validation queries
      against ns1/ns2 — catches outside-in resolution failures the Knot stats miss
+- [x] paired with alerts (`DNSResolutionProbeFailed` / `DNSSECProbeFailed` in
+      `alerts.nix`) and a "DNS probes (outside-in)" row on the CNX DNS dashboard

 ## 4. Third secondary off Hetzner (resilience)

 - [ ] A secondary nameserver on a different provider/network so a single-provider
      outage doesn't take all authoritative DNS down (architectural — new machine)

-## 5. Centralized logs
+## 5. Centralized logs (done — pending deploy)

- [ ] VictoriaLogs on control to grep journald across all three hosts, pairing
+VictoriaLogs on control (`:9428`, 30d retention, mesh-scoped) in
+`modules/monitoring/server.nix`, plus a VictoriaLogs Grafana datasource. All
+three hosts ship journald with systemd's own `services.journald.upload` to the
+`/insert/journald` endpoint (`modules/monitoring/exporters.nix`) — no extra
+agent. control uploads over loopback; ns1/ns2 over the mesh.
+
+- [x] VictoriaLogs on control to grep journald across all three hosts, pairing
      with the existing VictoriaMetrics setup
@@ -1,4 +1,6 @@
 let
+  hosts = import ./modules/hosts.nix;
+
  # This clan-core pins the zerotier `allowedIps` interface (admit by network
  # IPv6), but node IDs are the stable per-device handle (what `zerotier-cli
  # info` prints). Derive a member's IP on THIS network from the controller's
@@ -21,6 +23,8 @@ in
    control = { };
    ns1 = { };
    ns2 = { };
+    mx1 = { };
+    web01 = { };
  };

  inventory.instances = {
@@ -51,14 +55,12 @@ in
    };

    # Direct SSH to public IPs — clan's priority-1 connection path, with the
-    # ZeroTier mesh and Tor kept as automatic fallbacks. Raw IPs (not the
-    # ns1/ns2 DNS names) so reaching these hosts never depends on their own
-    # DNS being up.
-    internet = {
-      roles.default.machines.control.settings.host = "77.42.68.181";
-      roles.default.machines.ns1.settings.host = "46.224.170.206";
-      roles.default.machines.ns2.settings.host = "157.180.70.82";
-    };
+    # ZeroTier mesh and Tor kept as automatic fallbacks. Raw IPs (from
+    # modules/hosts.nix, not the ns1/ns2 DNS names) so reaching these hosts never
+    # depends on their own DNS being up.
+    internet.roles.default.machines = builtins.mapAttrs (_: h: {
+      settings.host = h.ipv4;
+    }) hosts;

    # Recovery root password for console access when a machine fails to boot.
    emergency-access = {
@@ -4,3 +4,4 @@
 - [ZeroTier mesh](./mesh.md)
 - [DNS](./dns.md)
 - [Monitoring](./monitoring.md)
+- [Backups](./backups.md)
@@ -0,0 +1,61 @@
+# Backups
+
+Encrypted, deduplicating backups via clan's `borgbackup` service, declared in
+`clan.nix`. The only critical, non-regenerable state is the **Knot DNSSEC
+keystore** on `ns1` (the KSK/ZSK private keys under `/var/lib/knot`); losing it
+forces an emergency DS rollover at the registrar.
+
+## Topology
+
+- **control** is the borgbackup **server** — it hosts the repos under
+  `/var/lib/borgbackup/<client>` (so `ns1`'s repo is `/var/lib/borgbackup/ns1`).
+- **ns1** is the **client**. It backs up everything it declares as clan state
+  (`clan.core.state.knot.folders = [ "/var/lib/knot" ]`) once a day at 01:00,
+  over the ZeroTier mesh.
+
+The backup is cross-host so that losing `ns1` is recoverable, and stays
+self-contained (no third-party storage). Encryption is `repokey` with a
+generated passphrase, so `control` only ever stores ciphertext.
+
+Mesh peers have no name resolution, so `ns1` maps the `control` machine name to
+its ZeroTier address via `networking.hosts`; that is how the `borg@control` repo
+URL resolves.
+
+## Secrets
+
+The borgbackup ssh keypair and repokey passphrase are clan vars, generated once
+(needs the YubiKey). `control` will not evaluate until `ns1`'s public key
+exists, so generate before the first deploy:
+
+```
+clan vars generate ns1
+clan machines update ns1
+clan machines update control
+```
+
+## Operating
+
+Backups are driven by systemd on `ns1` (`borgbackup-job-control.timer`).
+
+```
+# trigger a backup now (on ns1)
+borgbackup-create
+
+# list archives (on ns1)
+borgbackup-list
+
+# restore selected folders from an archive (on ns1)
+NAME='<archive-name>' FOLDERS=/var/lib/knot borgbackup-restore
+```
+
+Retention is pruned automatically: all archives from the last day, then 7 daily
+and 4 weekly.
+
+## Monitoring
+
+The **CNX Backups** Grafana dashboard
+(`modules/monitoring/dashboards/backups.json`) tracks job health, time since the
+last successful run, and per-job state — all from the node_exporter systemd
+collector on the client. There is no dedicated borg metrics exporter; the unit
+state and the timer's last-trigger timestamp are enough to catch a backup that
+stops running or fails.
@@ -32,18 +32,54 @@ Automatic signing on `ns1` only, policy `cnx`: ECDSA P-256/SHA-256. The ZSK
 auto-rolls; the KSK is kept stable, so the DS at the registrar only changes on a
 manual KSK rollover.

+### Registrar DS records
+
+Knot manages all key material itself on `ns1` (the only signer); the KSK/ZSK
+private keys live in the KASP keystore under `/var/lib/knot` (backed up nightly —
+see [Backups](./backups.md)). You never touch the private keys directly.
+
+What a registrar needs is the **DS record** for a zone's KSK, which anchors the
+zone into the parent's chain of trust. Generate it on `ns1` — the `keymgr` wrapper
+is already pointed at Knot's config, and it runs as the `knot` user that owns the
+keystore:
+
+```
+sudo -u knot keymgr <zone> ds
+```
+
+e.g. `sudo -u knot keymgr cnx.email ds`. Paste the printed DS record (key tag,
+algorithm 13, digest type, digest) into the registrar's DNSSEC form for that
+domain. Repeat per signed zone (`cnx.network`, `buildfor.life`, `cnx.email`) at
+whichever registrar holds each delegation. After submitting, confirm the parent
+publishes it with `dig +short DS <zone>`.
+
+The ZSK rolls automatically and needs no registrar action; only a **KSK rollover**
+requires re-submitting the DS.
+
 > **Pending (manual):** submit DS records for `buildfor.life` and `cnx.email`
 > once they're at a DNSSEC-capable registrar.

 ## ACME DNS-01

-A dedicated TSIG key (`acme_ddns`), scoped by `acl_acme` to `TXT` updates at or
-under `_acme-challenge.<zone>` on `ns1` only. Knot signs the record and transfers
-it to `ns2`, which never needs this key. Retrieve the client config with:
+Certificates are issued by `_acme-challenge` TXT updates that `ns1` accepts over
+TSIG, signs, and transfers to `ns2` (which never needs these keys). Each consumer
+gets its **own** key, scoped by an ACL to exactly the owner names it needs and
+attached only to the zone it lives in — so a leaked key can write nothing but its
+own challenges.

-```
-clan vars get ns1 dns-acme-tsig/acme.conf
-```
+- **`acme_ddns`** (`acl_acme`) — the general key, scoped to `TXT` at or under
+  `_acme-challenge.<zone>` and attached to every zone. Client config:
+  ```
+  clan vars get ns1 dns-acme-tsig/acme.conf
+  ```
+- **`acme_mx1`** (`acl_acme_mx1`) — held only by `mx1`, scoped to
+  `_acme-challenge.{mx1,mta-sts,mail}` and attached only to `cnx.email` (the mail
+  cert plus its MTA-STS and client-alias SANs). Secret shared via the
+  `dns-acme-mx1-secret` generator.
+- **`acme_web01`** (`acl_acme_web01`) — held only by `web01`, scoped to
+  `_acme-challenge` and attached only to `cnx.network` (where the wildcard
+  `*.cnx.network` challenge lands, at the apex). Secret shared via the
+  `dns-acme-web01-secret` generator.

 ## Runbook: stale secondary

@@ -1,6 +1,7 @@
 # Monitoring

-Metrics and dashboards live on `control`, reachable only over the ZeroTier mesh.
+Metrics and logs live on `control` over the ZeroTier mesh; the Grafana dashboards
+are also published publicly through `web01` (see [Dashboards](#dashboards)).

 ## Collection

@@ -9,12 +10,17 @@ Metrics and dashboards live on `control`, reachable only over the ZeroTier mesh.
  (`modules/monitoring/exporters.nix`).
 - **knot-exporter** (`:9433`) on `ns1`/`ns2` only — reads Knot's control socket,
  fed by the `mod-stats` module (query/response counters per zone).
+- **blackbox_exporter** (`127.0.0.1:9115`) on `control` only — outside-in DNS
+  probes. For every zone it queries each nameserver's **public** address (v4 and
+  v6) for SOA (is the zone served?) and DNSKEY (is it still signed?). This is the
+  resolver's-eye view that the Knot stats can't see. Probe definitions are shared
+  between the exporter and the scrape jobs in `modules/monitoring/blackbox-probes.nix`.

 ## Storage & scraping

 **VictoriaMetrics** on `control`, bound to `127.0.0.1:8428`, 180-day retention
-(`modules/monitoring/server.nix`). It scrapes `control` over loopback and `ns1`/
-`ns2` over the mesh.
+(`modules/monitoring/server.nix`). It scrapes `control` over loopback and
+`ns1`/`ns2`/`mx1`/`web01` over the mesh.

 > The scraper dials IPv4-only by default, so mesh (IPv6) targets need
 > `extraOptions = [ "-enableTCP6" ]`. Without it, ns1/ns2 are dropped with
@@ -26,13 +32,70 @@ Metrics and dashboards live on `control`, reachable only over the ZeroTier mesh.

 ## Dashboards

-**Grafana** on `control` (`:3000`), mesh-only, anonymous access disabled. The
-admin password is a clan var:
+**Grafana** on `control` (`:3000`), anonymous access disabled. Reachable directly
+over the mesh, and publicly at `https://grafana.cnx.network` via `web01`'s reverse
+proxy (TLS termination — see [Overview](./overview.md)). The admin password is a
+clan var:

 ```
 clan vars get control grafana-admin/password
 ```

-The provisioned **CNX DNS** dashboard (`modules/monitoring/dashboards/dns.json`)
-shows per-nameserver SOA serials, zone expiry countdowns, query/response rates,
-and host CPU/memory/disk/load.
+Dashboards are provisioned from `modules/monitoring/dashboards/` (any JSON file
+there is picked up):
+
+- **CNX DNS** (`dns.json`) — firing alerts, per-nameserver SOA serials, zone
+  expiry countdowns, query/response rates, host CPU/memory/disk/load, and the
+  outside-in DNS probes.
+- **CNX Backups** (`backups.json`) — borgbackup job health, time since the last
+  run, and per-job state. See [Backups](./backups.md).
+- **CNX Uptime** (`uptime.json`) — per-host up/down status, current uptime,
+  availability over the selected window, and up/down history. Label-driven, so
+  every scraped host appears automatically.
+- **parsedmarc** — DMARC aggregate/forensic report viewer. Auto-provisioned by
+  the `parsedmarc` module (not from `dashboards/`); reads its own Elasticsearch
+  datasource, not VictoriaMetrics. See [DMARC reports](#dmarc-reports) below.
+
+## DMARC reports
+
+The `cnx.email` DMARC record (`rua`/`ruf`) points at the `dmarc@cnx.email`
+mailbox on `mx1`. **parsedmarc** on `control` (`modules/monitoring/parsedmarc.nix`)
+polls that mailbox over IMAPS, parses the XML reports, and stores them in a local
+**Elasticsearch** (`127.0.0.1:9200`, loopback-only); Grafana renders them via the
+auto-provisioned parsedmarc dashboard + Elasticsearch datasource.
+
+The IMAP fetch rides the **mesh**, not the public net: `control` pins
+`mx1.cnx.email` to mx1's mesh address in `/etc/hosts`, so TLS still validates
+against the public cert while the bytes stay on the overlay. The mailbox
+passphrase is the shared `mail-dmarc-cred` clan var (so both mx1's mailserver and
+control's parsedmarc see the same value):
+
+```
+clan vars get mx1 mail-dmarc-cred/passphrase
+```
+
+## Logs
+
+**VictoriaLogs** on `control` (`:9428`), 30-day retention
+(`modules/monitoring/server.nix`). All three hosts ship journald to it via
+systemd's own `services.journald.upload` → the `/insert/journald` endpoint
+(`modules/monitoring/exporters.nix`); no extra agent. `control` uploads over
+loopback so its logs survive a mesh outage, the other hosts push over the mesh,
+and 9428 is firewall-scoped to the mesh like everything else.
+
+> Same IPv4-only default as the scraper: VictoriaLogs binds `0.0.0.0:9428` for a
+> bare `:9428`, so mesh (IPv6) pushes from ns1/ns2 are refused until you pass
+> `extraOptions = [ "-enableTCP6" ]` (binds `[::]`). Verify the bind on `control`:
+>
+> ```
+> ss -tlnp | grep 9428   # want [::]:9428, not 0.0.0.0:9428
+> ```
+
+Query logs from Grafana via the provisioned **VictoriaLogs** datasource (Explore
+view, LogsQL), or directly in the built-in UI at `http://[control]:9428/select/vmui`.
+Logs are tagged with `_HOSTNAME` and `_SYSTEMD_UNIT`, so to follow one service
+across hosts:
+
+```
+_SYSTEMD_UNIT:"knot.service"
+```
@@ -6,11 +6,13 @@ this book is built from `docs/` and served on `control` over the ZeroTier mesh.

 ## Machines

-| Machine   | Role                                   | Public IPv4      | Public IPv6                 |
-| --------- | -------------------------------------- | ---------------- | --------------------------- |
-| `control` | ZeroTier controller, monitoring, docs  | `77.42.68.181`   | `2a01:4f9:c013:e6d0::1`     |
-| `ns1`     | Knot DNS **primary** (master)          | `46.224.170.206` | `2a01:4f8:c014:b5c5::1`     |
-| `ns2`     | Knot DNS **secondary** (slave)         | `157.180.70.82`  | `2a01:4f9:c014:6d87::1`     |
+| Machine   | Role                                   | Public IPv4      | Public IPv6             |
+| --------- | -------------------------------------- | ---------------- | ----------------------- |
+| `control` | ZeroTier controller, monitoring, docs  | `77.42.68.181`   | `2a01:4f9:c013:e6d0::1` |
+| `ns1`     | Knot DNS **primary** (master)          | `46.224.170.206` | `2a01:4f8:c014:b5c5::1` |
+| `ns2`     | Knot DNS **secondary** (slave)         | `157.180.70.82`  | `2a01:4f9:c014:6d87::1` |
+| `mx1`     | Mail server (**MX** for cnx.email)     | `5.223.65.38`    | `2a01:4ff:2f0:1963::1`  |
+| `web01`   | Public reverse proxy (TLS termination) | `5.223.55.246`   | `2a01:4ff:2f0:2d8f::1`  |

 ## Access

@@ -1,5 +1,21 @@
 {
  "nodes": {
+    "blobs": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1604995301,
+        "narHash": "sha256-wcLzgLec6SGJA8fx1OEN1yV/Py5b+U5iyYpksUY/yLw=",
+        "owner": "simple-nixos-mailserver",
+        "repo": "blobs",
+        "rev": "2cccdf1ca48316f2cfd1c9a0017e8de5a7156265",
+        "type": "gitlab"
+      },
+      "original": {
+        "owner": "simple-nixos-mailserver",
+        "repo": "blobs",
+        "type": "gitlab"
+      }
+    },
    "clan-core": {
      "inputs": {
        "data-mesher": "data-mesher",
@@ -73,6 +89,22 @@
        "type": "github"
      }
    },
+    "flake-compat": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1767039857,
+        "narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=",
+        "owner": "edolstra",
+        "repo": "flake-compat",
+        "rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab",
+        "type": "github"
+      },
+      "original": {
+        "owner": "edolstra",
+        "repo": "flake-compat",
+        "type": "github"
+      }
+    },
    "flake-parts": {
      "inputs": {
        "nixpkgs-lib": [
@@ -94,6 +126,54 @@
        "type": "github"
      }
    },
+    "git-hooks": {
+      "inputs": {
+        "flake-compat": [
+          "nixos-mailserver",
+          "flake-compat"
+        ],
+        "gitignore": "gitignore",
+        "nixpkgs": [
+          "nixos-mailserver",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1772893680,
+        "narHash": "sha256-JDqZMgxUTCq85ObSaFw0HhE+lvdOre1lx9iI6vYyOEs=",
+        "owner": "cachix",
+        "repo": "git-hooks.nix",
+        "rev": "8baab586afc9c9b57645a734c820e4ac0a604af9",
+        "type": "github"
+      },
+      "original": {
+        "owner": "cachix",
+        "repo": "git-hooks.nix",
+        "type": "github"
+      }
+    },
+    "gitignore": {
+      "inputs": {
+        "nixpkgs": [
+          "nixos-mailserver",
+          "git-hooks",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1709087332,
+        "narHash": "sha256-HG2cCnktfHsKV0s4XW83gU3F57gaTljL9KNSuG6bnQs=",
+        "owner": "hercules-ci",
+        "repo": "gitignore.nix",
+        "rev": "637db329424fd7e46cf4185293b9cc8c88c95394",
+        "type": "github"
+      },
+      "original": {
+        "owner": "hercules-ci",
+        "repo": "gitignore.nix",
+        "type": "github"
+      }
+    },
    "nix-darwin": {
      "inputs": {
        "nixpkgs": [
@@ -144,6 +224,30 @@
        "type": "github"
      }
    },
+    "nixos-mailserver": {
+      "inputs": {
+        "blobs": "blobs",
+        "flake-compat": "flake-compat",
+        "git-hooks": "git-hooks",
+        "nixpkgs": [
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1773912645,
+        "narHash": "sha256-QHzRqq6gh+t3F/QU9DkP7X63dDDcuIQmaDz12p7ANTg=",
+        "owner": "simple-nixos-mailserver",
+        "repo": "nixos-mailserver",
+        "rev": "25e6dbb8fca3b6e779c5a46fd03bd760b2165bb5",
+        "type": "gitlab"
+      },
+      "original": {
+        "owner": "simple-nixos-mailserver",
+        "ref": "nixos-25.11",
+        "repo": "nixos-mailserver",
+        "type": "gitlab"
+      }
+    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1778003029,
@@ -163,6 +267,7 @@
    "root": {
      "inputs": {
        "clan-core": "clan-core",
+        "nixos-mailserver": "nixos-mailserver",
        "nixpkgs": [
          "clan-core",
          "nixpkgs"
@@ -3,6 +3,9 @@
  inputs.nixpkgs.follows = "clan-core/nixpkgs";
  inputs.treefmt-nix.url = "github:numtide/treefmt-nix";
  inputs.treefmt-nix.inputs.nixpkgs.follows = "nixpkgs";
+  # Simple NixOS Mailserver, pinned to the branch matching clan-core's nixpkgs.
+  inputs.nixos-mailserver.url = "gitlab:simple-nixos-mailserver/nixos-mailserver/nixos-25.11";
+  inputs.nixos-mailserver.inputs.nixpkgs.follows = "nixpkgs";

  outputs =
    {
@@ -24,6 +24,7 @@

      # No formatter, or reformatting would corrupt them.
      "*.zone" # Knot zone files
+      "docs/book.toml" # mdBook config; no TOML formatter enabled
      "flake.lock"
      ".envrc"
      ".gitignore"
@@ -8,6 +8,12 @@
    },
    "ns2": {
      "installedAt": 1781418857
+    },
+    "mx1": {
+      "installedAt": 1781757322
+    },
+    "web01": {
+      "installedAt": 1781983723
    }
  }
 }
@@ -1,19 +1,25 @@
+{ config, ... }:
+let
+  hosts = import ../../modules/hosts.nix;
+in
 {
  imports = [
    ../../modules/hetzner-firewall.nix
    ../../modules/static-ipv6.nix
    ../../modules/monitoring/exporters.nix
    ../../modules/monitoring/server.nix
+    ../../modules/monitoring/blackbox.nix
    ../../modules/monitoring/alerts.nix
+    ../../modules/monitoring/parsedmarc.nix
    ../../modules/docs.nix
  ];

  clan.core.sops.defaultGroups = [ "admins" ];

-  # Public IPv6; SLAAC doesn't bring it up here.
+  # Public IPv6 (from modules/hosts.nix); SLAAC doesn't bring it up here.
  cnx.staticIPv6 = {
    enable = true;
-    address = "2a01:4f9:c013:e6d0::1";
+    address = hosts.${config.networking.hostName}.ipv6;
  };

  time.timeZone = "Etc/GMT-3"; # UTC+3 (fixed offset, no DST)
@@ -0,0 +1,23 @@
+{ config, inputs, ... }:
+let
+  hosts = import ../../modules/hosts.nix;
+in
+{
+  imports = [
+    inputs.nixos-mailserver.nixosModules.default
+    ../../modules/mail.nix
+    ../../modules/static-ipv6.nix
+    ../../modules/monitoring/exporters.nix
+  ];
+
+  clan.core.sops.defaultGroups = [ "admins" ];
+
+  # Public IPv6 (from modules/hosts.nix); SLAAC doesn't bring it up here.
+  cnx.staticIPv6 = {
+    enable = true;
+    address = hosts.${config.networking.hostName}.ipv6;
+  };
+
+  time.timeZone = "Etc/GMT-8"; # UTC+8 (Singapore, fixed offset, no DST)
+  services.timesyncd.enable = true;
+}
@@ -0,0 +1,50 @@
+# ---
+# schema = "single-disk"
+# [placeholders]
+# mainDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_117494657"
+# ---
+# This file was automatically generated!
+# CHANGING this configuration requires wiping and reinstalling the machine
+{
+
+  boot.loader.grub.efiSupport = true;
+  boot.loader.grub.efiInstallAsRemovable = true;
+  boot.loader.grub.enable = true;
+  disko.devices = {
+    disk = {
+      main = {
+        name = "main-5a0919ffeb6044a39b7d44bba8895ff2";
+        device = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_117494657";
+        type = "disk";
+        content = {
+          type = "gpt";
+          partitions = {
+            "boot" = {
+              size = "1M";
+              type = "EF02"; # for grub MBR
+              priority = 1;
+            };
+            ESP = {
+              type = "EF00";
+              size = "500M";
+              content = {
+                type = "filesystem";
+                format = "vfat";
+                mountpoint = "/boot";
+                mountOptions = [ "umask=0077" ];
+              };
+            };
+            root = {
+              size = "100%";
+              content = {
+                type = "filesystem";
+                format = "ext4";
+                mountpoint = "/";
+              };
+            };
+          };
+        };
+      };
+    };
+  };
+}
@@ -1,11 +1,19 @@
-{ config, lib, pkgs, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
 let
  domains = import ../../modules/dns/domains.nix;
  mesh = import ../../modules/mesh-hosts.nix { inherit config lib; };
+  hosts = import ../../modules/hosts.nix;
 in
 {
  imports = [
    ../../modules/dns/authoritative.nix
+    ../../modules/dns/acme-mx1-secret.nix
+    ../../modules/dns/acme-web01-secret.nix
    ../../modules/static-ipv6.nix
    ../../modules/monitoring/exporters.nix
  ];
@@ -22,20 +30,19 @@ in
  # resolution, so map the control machine name to its ZeroTier mesh address.
  networking.hosts.${mesh.hosts.control} = [ "control" ];

-  # Public IPv6 (matches the ns1 AAAA glue); SLAAC doesn't bring it up here.
+  # Public IPv6 (from modules/hosts.nix; matches the ns1 AAAA glue); SLAAC
+  # doesn't bring it up here.
  cnx.staticIPv6 = {
    enable = true;
-    address = "2a01:4f8:c014:b5c5::1";
+    address = hosts.${config.networking.hostName}.ipv6;
  };

  time.timeZone = "Etc/GMT-1"; # UTC+1 (fixed offset, no DST)
  services.timesyncd.enable = true;

-  # ACME DNS-01 (RFC 2136): a dedicated TSIG key, scoped to ns1 only, that an
-  # external ACME client uses to write _acme-challenge TXT records. acl_acme
-  # (referenced by each zone below) limits the key to TXT updates at or under
-  # _acme-challenge.<zone>; Knot then signs the record and transfers it to ns2,
-  # which never needs this key. Retrieve the secret for the client with:
+  # ACME DNS-01 (RFC 2136), general key. A dedicated TSIG key scoped by acl_acme
+  # (referenced by every zone below) to TXT updates at or under _acme-challenge.
+  # Retrieve the client config with:
  #   clan vars get ns1 dns-acme-tsig/acme.conf
  clan.core.vars.generators.dns-acme-tsig = {
    files."acme.conf" = {
@@ -49,8 +56,47 @@ in
    '';
  };

+  # ACME DNS-01, dedicated mx1 key. A *separate* TSIG key (acme_mx1) that only
+  # mx1 holds, rendered from the shared secret (generator dns-acme-mx1-secret,
+  # imported above). acl_acme_mx1 scopes it to TXT updates at exactly
+  # _acme-challenge.{mx1,mta-sts,mail} (the mail cert and its MTA-STS + client-
+  # alias SANs), and it is attached only to the cnx.email zone below — so this
+  # credential can write nothing but mx1's own cert challenges.
+  clan.core.vars.generators.dns-acme-mx1-knot = {
+    files."acme.conf" = {
+      secret = true;
+      owner = "knot";
+      group = "knot";
+    };
+    dependencies = [ "dns-acme-mx1-secret" ];
+    script = ''
+      printf 'key:\n  - id: acme_mx1\n    algorithm: hmac-sha256\n    secret: %s\n' \
+        "$(cat "$in"/dns-acme-mx1-secret/secret)" > "$out"/acme.conf
+    '';
+  };
+
+  # ACME DNS-01, dedicated web01 key. A *separate* TSIG key (acme_web01) that only
+  # web01 holds, rendered from the shared secret (generator dns-acme-web01-secret,
+  # imported above). acl_acme_web01 scopes it to TXT updates at _acme-challenge on
+  # the cnx.network zone — the owner the wildcard *.cnx.network challenge uses — so
+  # this credential can write nothing but web01's own cert challenges.
+  clan.core.vars.generators.dns-acme-web01-knot = {
+    files."acme.conf" = {
+      secret = true;
+      owner = "knot";
+      group = "knot";
+    };
+    dependencies = [ "dns-acme-web01-secret" ];
+    script = ''
+      printf 'key:\n  - id: acme_web01\n    algorithm: hmac-sha256\n    secret: %s\n' \
+        "$(cat "$in"/dns-acme-web01-secret/secret)" > "$out"/acme.conf
+    '';
+  };
+
  services.knot.keyFiles = [
    config.clan.core.vars.generators.dns-acme-tsig.files."acme.conf".path
+    config.clan.core.vars.generators.dns-acme-mx1-knot.files."acme.conf".path
+    config.clan.core.vars.generators.dns-acme-web01-knot.files."acme.conf".path
  ];

  services.knot.settings.acl = [
@@ -63,6 +109,30 @@ in
      "update-owner-match" = "sub-or-equal";
      "update-owner-name" = [ "_acme-challenge" ];
    }
+    {
+      id = "acl_acme_mx1";
+      key = "acme_mx1";
+      action = [ "update" ];
+      "update-type" = [ "TXT" ];
+      "update-owner" = "name";
+      "update-owner-match" = "sub-or-equal";
+      "update-owner-name" = [
+        "_acme-challenge.mx1"
+        "_acme-challenge.mta-sts"
+        "_acme-challenge.mail"
+      ];
+    }
+    {
+      id = "acl_acme_web01";
+      key = "acme_web01";
+      action = [ "update" ];
+      "update-type" = [ "TXT" ];
+      "update-owner" = "name";
+      "update-owner-match" = "sub-or-equal";
+      # Wildcard *.cnx.network places its challenge at _acme-challenge.cnx.network,
+      # i.e. _acme-challenge at the cnx.network apex (where this acl is attached).
+      "update-owner-name" = [ "_acme-challenge" ];
+    }
  ];

  # Automatic DNSSEC signing policy (primary only). ECDSA P-256/SHA-256 with
@@ -91,9 +161,13 @@ in
    "dnssec-signing" = true;
    "dnssec-policy" = "cnx";
    notify = [ "ns2" ];
+    # ns2 transfers; acme_ddns does general DNS-01 updates. The dedicated
+    # acme_mx1 key is attached only to cnx.email, so it can't touch other zones.
    acl = [
      "acl_ns2"
      "acl_acme"
-    ]; # ns2 transfers; acme_ddns key does DNS-01 updates
+    ]
+    ++ lib.optionals (d == "cnx.email") [ "acl_acme_mx1" ]
+    ++ lib.optionals (d == "cnx.network") [ "acl_acme_web01" ];
  }) domains;
 }
@@ -1,6 +1,7 @@
-{ ... }:
+{ config, ... }:
 let
  domains = import ../../modules/dns/domains.nix;
+  hosts = import ../../modules/hosts.nix;
 in
 {
  imports = [
@@ -11,10 +12,11 @@ in

  clan.core.sops.defaultGroups = [ "admins" ];

-  # Public IPv6 (matches the ns2 AAAA glue); SLAAC doesn't bring it up here.
+  # Public IPv6 (from modules/hosts.nix; matches the ns2 AAAA glue); SLAAC
+  # doesn't bring it up here.
  cnx.staticIPv6 = {
    enable = true;
-    address = "2a01:4f9:c014:6d87::1";
+    address = hosts.${config.networking.hostName}.ipv6;
  };

  time.timeZone = "Etc/GMT-3"; # UTC+3 (fixed offset, no DST)
@@ -0,0 +1,22 @@
+{ config, ... }:
+let
+  hosts = import ../../modules/hosts.nix;
+in
+{
+  imports = [
+    ../../modules/static-ipv6.nix
+    ../../modules/monitoring/exporters.nix
+    ../../modules/web-proxy.nix
+  ];
+
+  clan.core.sops.defaultGroups = [ "admins" ];
+
+  # Public IPv6 (from modules/hosts.nix); SLAAC doesn't bring it up here.
+  cnx.staticIPv6 = {
+    enable = true;
+    address = hosts.${config.networking.hostName}.ipv6;
+  };
+
+  time.timeZone = "Etc/GMT-8"; # UTC+8 (Singapore, fixed offset, no DST)
+  services.timesyncd.enable = true;
+}
@@ -0,0 +1,50 @@
+# ---
+# schema = "single-disk"
+# [placeholders]
+# mainDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_108706511"
+# ---
+# This file was automatically generated!
+# CHANGING this configuration requires wiping and reinstalling the machine
+{
+
+  boot.loader.grub.efiSupport = true;
+  boot.loader.grub.efiInstallAsRemovable = true;
+  boot.loader.grub.enable = true;
+  disko.devices = {
+    disk = {
+      main = {
+        name = "main-ddd46ebf135244608078712d6ec76691";
+        device = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_108706511";
+        type = "disk";
+        content = {
+          type = "gpt";
+          partitions = {
+            "boot" = {
+              size = "1M";
+              type = "EF02"; # for grub MBR
+              priority = 1;
+            };
+            ESP = {
+              type = "EF00";
+              size = "500M";
+              content = {
+                type = "filesystem";
+                format = "vfat";
+                mountpoint = "/boot";
+                mountOptions = [ "umask=0077" ];
+              };
+            };
+            root = {
+              size = "100%";
+              content = {
+                type = "filesystem";
+                format = "ext4";
+                mountpoint = "/";
+              };
+            };
+          };
+        };
+      };
+    };
+  };
+}
@@ -0,0 +1,19 @@
+# Shared TSIG secret for the dedicated acme_mx1 key.
+#
+# This key lets mx1 — and only mx1 — write _acme-challenge.mx1.cnx.email TXT
+# records on ns1 to obtain its mail TLS cert via ACME DNS-01. ns1 scopes it with
+# acl_acme_mx1 (attached only to the cnx.email zone) so the credential can touch
+# nothing else. ns1 renders this secret into a Knot key file; mx1 into a lego
+# rfc2136 env file; both must carry the same secret, hence one shared generator
+# with a per-host renderer that depends on it. Imported by ns1 and (via mail.nix)
+# mx1.
+{ pkgs, ... }:
+{
+  clan.core.vars.generators.dns-acme-mx1-secret = {
+    share = true;
+    files."secret".secret = true;
+    runtimeInputs = [ pkgs.openssl ];
+    # 32 random bytes, base64 — a valid hmac-sha256 TSIG secret.
+    script = ''openssl rand -base64 32 | tr -d '\n' > "$out"/secret'';
+  };
+}
@@ -0,0 +1,19 @@
+# Shared TSIG secret for the dedicated acme_web01 key.
+#
+# This key lets web01 — and only web01 — write _acme-challenge.cnx.network TXT
+# records on ns1 to obtain its wildcard (*.cnx.network) TLS cert via ACME DNS-01.
+# ns1 scopes it with acl_acme_web01 (attached only to the cnx.network zone) so the
+# credential can touch nothing else. ns1 renders this secret into a Knot key file;
+# web01 into a lego rfc2136 env file; both must carry the same secret, hence one
+# shared generator with a per-host renderer that depends on it. Imported by ns1
+# and (via web-proxy.nix) web01.
+{ pkgs, ... }:
+{
+  clan.core.vars.generators.dns-acme-web01-secret = {
+    share = true;
+    files."secret".secret = true;
+    runtimeInputs = [ pkgs.openssl ];
+    # 32 random bytes, base64 — a valid hmac-sha256 TSIG secret.
+    script = ''openssl rand -base64 32 | tr -d '\n' > "$out"/secret'';
+  };
+}
@@ -1,4 +1,9 @@
-{ config, lib, pkgs, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
 let
  # ZeroTier addresses — zone transfers run over the mesh, not the public net.
  mesh = import ../mesh-hosts.nix { inherit config lib; };
@@ -11,8 +11,36 @@ $TTL 3600
@       IN  NS      ns1.cnx.network.
@       IN  NS      ns2.cnx.network.

-; ---- Mail (fill in once the mail host exists) ----
-;@      IN  MX 10   mail.cnx.email.
-;mail   IN  A       <mail-ipv4>
-;@      IN  TXT     "v=spf1 mx -all"
-;_dmarc IN  TXT     "v=DMARC1; p=quarantine; rua=mailto:postmaster@cnx.email"
+; ---- Mail ----
+mx1     IN  A       5.223.65.38
+mx1     IN  AAAA    2a01:4ff:2f0:1963::1
+; Client-facing alias for IMAP/submission (Thunderbird etc.); the cert carries
+; mail.cnx.email as a SAN. The MX must never point here (CNAMEs are illegal MX
+; targets) — server-to-server delivery and DANE stay on mx1.cnx.email.
+mail    IN  CNAME   mx1.cnx.email.
+@       IN  MX 10   mx1.cnx.email.
+@       IN  TXT     "v=spf1 mx -all"
+; Aggregate (rua) + forensic (ruf) reports go to the dmarc@cnx.email mailbox,
+; which parsedmarc on control polls and feeds into Grafana. fo=1 asks reporters
+; to send a forensic report on any SPF/DKIM failure.
+_dmarc  IN  TXT     "v=DMARC1; p=quarantine; rua=mailto:dmarc@cnx.email; ruf=mailto:dmarc@cnx.email; fo=1"
+
+; ---- DANE / TLSA ----
+; "3 1 1" = DANE-EE, SPKI, SHA-256: the digest of mx1's certificate public key.
+; Valid because the zone is DNSSEC-signed and the lego cert uses --reuse-key, so
+; the key (and thus this digest) is stable across renewals. Compute it AFTER the
+; first issuance and paste the hex below:
+;   ssh mx1 'openssl x509 -in /var/lib/acme/mx1.cnx.email/cert.pem -noout -pubkey \
+;     | openssl pkey -pubin -outform DER | openssl dgst -sha256 -binary | xxd -p -c256'
+_25._tcp.mx1  IN  TLSA  3 1 1  bd9a51f60b6d2dd20f18b3553d2795053ac52f87567a46bc892006bb58506404
+
+; ---- MTA-STS ----
+; Policy host (A/AAAA point at mx1); the _mta-sts TXT id MUST be bumped whenever
+; the policy file in modules/mail.nix changes, or senders keep the cached policy.
+mta-sts   IN  A       5.223.65.38
+mta-sts   IN  AAAA    2a01:4ff:2f0:1963::1
+_mta-sts  IN  TXT     "v=STSv1; id=2026061801"
+mail._domainkey IN TXT ( "v=DKIM1; k=rsa; "
+        "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAr9QxTs5dLtY76bo156+Tp0GUoE554rMwIooIYa2MMYHNs8zPb0thFmaCKGAINdHKNIq2phXAlk51iBTfdqXjx7gVWSrs+ftykqO3b5hUjgImsgqPWGUTzy5/bUgcDELiD9KKEyKYD3+ebZEw6d0uvBvEsA6a1CPzOsufoCDtyKjByCuQzkCBrK25TUHFolGvEYcZexR0LSF+8hMss"
+        "xyw9NYiPpTXVCWQJnrZZpuOBiX0K2l5CAXVyuT/B5RcBXlAUhBTp3390VEhL0wAZMTOnvtvBYK3NnsTIh96fkh6MfWmre7Fi9hEq//xGf40N5/aomMjJrJdqFZJLZpDotb/XwIDAQAB"
+        )
@@ -25,3 +25,10 @@ control IN  AAAA    fd06:1bad:ece2:92ad:ba99:9306:1bad:ece2
 ;@      IN  A       <web-ipv4>
 ;www    IN  CNAME   cnx.network.
 monitor IN  A       5.223.66.36
+
+; ---- web01 (public reverse proxy / TLS termination) ----
+; Serves a wildcard *.cnx.network TLS cert (ACME DNS-01) and forwards to internal
+; services over the mesh. Add a vhost in modules/web-proxy.nix and a CNAME here.
+web01   IN  A       5.223.55.246
+web01   IN  AAAA    2a01:4ff:2f0:2d8f::1
+grafana IN  CNAME   web01.cnx.network.
@@ -30,6 +30,12 @@ in
    enable = true;
    virtualHosts.":${toString port}".extraConfig = ''
      root * ${site}
+      # mdBook doesn't fingerprint asset filenames, and every file in the Nix
+      # store carries an epoch (1970) mtime, so file_server's only validator never
+      # changes across redeploys — conditional requests would 304 forever and pin
+      # browsers to stale docs. no-store sidesteps caching entirely; pages are tiny
+      # and mesh-only, so refetching on each load is free.
+      header Cache-Control "no-store"
      file_server
    '';
  };
@@ -24,6 +24,46 @@ let
    description = "ICMP (ping / PMTUD)";
  };

+  # Public mail ports for mx1 (MX for cnx.email). 25 is server-to-server
+  # delivery; 587/465 are client submission; 143/993 are IMAP. 443 serves only the
+  # MTA-STS policy (https://mta-sts.cnx.email/.well-known/mta-sts.txt); the cert
+  # itself uses ACME DNS-01 so port 80 stays closed. Admin still rides the mesh.
+  mailPort = port: description: {
+    direction = "in";
+    protocol = "tcp";
+    inherit port;
+    source_ips = world;
+    inherit description;
+  };
+  mailRules = [
+    (mailPort "25" "SMTP (inbound mail)")
+    (mailPort "587" "Submission (STARTTLS)")
+    (mailPort "465" "Submission (implicit TLS)")
+    (mailPort "143" "IMAP (STARTTLS)")
+    (mailPort "993" "IMAP (implicit TLS)")
+    (mailPort "443" "MTA-STS policy (HTTPS)")
+  ];
+
+  # web01 is a public reverse proxy with TLS termination. 443 serves the proxy;
+  # 80 only carries Caddy's HTTP->HTTPS redirect (the cert uses ACME DNS-01, not
+  # HTTP-01). Admin rides the mesh.
+  webRules = [
+    {
+      direction = "in";
+      protocol = "tcp";
+      port = "80";
+      source_ips = world;
+      description = "HTTP (redirect to HTTPS)";
+    }
+    {
+      direction = "in";
+      protocol = "tcp";
+      port = "443";
+      source_ips = world;
+      description = "HTTPS (reverse proxy / TLS termination)";
+    }
+  ];
+
  dnsRules = [
    {
      direction = "in";
@@ -50,4 +90,12 @@ in
  ];
  "clan-ns1" = dnsRules;
  "clan-ns2" = dnsRules;
+  "clan-mx1" = mailRules ++ [
+    zerotier
+    ping
+  ];
+  "clan-web01" = webRules ++ [
+    zerotier
+    ping
+  ];
 }
@@ -0,0 +1,32 @@
+# Per-host public network facts: single source of truth for each machine's
+# public IPv4 and its static public IPv6. Consumed by clan.nix's `internet`
+# connection hosts (ipv4) and each machine's `cnx.staticIPv6` (ipv6), so an
+# address is written once instead of being duplicated across configs.
+#
+# NOT a driver for the DNS zone files — those stay hand-edited text, so a record
+# here that also appears as A/AAAA glue still needs a matching manual zone edit.
+#
+# ipv6 is the single address to assign from the host's allocated /64 (we take
+# ::1), without prefix length; cnx.staticIPv6 supplies the /64 default.
+{
+  control = {
+    ipv4 = "77.42.68.181";
+    ipv6 = "2a01:4f9:c013:e6d0::1";
+  };
+  ns1 = {
+    ipv4 = "46.224.170.206";
+    ipv6 = "2a01:4f8:c014:b5c5::1";
+  };
+  ns2 = {
+    ipv4 = "157.180.70.82";
+    ipv6 = "2a01:4f9:c014:6d87::1";
+  };
+  mx1 = {
+    ipv4 = "5.223.65.38";
+    ipv6 = "2a01:4ff:2f0:1963::1";
+  };
+  web01 = {
+    ipv4 = "5.223.55.246";
+    ipv6 = "2a01:4ff:2f0:2d8f::1";
+  };
+}
@@ -0,0 +1,26 @@
+# Shared credential for the dmarc@cnx.email mailbox.
+#
+# DMARC aggregate/forensic reports are delivered to dmarc@cnx.email on mx1;
+# parsedmarc on control fetches them over IMAPS across the mesh and needs the
+# *plaintext* passphrase, while mx1's mailserver only needs the sha-512 hash.
+# clan vars secrets are per-machine, so this generator is shared (share = true)
+# to make the same value available on both hosts. Files are root-owned: SNM reads
+# the hash as root, and parsedmarc's ExecStartPre reads the passphrase as root.
+# Imported by mx1 (via mail.nix) and control (via monitoring/parsedmarc.nix).
+{ pkgs, ... }:
+{
+  clan.core.vars.generators.mail-dmarc-cred = {
+    share = true;
+    files."passphrase".secret = true; # read by parsedmarc on control
+    files."hash".secret = true; # consumed by the mailserver on mx1
+    runtimeInputs = [
+      pkgs.xkcdpass
+      pkgs.mkpasswd
+    ];
+    script = ''
+      pass="$(xkcdpass --numwords=4 --delimiter=- --case=lower)-$((RANDOM % 90 + 10))"
+      printf '%s' "$pass" > "$out"/passphrase
+      printf '%s' "$pass" | mkpasswd -s -m sha-512 > "$out"/hash
+    '';
+  };
+}
@@ -0,0 +1,161 @@
+# Declarative mail stack for mx1 (Simple NixOS Mailserver: Postfix + Dovecot +
+# Rspamd + OpenDKIM). Imported by machines/mx1 alongside the SNM flake module.
+#
+# Mailboxes are virtual (not system users): each address below is a login account
+# whose password is auto-generated by a clan vars generator as a four-word
+# passphrase with a trailing number (e.g. otter-lantern-cobalt-driftwood-42). The
+# generator stores both the passphrase and its sha-512 hash. To add a mailbox:
+# append the address to `accounts`, run `clan vars generate mx1`, redeploy mx1,
+# then hand the passphrase to the user:
+#   clan vars get mx1 mail-passwd-<addr>/passphrase
+# (addr with @ and . replaced by -at- and -, e.g. mail-passwd-postmaster-at-cnx-email)
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+let
+  hosts = import ./hosts.nix;
+  fqdn = "mx1.cnx.email";
+  mtaStsHost = "mta-sts.cnx.email";
+  # Client-facing alias (CNAME -> mx1) so Thunderbird etc. can use mail.cnx.email
+  # for submission/IMAP; added as a cert SAN so TLS validates against that name.
+  clientHost = "mail.cnx.email";
+
+  # MTA-STS policy served at https://mta-sts.cnx.email/.well-known/mta-sts.txt.
+  # enforce = a sending MTA that fetched this must use a valid, MX-matching TLS
+  # cert or refuse to deliver. Bump the _mta-sts TXT id (in the zone) whenever
+  # this changes.
+  mtaStsPolicy = pkgs.writeText "mta-sts.txt" ''
+    version: STSv1
+    mode: enforce
+    mx: ${fqdn}
+    max_age: 604800
+  '';
+
+  # The mailboxes mx1 serves. postmaster is required by RFC 5321.
+  accounts = [
+    "postmaster@cnx.email"
+  ];
+
+  genName = addr: "mail-passwd-" + lib.replaceStrings [ "@" "." ] [ "-at-" "-" ] addr;
+
+  passwdGenerators = lib.listToAttrs (
+    map (addr: {
+      name = genName addr;
+      value = {
+        files."passphrase".secret = true; # retrievable to hand to the user
+        files."hash".secret = true; # consumed by SNM's hashedPasswordFile
+        runtimeInputs = [
+          pkgs.xkcdpass
+          pkgs.mkpasswd
+        ];
+        script = ''
+          pass="$(xkcdpass --numwords=4 --delimiter=- --case=lower)-$((RANDOM % 90 + 10))"
+          printf '%s' "$pass" > "$out"/passphrase
+          printf '%s' "$pass" | mkpasswd -s -m sha-512 > "$out"/hash
+        '';
+      };
+    }) accounts
+  );
+
+  loginAccounts =
+    lib.listToAttrs (
+      map (addr: {
+        name = addr;
+        value.hashedPasswordFile = config.clan.core.vars.generators.${genName addr}.files."hash".path;
+      }) accounts
+    )
+    // {
+      # DMARC report inbox (rua/ruf target in the cnx.email zone). Its password
+      # comes from the *shared* mail-dmarc-cred generator instead of the per-machine
+      # set above, so parsedmarc on control can read the same passphrase over the
+      # mesh. Retrieve it with: clan vars get mx1 mail-dmarc-cred/passphrase
+      "dmarc@cnx.email".hashedPasswordFile =
+        config.clan.core.vars.generators.mail-dmarc-cred.files."hash".path;
+    };
+in
+{
+  imports = [
+    ./dns/acme-mx1-secret.nix
+    ./mail-dmarc-cred.nix
+  ];
+
+  clan.core.vars.generators = passwdGenerators // {
+    # Render the shared acme_mx1 TSIG secret into a lego rfc2136 env file. lego
+    # (via security.acme below) uses it to write the _acme-challenge.mx1.cnx.email
+    # TXT record to ns1, which authorizes the acme_mx1 key for exactly that owner.
+    dns-acme-rfc2136 = {
+      files."rfc2136.env".secret = true; # root-owned; systemd reads it as root
+      dependencies = [ "dns-acme-mx1-secret" ];
+      script = ''
+        printf 'RFC2136_NAMESERVER=${hosts.ns1.ipv4}:53\nRFC2136_TSIG_ALGORITHM=hmac-sha256.\nRFC2136_TSIG_KEY=acme_mx1\nRFC2136_TSIG_SECRET=%s\n' \
+          "$(cat "$in"/dns-acme-mx1-secret/secret)" > "$out"/rfc2136.env
+      '';
+    };
+  };
+
+  mailserver = {
+    enable = true;
+    # Fresh install: declare the latest layout the nixos-25.11 branch ships (3),
+    # so SNM uses the current dovecot mail directory layout with nothing to migrate.
+    stateVersion = 3;
+    inherit fqdn;
+    domains = [ "cnx.email" ];
+    inherit loginAccounts;
+
+    # Consume a security.acme cert we obtain ourselves via DNS-01 (below); no
+    # web server and no inbound HTTP needed, so port 80 stays closed. Add the
+    # MTA-STS host as a SAN so the one cert also covers the policy endpoint.
+    certificateScheme = "acme";
+    certificateDomains = [
+      mtaStsHost
+      clientHost
+    ];
+
+    dkimSelector = "mail";
+  };
+
+  security.acme = {
+    acceptTerms = true;
+    defaults.email = "postmaster@cnx.email";
+    certs.${fqdn} = {
+      dnsProvider = "rfc2136";
+      environmentFile = config.clan.core.vars.generators.dns-acme-rfc2136.files."rfc2136.env".path;
+      # ns1 is the only nameserver that accepts the acme_mx1 UPDATE; check
+      # propagation against it directly rather than a public resolver.
+      dnsResolver = "${hosts.ns1.ipv4}:53";
+      # Keep the private key fixed across renewals so the DANE TLSA "3 1 1"
+      # record (public-key digest, published in the zone) stays valid.
+      extraLegoRenewFlags = [ "--reuse-key" ];
+      # Caddy serves the MTA-STS endpoint from explicit cert file paths, so it
+      # won't notice a renewal on its own — reload it whenever the cert changes.
+      # (Merges with the postfix/dovecot reloads SNM wires up for this cert.)
+      reloadServices = [ "caddy.service" ];
+    };
+  };
+
+  # The mail cert is owned group=acme (SNM adds postfix/dovecot); Caddy serves the
+  # MTA-STS endpoint from the same cert, so it needs to read the key too.
+  users.users.caddy.extraGroups = [ "acme" ];
+
+  # MTA-STS policy endpoint, served by Caddy (same web server as control's docs).
+  # The explicit `tls cert key` points at the lego-issued mail cert (which carries
+  # mta-sts.cnx.email as a SAN) and disables Caddy's automatic ACME, so no extra
+  # issuance happens and the DANE TLSA key stays stable. Only :443 is opened.
+  services.caddy = {
+    enable = true;
+    virtualHosts.${mtaStsHost}.extraConfig = ''
+      tls /var/lib/acme/${fqdn}/cert.pem /var/lib/acme/${fqdn}/key.pem
+      root * ${pkgs.writeTextDir ".well-known/mta-sts.txt" (builtins.readFile mtaStsPolicy)}
+      file_server
+    '';
+  };
+
+  # DKIM private keys are generated on first start under this dir. They're
+  # regenerable (rotate + republish the TXT), but declaring the path as clan
+  # state lets a borg client back it up to avoid a needless DNS round-trip on
+  # restore. Wiring mx1 into the borgbackup instance is a separate step.
+  clan.core.state.mail-dkim.folders = [ config.mailserver.dkimKeyDirectory ];
+}
@@ -11,10 +11,15 @@ let
  dir = config.clan.core.settings.directory;

  readVar =
-    machine: file:
-    builtins.readFile "${dir}/vars/per-machine/${machine}/zerotier/${file}/value";
+    machine: file: builtins.readFile "${dir}/vars/per-machine/${machine}/zerotier/${file}/value";

-  hosts = lib.genAttrs [ "control" "ns1" "ns2" ] (m: readVar m "zerotier-ip");
+  hosts = lib.genAttrs [
+    "control"
+    "ns1"
+    "ns2"
+    "mx1"
+    "web01"
+  ] (m: readVar m "zerotier-ip");

  # RFC 4193 prefix of this ZeroTier network: fd + the 8-byte network id + the
  # 0x9993 marker. The network id is a public var on the controller (control).
@@ -59,6 +59,50 @@ in
          }
        ];
      }
+      {
+        name = "backup";
+        rules = [
+          {
+            alert = "BackupJobFailed";
+            expr = ''node_systemd_unit_state{name=~"borgbackup-job-.+\\.service",state="failed"} == 1'';
+            for = "5m";
+            labels.severity = "warning";
+            annotations.summary = "Backup job {{ $labels.name }} on {{ $labels.instance }} failed";
+            annotations.description = "The borgbackup run did not complete. Check `systemctl status {{ $labels.name }}` and `journalctl -u {{ $labels.name }}` on the client; `borgbackup-create` re-runs it.";
+          }
+          {
+            alert = "BackupStale";
+            expr = ''time() - node_systemd_timer_last_trigger_seconds{name=~"borgbackup-job-.+\\.timer"} > 93600'';
+            for = "30m";
+            labels.severity = "warning";
+            annotations.summary = "No successful backup on {{ $labels.instance }} for over 26h";
+            annotations.description = "The daily backup timer {{ $labels.name }} has not fired within its expected window; the most recent archive is stale. A value far above 26h (or no data) means backups have stopped entirely.";
+          }
+        ];
+      }
+      {
+        # Outside-in DNS probes (blackbox on control). The `for` rides out a
+        # single dropped UDP packet; only a sustained failure fires.
+        name = "dns_probe";
+        rules = [
+          {
+            alert = "DNSResolutionProbeFailed";
+            expr = ''probe_success{query="SOA"} == 0'';
+            for = "5m";
+            labels.severity = "critical";
+            annotations.summary = "{{ $labels.zone }} is not resolving from {{ $labels.instance }}";
+            annotations.description = "The blackbox SOA probe to this public nameserver address is failing; from the outside the zone looks unavailable there, which the Knot stats would not show.";
+          }
+          {
+            alert = "DNSSECProbeFailed";
+            expr = ''probe_success{query="DNSKEY"} == 0'';
+            for = "5m";
+            labels.severity = "critical";
+            annotations.summary = "{{ $labels.zone }} DNSKEY missing from {{ $labels.instance }}";
+            annotations.description = "The DNSKEY probe to this public nameserver address is failing: the zone's signing keys are not being served, so validating resolvers will treat answers as bogus.";
+          }
+        ];
+      }
    ];
  };
 }
@@ -0,0 +1,108 @@
+# Blackbox DNS probe definitions, shared between the exporter module
+# (modules/monitoring/blackbox.nix, which renders these into the blackbox
+# config) and the scraper (modules/monitoring/server.nix, which turns them into
+# VictoriaMetrics scrape jobs). Kept in one place so the module list and the
+# scrape jobs can never drift apart.
+#
+# These query the nameservers' PUBLIC addresses, i.e. the path a real internet
+# resolver takes, not the mesh — the whole point is to catch outside-in
+# resolution failures the Knot stats can't see. For each zone we run two probes
+# per endpoint: an SOA query (is the zone being served at all?) and a DNSKEY
+# query (is it still DNSSEC-signed?). Blackbox has no DO-bit option, so we ask
+# for DNSKEY directly — an authoritative signed zone returns it without EDNS0,
+# and its absence means signing has broken.
+{ lib }:
+let
+  domains = import ../dns/domains.nix;
+
+  blackboxAddr = "127.0.0.1:9115";
+
+  # Public endpoints of the authoritative nameservers. The v4 addresses also
+  # appear in the `internet` instance in clan.nix; the v6 ones in each ns
+  # machine's cnx.staticIPv6. IPv6 literals are bracketed for host:port.
+  endpoints = [
+    {
+      instance = "ns1 v4";
+      target = "46.224.170.206:53";
+    }
+    {
+      instance = "ns1 v6";
+      target = "[2a01:4f8:c014:b5c5::1]:53";
+    }
+    {
+      instance = "ns2 v4";
+      target = "157.180.70.82:53";
+    }
+    {
+      instance = "ns2 v6";
+      target = "[2a01:4f9:c014:6d87::1]:53";
+    }
+  ];
+
+  queries = [
+    {
+      name = "soa";
+      type = "SOA";
+    }
+    {
+      name = "dnskey";
+      type = "DNSKEY";
+    }
+  ];
+
+  sanitize = lib.replaceStrings [ "." ] [ "_" ];
+  moduleName = zone: q: "dns_${q.name}_${sanitize zone}";
+
+  modules = lib.listToAttrs (
+    lib.concatMap (
+      zone:
+      map (
+        q:
+        lib.nameValuePair (moduleName zone q) {
+          prober = "dns";
+          timeout = "5s";
+          dns = {
+            query_name = "${zone}.";
+            query_type = q.type;
+            valid_rcodes = [ "NOERROR" ];
+            # Fail unless at least one answer RR of the queried type is present:
+            # a NOERROR with an empty answer (or a missing DNSKEY) still fails.
+            validate_answer_rrs.fail_if_not_matches_regexp = [ "\\s${q.type}\\s" ];
+          };
+        }
+      ) queries
+    ) domains
+  );
+
+  scrapeConfigs = lib.concatMap (
+    zone:
+    map (q: {
+      job_name = "blackbox_${moduleName zone q}";
+      metrics_path = "/probe";
+      params.module = [ (moduleName zone q) ];
+      static_configs = map (e: {
+        targets = [ e.target ];
+        labels = {
+          instance = e.instance;
+          zone = zone;
+          query = q.type;
+        };
+      }) endpoints;
+      # Hand the real DNS server to blackbox as ?target=, then point the scrape
+      # at the exporter itself.
+      relabel_configs = [
+        {
+          source_labels = [ "__address__" ];
+          target_label = "__param_target";
+        }
+        {
+          target_label = "__address__";
+          replacement = blackboxAddr;
+        }
+      ];
+    }) queries
+  ) domains;
+in
+{
+  inherit modules scrapeConfigs blackboxAddr;
+}
@@ -0,0 +1,24 @@
+# Blackbox exporter on control: outside-in DNS probes against the public
+# nameserver addresses (see blackbox-probes.nix for what and why). Bound to
+# loopback — only VictoriaMetrics on the same host scrapes its /probe endpoint,
+# and the scrape jobs that drive it live in server.nix. The probes leave control
+# over the public internet to reach ns1/ns2, which is the path we want to test.
+{
+  lib,
+  pkgs,
+  ...
+}:
+let
+  probes = import ./blackbox-probes.nix { inherit lib; };
+in
+{
+  services.prometheus.exporters.blackbox = {
+    enable = true;
+    listenAddress = "127.0.0.1";
+    port = 9115;
+    # JSON is valid YAML; enableConfigCheck runs the exporter's own --config.check
+    # against this file at build time, so a malformed prober is caught here.
+    configFile = pkgs.writeText "blackbox.yml" (builtins.toJSON { inherit (probes) modules; });
+    enableConfigCheck = true;
+  };
+}
@@ -0,0 +1,217 @@
+{
+  "uid": "cnx-backups",
+  "title": "CNX Backups",
+  "tags": ["backup", "borg", "cnx"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "1m",
+  "time": { "from": "now-7d", "to": "now" },
+  "templating": { "list": [] },
+  "annotations": { "list": [] },
+  "panels": [
+    {
+      "type": "row",
+      "title": "Backups",
+      "id": 1,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }
+    },
+    {
+      "type": "stat",
+      "title": "Backup health",
+      "description": "1 if any borgbackup job is in the failed state, 0 otherwise. A successful run leaves the oneshot unit inactive (still OK); only a real failure shows FAILED. Derived from the node_exporter systemd collector on the backup client (ns1).",
+      "id": 2,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 5, "w": 8, "x": 0, "y": 1 },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "thresholds" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [{ "color": "green", "value": null }]
+          },
+          "noValue": "no data",
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "0": { "text": "OK", "color": "green", "index": 0 },
+                "1": { "text": "FAILED", "color": "red", "index": 1 }
+              }
+            }
+          ]
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "colorMode": "background",
+        "graphMode": "none",
+        "textMode": "auto",
+        "orientation": "auto"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "max(node_systemd_unit_state{name=~\"borgbackup-job-.+\\\\.service\",state=\"failed\"})",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "Last backup run",
+      "description": "When the most recent backup timer last fired (the daily borgbackup job). 'No data' before the first run.",
+      "id": 3,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 5, "w": 8, "x": 8, "y": 1 },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "dateTimeFromNow",
+          "color": { "mode": "fixed", "fixedColor": "text" },
+          "noValue": "never"
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "colorMode": "none",
+        "graphMode": "none",
+        "textMode": "auto",
+        "orientation": "auto"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "max(node_systemd_timer_last_trigger_seconds{name=~\"borgbackup-job-.+\\\\.timer\"}) * 1000",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "Time since last backup",
+      "description": "Age of the most recent backup. Backups run daily, so anything past ~25h means a run was missed. Red over 25h.",
+      "id": 4,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 5, "w": 8, "x": 16, "y": 1 },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "color": { "mode": "thresholds" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "red", "value": 90000 }
+            ]
+          },
+          "noValue": "never"
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "colorMode": "background",
+        "graphMode": "none",
+        "textMode": "auto",
+        "orientation": "auto"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "time() - max(node_systemd_timer_last_trigger_seconds{name=~\"borgbackup-job-.+\\\\.timer\"})",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "type": "table",
+      "title": "Backup jobs (current state)",
+      "description": "Every borgbackup job and the systemd unit state it is currently in, per client. 'inactive' is the normal resting state of a oneshot job between runs.",
+      "id": 5,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 6 },
+      "options": { "showHeader": true },
+      "fieldConfig": {
+        "defaults": { "custom": { "align": "auto" } },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "node_systemd_unit_state{name=~\"borgbackup-job-.+\\\\.service\"} == 1",
+          "format": "table",
+          "instant": true
+        }
+      ],
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "Time": true,
+              "Value": true,
+              "__name__": true,
+              "job": true,
+              "type": true
+            }
+          }
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Failed state over time",
+      "description": "1 while a backup job is in the failed state. A spike here is a backup that did not complete and was not retried before the next scrape.",
+      "id": 6,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 6 },
+      "fieldConfig": {
+        "defaults": { "unit": "short", "min": 0, "max": 1 },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "node_systemd_unit_state{name=~\"borgbackup-job-.+\\\\.service\",state=\"failed\"}",
+          "legendFormat": "{{instance}} {{name}}"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Time since last backup (history)",
+      "description": "Age of the latest backup over time. The sawtooth should reset to near zero once a day; a steady climb without a reset means backups stopped running.",
+      "id": 7,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 14 },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "time() - node_systemd_timer_last_trigger_seconds{name=~\"borgbackup-job-.+\\\\.timer\"}",
+          "legendFormat": "{{instance}} {{name}}"
+        }
+      ]
+    }
+  ]
+}
@@ -4,7 +4,7 @@
  "tags": ["dns", "knot", "cnx"],
  "timezone": "browser",
  "schemaVersion": 39,
-  "version": 2,
+  "version": 3,
  "refresh": "30s",
  "time": { "from": "now-6h", "to": "now" },
  "templating": { "list": [] },
@@ -207,6 +207,120 @@
          "legendFormat": "{{instance}}"
        }
      ]
+    },
+    {
+      "type": "row",
+      "title": "DNS probes (outside-in)",
+      "id": 20,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 42 }
+    },
+    {
+      "type": "table",
+      "title": "Probe status (per zone / server)",
+      "description": "blackbox_exporter on control queries each nameserver's public address (v4 + v6) for every zone: an SOA query (zone served) and a DNSKEY query (still signed). UP = the resolver's-eye view is healthy.",
+      "id": 21,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 43 },
+      "options": { "showHeader": true },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "align": "auto",
+            "cellOptions": { "type": "color-background" }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "red", "value": null },
+              { "color": "green", "value": 1 }
+            ]
+          },
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "0": { "text": "DOWN", "index": 0 },
+                "1": { "text": "UP", "index": 1 }
+              }
+            }
+          ]
+        },
+        "overrides": [
+          {
+            "matcher": { "id": "byName", "options": "zone" },
+            "properties": [
+              { "id": "custom.cellOptions", "value": { "type": "auto" } }
+            ]
+          },
+          {
+            "matcher": { "id": "byName", "options": "query" },
+            "properties": [
+              { "id": "custom.cellOptions", "value": { "type": "auto" } }
+            ]
+          },
+          {
+            "matcher": { "id": "byName", "options": "instance" },
+            "properties": [
+              { "id": "custom.cellOptions", "value": { "type": "auto" } }
+            ]
+          }
+        ]
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "probe_success",
+          "format": "table",
+          "instant": true
+        }
+      ],
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": { "Time": true, "__name__": true, "job": true },
+            "renameByName": { "Value": "status" }
+          }
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Probe success (1 = ok)",
+      "description": "0 means the probe failed: the zone is not being served or not signed from that public address. Sustained failures fire DNSResolutionProbeFailed / DNSSECProbeFailed.",
+      "id": 22,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 43 },
+      "fieldConfig": {
+        "defaults": { "unit": "short", "min": 0, "max": 1 },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "probe_success",
+          "legendFormat": "{{zone}} {{query}} @ {{instance}}"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "DNS probe latency",
+      "description": "Total round-trip time of each blackbox DNS probe. A climbing trend points at a slow or overloaded nameserver before it starts failing outright.",
+      "id": 23,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 51 },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "probe_duration_seconds",
+          "legendFormat": "{{zone}} {{query}} @ {{instance}}"
+        }
+      ]
    }
  ]
 }
@@ -0,0 +1,194 @@
+{
+  "uid": "cnx-uptime",
+  "title": "CNX Uptime",
+  "tags": ["uptime", "availability", "cnx"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "30s",
+  "time": { "from": "now-24h", "to": "now" },
+  "templating": { "list": [] },
+  "annotations": { "list": [] },
+  "panels": [
+    {
+      "type": "row",
+      "title": "Uptime",
+      "id": 1,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }
+    },
+    {
+      "type": "stat",
+      "title": "Host status",
+      "description": "Whether VictoriaMetrics is currently able to scrape each host's node_exporter. UP means the host (and its mesh path) is reachable; DOWN means the scrape failed. One tile per machine.",
+      "id": 2,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 6, "w": 12, "x": 0, "y": 1 },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "thresholds" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [{ "color": "green", "value": null }]
+          },
+          "noValue": "no data",
+          "mappings": [
+            {
+              "type": "value",
+              "options": {
+                "0": { "text": "DOWN", "color": "red", "index": 0 },
+                "1": { "text": "UP", "color": "green", "index": 1 }
+              }
+            }
+          ]
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "colorMode": "background",
+        "graphMode": "none",
+        "textMode": "value_and_name",
+        "orientation": "auto"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "up{job=\"node\"}",
+          "legendFormat": "{{instance}}",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "Current uptime",
+      "description": "Time since each host last booted (now - node_boot_time_seconds). A value that drops back to near zero means the host rebooted.",
+      "id": 3,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 6, "w": 12, "x": 12, "y": 1 },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "dtdurations",
+          "color": { "mode": "fixed", "fixedColor": "text" },
+          "noValue": "no data"
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "colorMode": "none",
+        "graphMode": "none",
+        "textMode": "value_and_name",
+        "orientation": "auto"
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "time() - node_boot_time_seconds{job=\"node\"}",
+          "legendFormat": "{{instance}}",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "type": "bargauge",
+      "title": "Availability over window",
+      "description": "Fraction of successful scrapes over the selected time range, per host (avg of up over $__range). 100% means every scrape in the window succeeded; dips reveal flapping or outages. Red below 99%.",
+      "id": 4,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 7 },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "color": { "mode": "thresholds" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "red", "value": null },
+              { "color": "yellow", "value": 99 },
+              { "color": "green", "value": 99.9 }
+            ]
+          },
+          "noValue": "no data"
+        },
+        "overrides": []
+      },
+      "options": {
+        "displayMode": "gradient",
+        "orientation": "horizontal",
+        "showUnfilled": true,
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        }
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "avg_over_time(up{job=\"node\"}[$__range]) * 100",
+          "legendFormat": "{{instance}}",
+          "instant": true
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Uptime over time",
+      "description": "Host uptime across the window. The line should climb steadily; a reset to zero marks a reboot.",
+      "id": 5,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 7 },
+      "fieldConfig": {
+        "defaults": { "unit": "s", "custom": { "fillOpacity": 0 } },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "time() - node_boot_time_seconds{job=\"node\"}",
+          "legendFormat": "{{instance}}"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Up/down history",
+      "description": "1 while a host's node_exporter was scrapeable, 0 while it was not. Gaps to zero are outages or lost mesh connectivity.",
+      "id": 6,
+      "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+      "gridPos": { "h": 6, "w": 24, "x": 0, "y": 15 },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "short",
+          "min": 0,
+          "max": 1,
+          "custom": { "fillOpacity": 20, "lineInterpolation": "stepAfter" }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "refId": "A",
+          "datasource": { "type": "prometheus", "uid": "victoriametrics" },
+          "expr": "up{job=\"node\"}",
+          "legendFormat": "{{instance}}"
+        }
+      ]
+    }
+  ]
+}
@@ -1,5 +1,6 @@
-# Metric exporters, imported by every machine. Host metrics everywhere; Knot DNS
-# metrics on the nameservers. Everything is reachable only over the ZeroTier mesh
+# Per-host observability agents, imported by every machine. Host metrics
+# everywhere; Knot DNS metrics on the nameservers; journald shipped to
+# VictoriaLogs on control. Everything is reachable only over the ZeroTier mesh
 # (see the firewall rule at the bottom); the public side is already closed by the
 # Hetzner cloud firewall.
 {
@@ -86,6 +87,30 @@ in
    ];
  };

+  # Ship journald to VictoriaLogs on control (services.victorialogs in
+  # server.nix). control uploads to loopback so its own logs survive a mesh
+  # outage; ns1/ns2 push over the mesh to control's ZeroTier address.
+  services.journald.upload = {
+    enable = true;
+    settings.Upload.URL =
+      let
+        dest =
+          if config.networking.hostName == "control" then
+            "127.0.0.1:9428"
+          else
+            "[${mesh.hosts.control}]:9428";
+      in
+      "http://${dest}/insert/journald";
+  };
+
+  # systemd-journal-upload exits if the sink is unreachable. Upstream already
+  # restarts it (Restart=always/RestartSec=3sec), but the default start-rate limit
+  # (5 tries / 10s) lets it give up permanently — so a transient VictoriaLogs
+  # outage leaves the uploader dead until the next deploy. Disable the limit so it
+  # retries forever and self-heals once the sink returns. (A persistent failure
+  # still surfaces loudly in a deploy, which is what we want.)
+  systemd.services.systemd-journal-upload.startLimitIntervalSec = 0;
+
  # Scrape ports reachable only from the ZeroTier mesh.
  networking.firewall.extraInputRules = ''
    ip6 saddr ${mesh.subnet} tcp dport { ${lib.concatMapStringsSep ", " toString ports} } accept
@@ -0,0 +1,60 @@
+# DMARC report analyzer, imported by control only. parsedmarc fetches the
+# aggregate/forensic reports that land in the dmarc@cnx.email mailbox on mx1,
+# parses the XML, and stores results in a local Elasticsearch; the official
+# parsedmarc dashboard + an Elasticsearch datasource are auto-provisioned into
+# the Grafana instance that server.nix already runs on this host.
+#
+# IMAP runs over the ZeroTier mesh, not the public net: we pin mx1.cnx.email to
+# its mesh address in /etc/hosts so TLS still validates against the public
+# Let's Encrypt cert (primary domain mx1.cnx.email) while the bytes stay on the
+# overlay. The mailbox passphrase is the shared mail-dmarc-cred secret; parsedmarc
+# reads it as root in its ExecStartPre, so root-owned (clan default) is fine.
+{ config, lib, ... }:
+let
+  mesh = import ../mesh-hosts.nix { inherit config lib; };
+in
+{
+  imports = [ ../mail-dmarc-cred.nix ];
+
+  # Elasticsearch 7.x is under the (unfree) Elastic License; allow just this one
+  # package rather than opening allowUnfree globally.
+  nixpkgs.config.allowUnfreePredicate = pkg: lib.getName pkg == "elasticsearch";
+
+  # Keep mx1's IMAP traffic on the mesh while presenting the public cert name.
+  networking.hosts.${mesh.hosts.mx1} = [ "mx1.cnx.email" ];
+
+  services.parsedmarc = {
+    enable = true;
+    provision = {
+      # Local Elasticsearch on 127.0.0.1:9200 (loopback; no firewall change).
+      # datasource + dashboard default to true once ES and Grafana are both on.
+      elasticsearch = true;
+      # GeoIP needs a MaxMind account/license key; skip it (reports still parse,
+      # just without source-IP geolocation).
+      geoIp = false;
+      grafana = {
+        datasource = true;
+        dashboard = true;
+      };
+    };
+    settings = {
+      imap = {
+        host = "mx1.cnx.email";
+        port = 993;
+        ssl = true;
+        user = "dmarc@cnx.email";
+        password = {
+          _secret = config.clan.core.vars.generators.mail-dmarc-cred.files."passphrase".path;
+        };
+      };
+      mailbox = {
+        watch = true; # IMAP IDLE: process reports as they arrive
+        delete = false; # archive processed reports, don't delete
+      };
+      general = {
+        save_aggregate = true;
+        save_forensic = true;
+      };
+    };
+  };
+}
@@ -10,7 +10,9 @@
 }:
 let
  mesh = import ../mesh-hosts.nix { inherit config lib; };
+  probes = import ./blackbox-probes.nix { inherit lib; };
  vmPort = 8428;
+  logsPort = 9428;
  grafanaPort = 3000;
  controlV6 = mesh.hosts.control;

@@ -43,6 +45,8 @@ in
            (target "control" "127.0.0.1" 9100)
            (target "ns1" (v6 mesh.hosts.ns1) 9100)
            (target "ns2" (v6 mesh.hosts.ns2) 9100)
+            (target "mx1" (v6 mesh.hosts.mx1) 9100)
+            (target "web01" (v6 mesh.hosts.web01) 9100)
          ];
        }
        {
@@ -52,10 +56,31 @@ in
            (target "ns2" (v6 mesh.hosts.ns2) 9433)
          ];
        }
-      ];
+      ]
+      # Outside-in DNS probes via the blackbox exporter (blackbox.nix). The job
+      # list is generated from the same probe definitions the exporter uses.
+      ++ probes.scrapeConfigs;
    };
  };

+  # Centralized logs: VictoriaLogs ingests journald from all three hosts, each
+  # of which runs systemd-journal-upload against /insert/journald (exporters.nix).
+  # Binds all interfaces because ns1/ns2 push over the mesh; the firewall rule at
+  # the bottom scopes 9428 to the mesh subnet and the Hetzner firewall closes the
+  # public side. Retention is set via extraOptions (no dedicated NixOS option).
+  services.victorialogs = {
+    enable = true;
+    listenAddress = ":${toString logsPort}";
+    # -enableTCP6: like the scraper above, VictoriaLogs is IPv4-only by default
+    # for *listening* too — ":9428" binds 0.0.0.0 only, so ns1/ns2 pushing over
+    # the IPv6 mesh get "connection refused". This makes it bind [::] (dual-stack)
+    # so the mesh can reach it. Retention has no dedicated NixOS option.
+    extraOptions = [
+      "-retentionPeriod=30d"
+      "-enableTCP6"
+    ];
+  };
+
  # Admin password generated once and stored as a clan secret. Retrieve with:
  #   clan vars get control grafana-admin/password
  clan.core.vars.generators.grafana-admin = {
@@ -72,6 +97,9 @@ in

  services.grafana = {
    enable = true;
+    # VictoriaLogs datasource plugin so journald is greppable from Grafana,
+    # alongside the metrics datasource.
+    declarativePlugins = [ pkgs.grafanaPlugins.victoriametrics-logs-datasource ];
    settings = {
      server = {
        http_addr = "::";
@@ -98,6 +126,13 @@ in
            url = "http://127.0.0.1:${toString vmPort}";
            isDefault = true;
          }
+          {
+            name = "VictoriaLogs";
+            type = "victoriametrics-logs-datasource";
+            uid = "victorialogs";
+            access = "proxy";
+            url = "http://127.0.0.1:${toString logsPort}";
+          }
        ];
      };
      dashboards.settings = {
@@ -113,8 +148,9 @@ in
    };
  };

-  # Grafana reachable only from the ZeroTier mesh (admin laptops + servers).
+  # Grafana (admin laptops + servers) and VictoriaLogs ingestion (ns1/ns2 push
+  # journald over the mesh) reachable only from the ZeroTier mesh.
  networking.firewall.extraInputRules = ''
-    ip6 saddr ${mesh.subnet} tcp dport ${toString grafanaPort} accept
+    ip6 saddr ${mesh.subnet} tcp dport { ${toString grafanaPort}, ${toString logsPort} } accept
  '';
 }
@@ -0,0 +1,73 @@
+# Public reverse proxy with TLS termination for web01. Caddy fronts internal
+# services and forwards to them over the ZeroTier mesh, never the public net.
+# The cert is a single wildcard (*.cnx.network) obtained via ACME DNS-01, so
+# adding a vhost needs no new issuance. Public ports: 443 for the proxy and 80
+# only for Caddy's HTTP->HTTPS redirect (issuance never uses inbound HTTP).
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+let
+  mesh = import ./mesh-hosts.nix { inherit config lib; };
+  hosts = import ./hosts.nix;
+  certName = "cnx.network";
+in
+{
+  imports = [ ./dns/acme-web01-secret.nix ];
+
+  # Render the shared acme_web01 TSIG secret into a lego rfc2136 env file. lego
+  # (via security.acme below) uses it to write _acme-challenge.cnx.network TXT
+  # records on ns1, which authorizes the acme_web01 key for exactly that owner.
+  clan.core.vars.generators.dns-acme-web01-rfc2136 = {
+    files."rfc2136.env".secret = true; # root-owned; systemd reads it as root
+    dependencies = [ "dns-acme-web01-secret" ];
+    script = ''
+      printf 'RFC2136_NAMESERVER=${hosts.ns1.ipv4}:53\nRFC2136_TSIG_ALGORITHM=hmac-sha256.\nRFC2136_TSIG_KEY=acme_web01\nRFC2136_TSIG_SECRET=%s\n' \
+        "$(cat "$in"/dns-acme-web01-secret/secret)" > "$out"/rfc2136.env
+    '';
+  };
+
+  security.acme = {
+    acceptTerms = true;
+    defaults.email = "postmaster@cnx.email";
+    # One wildcard cert for every vhost this proxy serves, via DNS-01 (so issuance
+    # never depends on inbound HTTP). Port 80 is open only for Caddy's
+    # HTTP->HTTPS redirect, not for ACME.
+    certs.${certName} = {
+      domain = "*.cnx.network";
+      extraDomainNames = [ "cnx.network" ];
+      dnsProvider = "rfc2136";
+      environmentFile = config.clan.core.vars.generators.dns-acme-web01-rfc2136.files."rfc2136.env".path;
+      # ns1 is the only nameserver that accepts the acme_web01 UPDATE; check
+      # propagation against it directly rather than a public resolver.
+      dnsResolver = "${hosts.ns1.ipv4}:53";
+      # Caddy reads the cert from explicit file paths (tls directive below), so it
+      # won't notice a renewal on its own — reload it whenever the cert changes.
+      reloadServices = [ "caddy.service" ];
+    };
+  };
+
+  # The lego-issued cert is owned group=acme; Caddy needs to read the key.
+  users.users.caddy.extraGroups = [ "acme" ];
+
+  # Reverse proxy. The explicit `tls cert key` points Caddy at the wildcard cert
+  # and disables its automatic ACME, so no extra issuance happens. Backends are
+  # dialed over the mesh by their ZeroTier address (mesh.hosts.<name>).
+  services.caddy = {
+    enable = true;
+    virtualHosts."grafana.cnx.network".extraConfig = ''
+      tls /var/lib/acme/${certName}/cert.pem /var/lib/acme/${certName}/key.pem
+      reverse_proxy http://[${mesh.hosts.control}]:3000
+    '';
+  };
+
+  # 443 serves the proxy; 80 only carries Caddy's automatic HTTP->HTTPS redirect
+  # (the Hetzner cloud firewall also scopes these in
+  # modules/hetzner-firewall-rules.nix). Admin still rides the mesh.
+  networking.firewall.allowedTCPPorts = [
+    80
+    443
+  ];
+}
@@ -0,0 +1,6 @@
+[
+  {
+    "publickey": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+    "type": "age"
+  }
+]
@@ -0,0 +1,6 @@
+[
+  {
+    "publickey": "age1yey6gxgsyl4tj6ek0tve2pckt6qersqspk66ukkzum8mrr6zppqsj4jn3m",
+    "type": "age"
+  }
+]
@@ -0,0 +1 @@
+../../../groups/admins
@@ -0,0 +1,18 @@
+{
+	"data": "ENC[AES256_GCM,data:lAgbSvyxBl0/NG8rHweKEEqYyDsHa+SrPZnxVubB2x5H3cqydhTCs8NMxJp+RKTPW2nhfmB/lrKgLHFfFMJfMg5jZ2BvZtR1IzA=,iv:1PgMbra/ec7yiCm7K5yo+1lCLJ89ryfP3SQML7uYv4k=,tag:lmaG+dZPCLHtQ59uFeSkVg==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBELzcrdUc0cW9pSXhGbzRY\naTd5aWgzcFpLQ3MyVElaZU5JK2Z4ZGRtMXpJCk14bmZLRlY4Z3pDeUxPM203anVY\nbE1aUEdDZ3NkcGxrcCtWS1ZmRWIrcmMKLS0tIDVTOUJhenR0WVhZSWNWT2VtRFpv\nc1h6Nk95Z3Y4eFQ1NUdmWWJiaWdDdGMKB3whh/RgAePTJnGmeDJ/WFv4NI42vA5O\nB0F6jmSDNa5beP8Um2DjWdPENkJJjv9yv38b7hP8BLDe9Ba4WBNfBA==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBby9pdmFZ\nOElJVCtUQmJWVVljQi9KcHE2a2FaeU5HeUhyUm05Zm0wZDZKVwptY2hHRFJKMi9I\nN3NDY3ZjYVloTjF6R0l4RllDS1dpNzA1TDJDc0hGQnFvCi0tLSB1akhUWk9RU0Rt\naDV5MHdHcExlSzZWbSs0S1A3NGhES0V1U0pEaUd5WE04ChtGuEq0HnRiVTDwhJnO\nIWMhwCYaewHk+k0a0Z9qCwqqKxhfiGS6kg/YTKHTNhQ0bxIA8yqgQoaE8Hl5Zhzs\nKqk=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T04:33:03Z",
+		"mac": "ENC[AES256_GCM,data:Iffy/eS80qOJLdaGOIxti1otdxSLHA1TV06R8xq4zI1qFklJX8OWRy9CAYwAcTURS88gd5c2VGHsBI8yHOBj7LKiLJiu5/xRgD106hotBOJUy90UZ6MMbM2wwHomhQg5r5kNpRsJaPFaLFzm5YJN9gLUU1kxO4Nnt2L3bxZTogQ=,iv:cqbFD3ZIJnaZU7j7l9Qt1QqiUSQtclV7MGZWEZRZQoY=,tag:F0ZZQSQKYtBI65uutjNxow==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../users/berwn
@@ -0,0 +1 @@
+../../../groups/admins
@@ -0,0 +1,18 @@
+{
+	"data": "ENC[AES256_GCM,data:u+O5nWbFlvp6SGyHJggfkCLCT0ZuDy89e/VGGQNdt3yYzgdNmnrtd+2q+Ft3MtoOSSCLvStriGQzfLhcqEgqGgt3PqfIzCO1IG4=,iv:GyrZ1XUiOZe1I1Z/HebTy2NM2tfDHxIH5zGVk7HD+xQ=,tag:js6fbXWajVZSxt0hmnnA5g==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWenArV1pZMURzNElTeGlC\nRSttUFRwM2E5d2htVDlFcTkwVDIrV2JxQjE0CmE0NUZ4UERHdHpqYk9uK01GOTFQ\ndW5UNlRrY0hvU2pNQmpSanh5RG5YbVkKLS0tIFdYQmkva1NORG80U3dDSWszZTlO\nOXViWUxMTVR1NE00cjdpVXpVS1J5YU0KHBkeKAJZDc+R1GLKwDYLyQBlEW7tPnMh\nf3tsUvtD0flqPAXNeDgyOmKufP7U6oDy/OriFC9+zYQbWyEEc6CZHg==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBL05xT1BJ\nSldCSkFhaFErWDNMeXMvRksvekkwNUZLZFVtSmUydVRrNG9JTwpFaHNsVThZN2gw\nMlNxUXhnN2xYNFluK1hadGxzMFZuaWR3cVFRYW9mWndVCi0tLSBUUmRMSnJCeTRM\nUVdKY3hzWVRkQkFuQ3FaRDVZSEp1b2N1ajg5RnlhT3VRCjX/vWj0We88ATiz808w\nz60RL0BvDGJ6m1BNmqAdtfCCClH33YXQBGrKT2E5elvOTl0iOCrT7HPjzXxJZXuw\nkKg=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-20T19:26:29Z",
+		"mac": "ENC[AES256_GCM,data:uD+L3op6ZPtvmLreJ7S4GE6bQuItV87w1LTMFvjI1Kb5+Z0sXlL0TpYO8WLx8X0yaL3HddwlmBKYQGp/OlRPqZMFDbQuK25oeKB54jfm6YDn66rrMQJl1FOw68fLJaHYNjNelEg/bj2WG7YpfZBoWO67MW6F+44Rg4XF85w/1x0=,iv:X9TmEL5JkcC9waLumWpgpBwp3YWLMslZi++dv2HZ0mk=,tag:i7c1CxT1Xv/+T/jY9E0cdg==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../users/berwn
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/mx1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:Lkwniu1Pmu+kGb94ncTteN/CkBYE47+UJKRSij5APKyPa6wQkc33S00WVLSXkG6I/XGeRUAXxNpM7G3WqmkluBtJnuYdQ3+nLVdarDy015Zu207LbpaYBuDyMU4e5pxH2ekIGnyL7diDI/3/GG+fVrO4xxdrFPWaB0YDcD8+5mtpshUqa+4rDrU9CSikuRo+dkAHveX1+MFfpF0aJmRw2MMwXO4=,iv:LIOftLZQ63yEPJ5S08t97jGGkSUK1LxSMnvy9lEm070=,tag:+yYbq9RU91KB+6r9eC9/fg==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBDS0Fra0tVdHIwREVYRC9l\nZFNwek0zMWYvTkRBT1RKR3p1bTN4SmZncGxVClRoOFRNcklEeWllMW4wdzRCaXNX\nbWtNYUJNM2dGaDZqVGtVY0twZVhEY1UKLS0tIFUxZ1QyMWNLRGJrYkJTbTJCT0Fy\nd2xjbWthVG1JUW5XZmVSK3lWc2NLRGMK6/g42P7ZvAk+t2GmZammNxTLFMudK+Qv\nZt3YUF0+EYKlEENgtjku7SSZ7UElQ5NZNrldlk8ZYLIVTul+8XuvrQ==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJQ1ZMRzEreERmM1V3OFhC\ncEhZYjZ2ZGFWc1kveUpLM0MyYlB6S25jR3cwCkhjaTBpS3dHZG51V0RmN1ZVcmRk\nYnIwSlRKVXVoQ1NqdFQ1M2lPazEyRTgKLS0tIGRFTFRaNUMvdU5Fc3A2Nk10NWVw\nWlZOKzJLUDNwTXY4dm9uZTY0cWg4aDgKYDIEQHgomMuJFHHvtt3BbN4tuBiEcboc\nH6K4NmnDE6wMa/1EGGHrjCFb+tUdZSL/zgf5uVOXnXA6d9BEeCGLNQ==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBNk9Od2FL\neEV2Myt0eGI5emF6TlRGS3A3RGNHZ3o3WCtQUDdpYllCTFY1RwpyU2xyNkFIVmhV\nY0g1NDJHRy95SzRrTnRPTGc5ZXdZYTZtMyswWkZIUVdnCi0tLSBWU0Fsak42TUF1\ncndtNFpDUjVhb3RpeWU1eDd0UGtnUTJXeHNiVmNtUTFZCvUQDFKntKn+mZSuDR00\niTu/TdmOu7s89JvirWtFavSZhBzOoW8eXdX/SJCLVy08wdTjb2ksqDdxn0ceiqgL\nLOU=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T07:11:40Z",
+		"mac": "ENC[AES256_GCM,data:8hd2LItBnmG06HnUQ0avOOnbA4+JAPkJf/Wneqo/YexT/saEK+roa+iMkL2DxtcZK5UahPkJ+wT4q3MfNkOnrdbHmQHUUIkDSX+RxLatMtOseYxg8h9wT6MZehuxkRpN4Y9RlpQu/+l3zKQNbXfsfUj6i91fjH6lcxSYPHNO8ug=,iv:T8hYVO0houthJhFmV2UoOK8d5Z3sevv1pRvhdf5qaXk=,tag:2Q5bi8y+hq1TSkQ45Cmf5Q==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+$6$E/6oFzd0dMewfnBC$ppC/HUSkmazEB93xxR21v0YtEfCZRxPgqth3d0LgSbXsQ87akNMhWss8pPGI5Ez3QT93btGmGLQNY.kYFy465/
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1,18 @@
+{
+	"data": "ENC[AES256_GCM,data:hHmU59iiqnyvrQ3IIYihxajInbwAVUKT+al9sGubWTM=,iv:3UONUbekFlIwJj8mTR8tsXvlp8ReIPm5eR0djmHCXUs=,tag:wEfguaEgWW9/dYr0TldTJg==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzUmN6b1d3bWxMV1hsOUJO\nY3ozTmdmOGVsU1ZTU25FellURHFpUDdiYmcwCktEaFVzVVVjdkozR1lvY2FoVVMx\nZ3U1MVFBcDhQUTQ3bWRWVjNyR1l0dU0KLS0tIEZyRkZYRUdYN3lFMFI2SnE0WUx4\na0szbEpGc2FUSFZ6Z1NQK2RKMjV5NlUKJw8BghcP3qRELI9CuY/gq4SvDYBKDPaI\negNQYAQ9sjF7jY0vAyfkYI6KrwKxBa84WZrQYD0Dng1x9YigllSeyw==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBa3JWRUh1\nZnRqMWNrUFJvVmV3Nk1GbmNBdHJKOXVERU5KenBpYTdqRklrNApBT1RGVHJPNmox\nSWxidTI5dS91UnJyM05jVFFMQUlHS1V5bHhKUHRqNkNNCi0tLSBtQ1l2ajhtcTFo\ncmtWb25wQWIrOGtManc2MTM1b1o4ZGV1VE1ONVZPY05jCuxsR2wFaUHnkQxbqrPh\nAmBBWO0xQexWWdJPS1/nz7uS75Ike233UBSLxNapkac5Obfg9UaRYsTPw6qDkQHT\nyUg=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T04:33:03Z",
+		"mac": "ENC[AES256_GCM,data:sLjSs7asZNK3qJgUOROE/XepuXPqxn2A2idAkLdH95nstbwjQttxjWgSxeLl4V1WbvlUyxBS8O+6JbiPg3dgfFKTEJ2IIQsQS8OBbJV/uUZrFvvAGU3iOB1mMD7Mk/CEz1wuzfVevREQhnbf7YOSAxb6fIK3sY5REU4bNi2qUxI=,iv:PTSvvssFkU8hDMSe2D8EmhwN6KOKIX5SvqjZJJECwXU=,tag:dWyQR50jAxjJA3BrrrL9Ww==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/mx1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:SZIlr7nzcs5EUMVUXQ1KJhrd9JZY7ZR9EUpJ9Eygkpd2sAyMNtV0jMFyfW82PyvLP9/bqXf/5BUR98NuwvMsRmLrb4emEraNJH4qVfS/4s0kXySIJeA6XeMHB7GSuxoh5K/3pUR4tbdFSCM=,iv:grRynVFombEdRp0LfmPHIximhh4rlbQTjqJCjbGhRlw=,tag:qZZCnN63clfKKsb/WQNkbw==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFWUFGNDBLM3VCMFh2Sjc2\nY1FkOVhta3REK1Y0MUZlY1IxTHorRCtuOWxJClJqK2hjT2p0VFBEeE1xd1JDUTBk\nNzR4eG5zNGJnalVGb2ovRHVqUUpla1EKLS0tIEJMYTU3MlVKOHdvaUhhM2dsWEoy\nRVh6SHc2cVdTTW4xZlhpc3U1aTRLT0UK+WYlVCCJ1bUsuF/vy6+mSU0gpM8FGHDE\n9HfyAHPZLR30ZtkRHSq8LQ137hxmBKSUjv5ztyHc781tLkx9j3lRwQ==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYMVlBOWVRRitmcVBINGJK\nakxWZEdNeHh5QWIrSHJmVzhDRDl6MUhhdUVnClRFcXZNTGYrazNxWG5UVi8vVWlu\nK3NaN004MVhrR1R2YVpZcnArRmVVUjAKLS0tIG90aXpEVHZLVDMrQWhYMHg2bWZj\nN3B2SmpHTnEzNFZ4cXBmM3paRmhCMjAKZa1OlhBcW+4J0sR+aWv0lkLqDh+73Gay\nKl6ltN1EQI7ISH2azQFahzoz6XV8cyYUHAQaHaYZuyCZNb/XbG9VmQ==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBcHF1OUdP\nQk1VVTdla3ZSc1VUa2MrT0IwanpmRWNsVHlYc21jM3dXNkg4RwpSSENkYUVJa2lS\nTk1pb3NhNkVPU3hPeVVPUE1MOGxTNHpsYXJGSVRQVmVBCi0tLSBFVE96Q0luZ1Jv\nODd2eUl1aWhOR08wUFpPZHFkZEd5MktNU1R2ajBoTERzCrmM40bnvt2iHQERfrN9\n8724ZmXn4YpAiN/FwKpPJ+iF2atpPDbUb2PFG2s6s2kJISMCrpoblZHBTYbG322M\nGgw=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T07:11:36Z",
+		"mac": "ENC[AES256_GCM,data:bgu8Pv90Ljc1e6uG5oSDY4IwM85ZqiEQ2Uy0xt6Iu6/f8G/K/sh/+N3ZwcXAV3cpAsa5Cde8vHO8JgmAmcYN1frEgb9+rJyfD/sVvIEmy++WgLKIoUngcRyxX7tk+lLGv03tF09HcfoA+P4d5zNxZq5hR20jkT+2t3Y0nw7XN1M=,iv:gbvg2/5SBg3a2H/bvMFuF1bWSVHTppP59s7ivthe6TI=,tag:6zSYNSG2qHKv9YMLVyvSbA==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/mx1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:FYx5Vll+asvRpC56Wk7ZAB6tdGabWIR25fRD/fw5aTgLz3+wU6K3RkDy,iv:rPD4rRduxodp9e6PGSD9V3zDPaTTAxeSNpC3Q/Umi/k=,tag:DiIahCPXn/DOSgJunryXHg==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrVGRuNnBYL2NJUFFFZWho\nVDM1d1YxUTJLT2lma1NuYWtaNEVpNnUxV0NrClBOb2tBR0Z2a2lMMTNMeStTK2ln\nZjZ1cEFRZjRTTld5M2pNV1c4aHZlZzAKLS0tIHdRNnB2R1E3M1hVamtJWk9wWE1Y\nWG1yKzF5QkFQbWZZUTVObS9jcUtQVmMKdj+SEz7TcCe5Fk5B65EPBvGC0OWVafax\nws4qgi3O4CNQkVoOx4Jq6aWF0I7a0dR0mG2ERPeVUJKt6Kzap/KJ7w==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFbXhLYUhXSzZZNm40OUdJ\nZ0x3VTB3WDBPSzQ3RU03dS94RmNRMEM1ekg4Cmt0SDMxY1hsL0t6RXRZV29ObWJt\nUFFiZEFFL3p1RytVdXpXeU9sVlU1THMKLS0tIHptRlZqVS9NU3RydGJzbkZ4a05P\nVkRUNDErdklTV2ZWeVNIdVRmVTdxSWsKTH3olIgtm+rM7CsKeVq3GVYk+Y2JcoZ2\nE6/KdwBOsRDFvQpw6vuNVUD0LDDWh0T3+V4+3f9YBn1qdqWHFDA52Q==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBODdOY293\nWXZYWUNuOUxwcUh1Y05VcEFrQTZ3VnJiT3ArN1JFZTZLOFl6RApib2RnN3JnS3ZQ\nbFhFakNiVFdsR2tRa3JUVmJaZzJTY1ZOaUcyQlMrcm5jCi0tLSA1Z04vRTJ4NzVC\ncG5XYW1ZRUd4QkJMTGlJbEUwMmpTSnk4RXBTclRrOGN3Cj7t6HYbS8kdKaIYWMms\n74vQn/HJvnYnIwUqEsf3z8QzTfsXtPB4ueA1NjftyvlKMRozuKxb+ULFv6YkZNzX\nxvo=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T07:11:36Z",
+		"mac": "ENC[AES256_GCM,data:Jl/TzaI0c0v3JwJHDSvDUZEGKpgMGgFD1UoWC7qbc6LC+vVOpDjcm3WlfXfy2ljHpaqd74dl2kvy2ra75htI5YuLAisTqrPXhm+8km4tLzQzZOHz3JRIW+0fgnVC8z+GFrOHsz8CCc9SCX03HTyOlugvq0PD2sp4hz6Wgmp4cMc=,iv:Q/qUtb968vhyyRQyDYIP3WE49GLWExkmbarSZz9jPhM=,tag:M0crVWRGl2xvrAJEZvfFaA==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJJWavBJ7+x+VjALDd1UzkAnYkNiHgrkIzzvwT6ZQX4k 
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/mx1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:k00qZlgD7TOR8kD0nDU3phJgJ+5mNZNGuB0J+VMvn2KC639DshChmU2DMn0J7tVZ1pEc9EpsYJEDni1cBWGvP8lKT0fbGT8vHIMS3GIsSSM0MXeGIGe7TUf6iuI8vxomXR/jQQyBq9dv7tHUKVqmfWT2D5SR2pQimOsUxP/qxK1jtjZqBMFFOIkRl9v3Kl36WSXOpHvSWqC0Y9Sntvhu8UaEEjSE0qNRfADAFiJ7yH/gUhgx4SUqyaMwQHgC66+hqVHrGfkhnC0X0Mcw+2oZHDHn4gTit6RCIJyOXKCzd4D9a5ldUF4m9Ws42N9Ulc20ECkFVRrGfo3b6iI7W1EKo1C4tWEya4/+iYj+H051oM1vUPoMQ0wVoRrc9b7iXj8CDqt8jRBbPsCEeYTHLmXfbYuV8s0Q54byNN3gF0oXuP7sutkQl6bEgCkRIQYSqOAeF2ngr+sJHCuTLm5hnvFQVTvTEKIUd3H78+qta+q0J/nWU4Z6yEca+/dPnNOQr/U1aLw/,iv:eAGFaqHaY3Eu/BPGvx76MyJvUuyuC+XMRAjRNItuuu8=,tag:d2MeamgG3W22LK2GQLM2Sw==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsYmllbWh0NE93TFJ1b0NV\nNSt3a3VkOThFTVFxaWx3VzY1RnBOQndSaENnClg4eXh3cnJvSkhKdGs2dHAxa2tz\nZ1FqSkNyRzF0MUp4bnhqSWVrWUJGaUUKLS0tIFl1RU9qemMzS2ZjM3QrbWJZTFMr\nd1dNSVlLcWVEZ1hRWnQwR1pST0tVWGMKStjFxZgbz4GgVaNHH85O5Gtpgmpju8qU\n6Woe1nTo0LZjoHKCgvXNPetE25iKeG1SoPVV4LbARZ1tXDnnq6fmcA==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvK08wODlrN21qaFRTMGdz\nYlhrRGc4M3VnWCs1ajMzZkMvektqZVZRL0ZjCjhWb2hLYm5vMjNYeGFUYlBJMG5R\nNURTQnF3aG90aTFaQ210NElkTWlock0KLS0tIHRCM2Y4djZGYjRDOURXazZSN1hu\nb2l5VS9aNmdJQjBjckpseXhoVzc4M2sKqOw6XiDRlFxpDrhCGPxZG9RDKxtLgaJb\nwTdZjB3C1cF4y7ANwmkbegqLaMXfhRI3/QLwkvYR/bZiWsYdOKUnKg==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBNGNoRzhD\nN21ZcktHemNPallPMzlsRFl5bVNPcEJpUmJ3L00yVWJUT044KwpqRjdHbVNyeUEy\nbTRqRDJqZU82WmNQb1FsRjI3MUUxTjd5ekNqU1J1MnFJCi0tLSBBUEJxeXROWERi\nMlpuSDFxcklwZThXUElhWkUyTnUyMC9DWmJIQU1jbGVFCqIAsovpsuzHJJSQEeXB\naq8nha2lVQQk5aaCRmltzoAGaMM9axrh8bgNT2TrSYv4ZAfV09e4Vn9s/dGm3X5O\ni4M=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T04:33:03Z",
+		"mac": "ENC[AES256_GCM,data:tYw/e79GLeZI34PbiB3AsWVUP28kVad2NqdxvPjNinKF2yRmcML0LNDMSbVwSYUuGJWvyc0mbF4QFThtEiSPVcKqgYXcNm94fmfeouTrZKAvVxjGXE8pa7wDtsoV9yNcpLiQxqtMCZxy7mJPgNhHiMQjiPOqU7tb1O7+o13bCJU=,iv:DgSpKNC49edVc9kWGVY1hZIya40QP/0b52mbx58I0mU=,tag:qB3zc//bUSMbCK5dufPqzg==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/mx1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:5rAULXYJAzvJISpwR/HmZFRT47W3J312LlQLibsg42gq1Ddfskn+njf3CrHzkYHppRyNuGdo9NoX5Abd8PLrSucTyyNfTmNRZLP4YQ4DG7AIZJrDD9SosltNC0BaE7J25WRna0lNQDWwEA==,iv:mwNjVtr91DAL4B+s7DjZJM3t+5EmnwgaqixASjVstSI=,tag:wWy1wqvZS+zXdcmq4z4tuQ==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYcGtPSnpLODF5U0NIQk5H\neERBcm11NkpkanhLd0pKYVllbHV0SndyaVVVCkQ3T3A1OTRXb3Y4V3NpcWorQ1Fa\nenJMU3UxZnYzMmNraXk0YWRVZjF5T3MKLS0tIFZVcGd2d3BwQjVta1RpSzJMTzhQ\nVWpQL2ZkOXFXa0RhejBDZUlMS2VVOFkKfB+wbosTER+rcLGCeAWMcalT9DsvYMQf\n+k4RUA57zSzNFdX/2azltJ3L58IK9UeSxyEgcYobWSnpZmvqr160Jg==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxOE84VVkzV3EvRFo1M0Fa\ncnhZa0NxajZFdnZFOXI1N05tOHUzVVB3cUIwCktvVXU5NGh3eW9wa2NoRWNxU2lP\nUDdiU3hURGFvSHJ6ZEN5QnlMNzkzNHMKLS0tIHRjdXdkMEhQZmlTN2JOV3R5WllE\nV1FnYmhFcnZ1MU44TzJqb2J3cmRtcjgKts2tmDm1F5AYNn34UUdEw0wyqMB0OTwx\ne+yuiSYM3WogVbve6c/fMR84k5Wm+yp/PN77JigzM7SvRwLNmAJqgw==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBcFNIdXVL\nYmlXUnV1T3Q3V3pETUVsY29lcy84RU41UmR0OE9DK3RrbStGQgpxbFVpNlQyS0o4\nVXIrcG1XREc2bFVvODZxTmx3ODBNelI3V3ZLUjhmallvCi0tLSA3emNPb1FLdSs4\neXdrSzA4aVB2ZmVoZ1F1MHVqcVFpbjhoVExmbklKUWR3Cu1BH93XQt5sHPuNJPNR\n0pQ4qS/a1iXbz4A9FSpoerc6esLz7s1r4W2i/Vpc13QKhCr+/q5/n7URFJAPAjFn\nXzo=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T04:33:04Z",
+		"mac": "ENC[AES256_GCM,data:L1KdxE5KVrwbqa6LkpJNicDqOND69rNZKJCwW8+wYKAcxBwFrXF3fs584hFYydLdSlEWDTO2BzVvi9L7BB10gJuRUmw9Pk1rplCe+EkrrTl8ePbfjfGDK5agijsiNYdgRbF6JwHIYOQGVznYKIp28HZnwAi8eXuXzMDo7Fd3nqg=,iv:Xz0c4phNz8a+r/qDO8qf+0IHsTNkV96xBMgH9nh1hAs=,tag:lhxuokbeoPLQrdapMdmneA==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1,18 @@
+{
+	"data": "ENC[AES256_GCM,data:E8Xkux+tMjrL01nbzeoRzhtJLD64VWwmHtDnOTT6ehaJQ+eHEalV/SusGBY=,iv:a59INtTPfRTMfGyCekMhlPW8GNUdvudWGtcDd7bcUT4=,tag:2uhTF6b9vNUaLKzzTns0mQ==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGRHZQVWFQcmdhUlZjM3A1\nbFFUL01VSThCa2NQOE91bS9aTXcreitDY1dVCk9tY2FBcDdWSjdXajN2QkliQ09X\nSU9FWEQwMEFzaHh2WUJLSy9tRFBXbzQKLS0tIFgyWSsxVVFSWVR3RUx6UncvTGZ0\na2p0c3MvUytMWTNxR2ZPRkxMUDJuQUkKi4j4I2yDM7cC9nyHjuN/Hde27YLSuZ2m\nTLOKiZ7wSPP9E9eK41thADTDK+Hd4SkiIQwhkd0I9iUESPl3UQmaOA==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBcUNwQ1BG\ncXZnVE1CMm5jOVFyT3hNbGUxOUoxekxRL29RczdZNjQzOVFPdQp0V2UyUnp3dmxy\nQWNaait0TWcwaU0wbVVNcWdFWU1nYnk0TVVpSjkxazJJCi0tLSB3SDVOb1lCRVEx\nZHZaSjNnYzdEVEtObmE0c0l6NmIwazB5REQ4YVRRS2pvCiMU8QKROV74xvLTvyQ8\npzaBdWRypWjjTegjex8sWISeEix/zBS1rprP8eUnNtXrOc+rAYL8rIEaGTbSjsk7\nccE=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T04:33:03Z",
+		"mac": "ENC[AES256_GCM,data:TGWuRcoVxnSh9G1ByKAl1PB09Q684NMfo4eR5m1bn686DL4t5LkjowzQGvxlhf0I5NnotD8+4ku/qaLER2RgMNSIzLGcCbbQXPGplS+8llIq48oNKcEW8DLwOBaZ9mo/FdFBmKPa/NXaznrTitusTDp9ZiSQJG6LZHjWL8wMYYc=,iv:pYIzkcJUFAbTTi/GDHWKH7pkZ1TH4XOgNZ3QX7/FcXI=,tag:NnaTDdSpJc8EwSIAXpNFuw==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+25.11
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/mx1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:8ScQIuagOWobAFAzs1dVCC3TcR68Qc5aEsxaKfXNxSm9pxEWRC+6VWTdwpQZXOmlalh0iIYnwPMvtnQtDN6Y,iv:j9uxxaKIKMZPtxOvGWrxsvfRsQuYgM0nt+9ceULUPpo=,tag:VUnIp6o1w1fDg6MrjKYw/Q==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqODVlNmRiTFE0WXBMQ0V6\ndGo5SngyVnh1VTFKY1JseDFJakZJNEFmKzFNCmxZNTA4Yjd1RnZuOG01MmlyM1ph\nM1VBbWlYSFdacDZEZDd1ZVExa0ZqelkKLS0tIDhsU0dtSkpvSlVIWGY5TmkxSWQ5\nSW5Zbm1yZ1NibVF0cWhmL2ZUQms0OVUKUGLPD4U2ICKY3ncH4HdToCvJOvIOMUAE\ns45ge1vlE3f3gVRy3YvOX0Us492N2mhRk7KJYc9+/T5Hlaa2r4r1Wg==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoNER4RzZMNFYrMVpJRXdW\nZE9ROVcwODFDcnpPcVc2MHNvUGxoTWlCdXlVCkdCaGJ5bmJLUEM2cDJoS1VBLzFG\naEZHeTZJZDQzMkZnNjdVL3poWUN3QncKLS0tIHN1ZGpGSHVocndEd2lkYnZUSGIx\nU2V1ZDErd0hHTHFlMzB0Z04waDlVOTgKMsap3dhumT9gfahTTIzja8THDuoh1Azq\nuKX546MXQ1QMnEd6eoqW9JVcC0EyJ3kpJuiqJof3dDhWNsIuQq90iA==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBMWRhSGhM\ncEFad05pblcremUyWW04MVpnQXlwcWhsenFhWlRuTVNuUFdBMAozdHVPeXJqaXQ4\nOWo5R0FWYnRZbWQwSkN1bVFDMzJwNlRITkZUbVZIU2dZCi0tLSB4aHlIRFN6cklR\nYTJYZ1RGMWorTXpucGozT0dpRVFvbngyeSt4QkVRVVNZCpaR4D0bLyiZdxBaY9mW\nAiI2jaGZSGDwGNiMRnyPoR24tJjHo619t8CLIto6TohqSWp2SYqLIgMsx0utaqk7\n8SM=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T04:33:06Z",
+		"mac": "ENC[AES256_GCM,data:JQJ1Ph0U60y0TbRy2+8CbrUdjPQ9kTlVeqvzF5Wgx9TqVeF9AAqTzCnMIYp/bcuTRp8O5FBlsKANhqjOuaYzcEnLp8Klu/KYvNcBjl6he4N2maDOCXPQUZ9amPfMKQkPwTh0N8fTnKpQ/c5kOYrI0Mre/R3WDP04m5b0VCsOFIg=,iv:GAda9XfTcRG2R5zDxn0mVZhc2ht0ZQ9SfzMbjLcEccI=,tag:v9Sk5ZiCZk3+ylFAdmeYjQ==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/mx1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:bfy8ekOqcrGllIig0IkLgFY+zh24mVteHvSVH+CBIgPhnFSKM3KxL1Q6y54SqmiVDRTF3T0eVsIxgXAevXsb2xQ2KTC22Mx4vEW0IlQCPkzwG+RDEofs2S4iAxjdUVnz,iv:fN6UD5u4/Yhptp2LbomV5Ed8pC93mlJP8PS9WxSVxGc=,tag:oGxp7LuUnVlwbo5wgNAGXQ==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaNzBDZ2tNOUEwLy9DNHRm\nUkltSnVOTFpobDQyNEk4em43MW5ERHBOYnpZCkdtaXNGaEgrR2NsaXZFcUZxTXlk\nYjc0M0FhRHRTdEEvOURUdVlsbjVUVWMKLS0tIDJ3dml5Ukc3TXlUWHhrcE1EbURP\nUFJ5NlBaSG1jbXJtSDRJUkwwdkw3YXMK4eh+hxHiWx04aMKtDE924De07izXN3xM\nTAWcnQN0z2DPi0smXPsOhd87hmAadUqYSDLNoF5+v0jkpsmJXS3MGA==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtSURZMSt1UWI4NUVSN00v\nR0xjSzlwbjdiUHhtUGh1UVUxZWxFYlN1M0VzCmhXVVEyVlhEOU1aR3dGZ1JPTTVW\nU1JQTlFjQUM1NDhXNVcyN3BYd1VhWUEKLS0tIEtsUDBTQnZPaUM0UkpqRFQ4VU5o\nekZhNE9ZOUNGYSt5TXFHYkJYRmtUZUkKdgQKoa8JbiEtL0ovzgrImO9xVP1+5dMQ\nORkia2j6cXuCKqMegMROK5zk7JdUW81fcsKrI+qwZJdzOO4olJr8mg==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBeVdCNVJS\nOHJydE5DMjFsTC9pMUhRTis3WnRHYVdFNS85ZmNxREpBTmorSQoxMFJXTThpY21G\nYnI2bkszMEJqeHZFdEIrT2ppRExvVGNyV21NYThSaG9FCi0tLSByNVpnZ1d2dW1i\nU0xxSENXalNoZDJjcnNNSzlXSkxFaTU1dUFzNFIwYjY0CtkCquloFj4znepzmXk7\nnJGvvLbx7sXEr4mNTdgcHGYMrRcLaF/XfYsnaWIFXkFTgregD/pJ7UyecgKCma3h\nhKw=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T04:33:06Z",
+		"mac": "ENC[AES256_GCM,data:aJL2X/YalG6d/BN08+p30c0TuXqbCRNUs+4P0mUhaNujO2OtJkYgRGf1+EqcjTv15oqhOGaGxmk8fbNMhEIleEbc8YVWLvKnW0y6A3ezbFRggnR7G0yJw6cHEfE9ruaK3F2b40a4gYvy34QmuAWWkPHs19CZERydzy+vSl6h5oA=,iv:hcgPX1pyqnuUGDBduK1JCXnQHZoIkmDTx3Tcm2dqwbs=,tag:HkzT6FquoCz3uE/WYVs7pg==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/mx1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:VtBJhlWjJXNxqbNGmB95gek4+5oRCHbJqC4O7b64PzeV+HA2AEVaXxBirrZUnY7+uNB2m1F4sftnI9Gzp41if1OSPi//Bak4EGvIS7XRyBQuU9V7gQJPu4zu+tTQa/7hPo8GSi3U7+8zoJSJy9qh9QXkE9Lo8bgTdUs+YBvrE8bb+ioDeXz/fi9f+NAxFE8J6QExyfRfJDTkB0KoRZviBhsIhxR7AwuNAIgDYqZeMYlhclFE2wyaTpdMasNi/wzyJRR3XQltRxFLwPjgAxWek94z5m2uVcQ730FglStYVFBqp7PJij3e9SlAkALcOOBwmblMV5bxoB1cHPgz5X8Jtxk4QN6wqkMKIn+icx4A,iv:uJAfZ0T9+S7viMcpF4rFtuQUyJavFjWBqOXm03SO53U=,tag:18y3lftW7Dop8KZuPpewxQ==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTSTh1OU9qMEdUUVN4SHV3\nYVJqK0prYkdzZm5ZaC9jOHcvbmVYcWJpYjAwCm1NZHlLQ2xINTJxdTdHNjdWcGpH\ndkhlZS91QlNBUGE4YVNaWTFVMmxncnMKLS0tIGJVVkpsY3NZQWdKZTVlQjluZ2FK\nTDNHUWdYYXlJOVErWDJDb2dUOEc2V2cKnom7DpO7sctpJ4lox5npXTd7qKecHx+u\nGVOh51H8hCOKA+EAc4X4KFT9ZjOYi8V9mfeycopfCCH71AaH6kWRmA==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1l5hw95p5h4sthrgn0usms9yfkwwmcvv34tjgrtv9s4e6x39chacshgxavs",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSc0thd0E4TkVZV25sbW03\nYjJPRWtpWE55dlBMSjMyajlKM0JSZ1FwVUFBCm9MR25MVmVYTzQvcXNrRWx6cFk4\ncUpVN2YrV0FVek5PMXJwQlRQcTJ5bGsKLS0tIEdCdHl1ZjVPVE4yWm1BR3ZVWHF0\nd1ZwNzFLV3BWYTRPK3lSUzBCYzdCSUkKDa3Biu+QKP0tU/Gov/y3izgdVm8fhQmv\nW54rHFLsxIr++ihKFZ00rP7ZSICZ+1DgqCnbgFUluG6qFvikxi4Nqw==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBbWU5NDlZ\nUG1rSk45WThzcUhLejdOVVFzanh3TDFKLzlBUUtQZ2xVdnNpVApQdTdRWVkxZ2VK\nczU2Umg4eDl0VUZDWVZTbkhnRjREVEU5L3Y5YzY1ZVljCi0tLSB1RU1RSkdhWUEx\nSkJSOUI4a1FRSDRjQkNQRFY0dEh4NVB3WFVMTzNZd2xzCuXWT+VoW98pvejrvgae\nZrhjW89G7t2NlX9yCQDfN523hUNGNMA3s4wmHOZdquUZyB0SXaDQQosIrrrLcsqG\nut0=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-18T04:33:06Z",
+		"mac": "ENC[AES256_GCM,data:C5Vn56dfkIX2lIAeuAcyxQyWbRmjAOruK2Kw1yEVYgmpd+XSZhZAnPccC94uWJMTUyrmbbfGtjzDjy8HOATl/MmdhyBkt4rFbri94P5xkILl2bvPZADAXVAfuMNjvZS8M6HNAlGLmNgcY7OMEDX65QMWy8fOwdJWKGbjxtdm7n8=,iv:3g8W6UBsqaZn1swtjCd9np+26AFPqZ8RsBDcgmtcbiA=,tag:212tfq3DHseYyxu+DfxG9A==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+fd06:1bad:ece2:92ad:ba99:93e6:d91c:e707
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/ns1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:vyvZucue+ciufz/bP77IImR2lGaJZRbO6iaSkuWzzda2aL3l65M=,iv:jjtXLphn3dkd3t9cBpOfX6gra2QTmzf7OYcEqW77+1M=,tag:iggC2dJx4zvJmna6QQUcnw==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1fanu282vm7njjweqhrpcfcwpttuhce8js4tsyfry98l0neaqpewqs5s7nt",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBTTJDQzh6NGtOTWo1S0Jl\nQlMvNzZsTE9NdWJqY21rRHhiTFlmaWpja2hBCnFQdWoyeDBxV3Q4YVFWU1ZyZkdW\neEdyRndrbUQ1VkFIbkpZUFlTYVdMQTAKLS0tIEVSYWlYbTJJSUNPbWtlME9vUmo1\ncWc2dXBVUXhScTNadlNZNWZVR1MzbUkKZoZq6Kwq8P97PQqKjQXrpUSJijLhvym8\n8r2ytkxZXnOjYNSRpAUCKdQnZWBqPp0sEP5Ry4nPB0h9GDP2D8VBoQ==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6OWowU0MyNlBpamQxOVB1\nZitFNjRCZnZ4a2VzditsRDFYRmx6WGJEYkdNCm9HTFdBdXpjU3hMS2Jsa1BMZCtp\nSTd5NEtPL2NsZXN4anRyT080RndsOGcKLS0tIFF4QTdsUFdPeW1aNm42SXZ0b09h\nQjA4MzZ4UXBxWmFySG5CYkkzZnpTUVkKSR7ZN9fCVzSTfzEC0HrhRM7NcVTb93N/\nioq2auI+l+BJovqzp1gr8STrW+qtn6uwtToo8+9Mz3sfF9AN4rBNgw==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBaVR4WUFZ\nbDJmMmpLVGRHUGtHWkc0TlZ0aU1tdFBzQ0dzVml0eHVCbE9lNgoyWTJHaTNKaGFw\nU0hzclZZY0ZRSHQ4SVRtU0hTY2pNUEc4VW11dzNjRkV3Ci0tLSBHS3lFUFhTVFo5\ndm5xdFJYWmZuL3BnY1hZS3VQL2RFZ2pQMStkTHMrbWtBCk3lMHBj2/GV4fz8dVTY\ni6ybl4FrAiD6QgavX9fos4ruOtLqpPGLdZt0pBpe9I9FEKyQSPXFWuSTz8UxbRGt\n9yU=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-17T08:07:13Z",
+		"mac": "ENC[AES256_GCM,data:kqgU9TV7vGfEWTVGb6QtHLdBPXoRdx0O/Z9SN09VqjZVlhS9J8ROzf4qQWa9JE6fjiGJBQLWDuJlCYcOBx8rtT5cS/hxsBhIS9isCVgkVwjVGpaaf6WuD58eKZB8Z7cuqLfw3Zwlo2tTBGjbIjGHTZcZbEAeT3Jz9rWVW7Q9XIo=,iv:Sy4Lou8jFqxPB701ElaeqAN3EHRYoZmf9bdxDJ/dd90=,tag:M9GnkO0Sqh1yKey/9YB6yQ==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH12h7xNDiJB5JXsKVKbsUYiAxPFqo3Klr2K7LDY5ZWG 
@@ -0,0 +1 @@
+../../../../../../sops/groups/admins
@@ -0,0 +1 @@
+../../../../../../sops/machines/ns1
@@ -0,0 +1,22 @@
+{
+	"data": "ENC[AES256_GCM,data:TI5IZHMlZwRL622yWmG3XMjuIJ8QkB/0D8FFkSKwZaO1PxuLvIZNeoo4v6CM/e3+Bzzur7fBiJZO2o3r50jxY3Hnbel4hGqXf2KY9IJp51MEYn/OjkfSR0XNyci5LZx5Pqj9wppyVyeqW1W6Hk7UG0e23HCNUHUZsOikIa65UTXPu367iF6VDeNAE7XpPd6VVCiUe2s68cdCVE0cwrlEgo06EVGreJOQ25L9Y3Xh7EOl/CqoVt5T8IfPj4zEkemGjSnuUPR4OKTtlF7UgRSKnvHTtIcj35I/uS07n9238HT218tcWsEjRKD0tYJ5NHx+wCvoZn4HSTfUKVG5NpYOgMdIufcQrz0O+nj89h6cU3dkKd26TF6wHeJW/NRQGVCZOg+Bd6SJDRcvfnPG2PukGeBpf5Y+3XM8RuPfj1k42e55kC/ADEUw7HhlPrYLLv8KB9UXDSNICGNptwSdfTY3xlbrKYCdK2kh5YsY2ykKhuJ6fuq0e+5mXUTGs51Bjm7Fq51b,iv:ONXGx48Ha9/y/nqXIoXtXY3/knjj66jnoolTqz6AMxI=,tag:bx2pi9axH7dUm22CFWM3uw==,type:str]",
+	"sops": {
+		"age": [
+			{
+				"recipient": "age1fanu282vm7njjweqhrpcfcwpttuhce8js4tsyfry98l0neaqpewqs5s7nt",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZREtVYlQ4cnRXeEZCYWhJ\ndTVPT1FDeWNMWnR5eXlzUGtaYnhJcW4zaFVNCk8vZ2J1aXUrazlGNVc3amhCOWYr\nMWxoblZtM0M0TnZick1PSGVzazZKOFkKLS0tIENmMmFxNWxJemIzWHVjZXg5NEtM\nK3Q0M0xmQVZjaWVJQ2FFSjZmSGFhcncKbFD7tC1EBkwxa09ICPMMI9rfsEltPhLA\nGVjLNt+08/GIXY8GOBCmGLsN6sQY42fb9kQuEqkWxRgb4/Sifoxbbg==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1hlzrpqqgndcthq5m5yj9egfgyet2fzrxwa6ynjzwx2r22uy6m3hqr3rd06",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2aWJRNWxIT25HZWRvOFJo\nWkNjWExTNTZzR213bEpaOG85SXRDNUMyRWtjCk04WE5JNzIyTktPb2MzdkxUVVBP\nUmR3UncxZDFvcVBaZGUwUHhDeVR5UTAKLS0tIHJOc0tPOFBhcjdhTUEvUloxQVhx\ncmJqQWVWa0E4UG9MaUtGaFV4Wm9TUUkKi8QrZbQA+G6olFEoSlIPcyewcuEVw3Bl\n8QNbpSSqupRZvx58p2dEOoG1LXX/8Z7mR2iYSsYtrtMR1Y3CFvtqiQ==\n-----END AGE ENCRYPTED FILE-----\n"
+			},
+			{
+				"recipient": "age1yubikey1qd859y9ehz2ya8j2cftwrtmdeqhuk7r7yc52zp64wpff6068gwrac3q6nsa",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHBpdi1wMjU2IHFWcTVydyBBcGY2ZXN3\nMzRDQjcwcGd6UTJOVTdTc00wUlpsbnFmTzVZRzlORjlabGJOKwpJaE1DbWtGejhi\nN3phMENzNEpvMkVyQ2VpVDZHbnpMNmJ5VjdoQ2g3WHlVCi0tLSAvVWpyQXlqNjZl\na0ttRVJHTFFXNDJ0cWV2VVhoUFp2dU9qcFN5V0paR2hvCgWYzW/nsvD9RcVXP0Z+\nPtV1wBMWT+/rHmx8i9RIbn8eNf01immQozFzz+F+/FkdAfKVJ722UGR6MPSdzy4d\nA/k=\n-----END AGE ENCRYPTED FILE-----\n"
+			}
+		],
+		"lastmodified": "2026-06-17T08:07:13Z",
+		"mac": "ENC[AES256_GCM,data:cpipxsIxRvhm8HukKx24qv5g9zEgjuVllNgfQcAuSJxGg+GpAY95JgjGhbG+JekKi7O2B1/Tsb3w1UwXZToAeM6T7XOzG+h5790ufQLcWuKhONtNxTQKKrmTxR6e8pOsCOROKd4uKjQWPhrvhpnmLLmsetjEhQc8igrfJRScqlw=,iv:Kb5ZpWBI/F4dqzJEY2m+bycyLi2mA/VZyA3pPwY9g5c=,tag:4b42mtSzonaHwZYySKCrtw==,type:str]",
+		"version": "3.12.1"
+	}
+}
@@ -0,0 +1 @@
+../../../../../../sops/users/berwn
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Berwn	d8bbf08c7a	Add mx1 to secret vars/shared/mail-dmarc-cred/passphrase	2026-06-21 03:28:02 +07:00
Berwn	e6036d9d1b	Add mx1 to secret vars/shared/mail-dmarc-cred/hash	2026-06-21 03:28:01 +07:00
Berwn	f7b64617b9	Update vars via generator mail-dmarc-cred for machine control	2026-06-21 03:27:56 +07:00
Berwn	60db8c60b0	Add parsedmarc DMARC report analyzer on control Deliver cnx.email DMARC aggregate/forensic reports to a dedicated dmarc@cnx.email mailbox on mx1 and analyze them with parsedmarc on control, storing parsed reports in a local loopback Elasticsearch and visualizing via the auto-provisioned Grafana dashboard. parsedmarc fetches the mailbox over IMAPS across the mesh (mx1.cnx.email pinned to its mesh address so TLS still validates), using a shared mail-dmarc-cred clan var so mx1's mailserver and control see the same password.	2026-06-21 03:27:23 +07:00
Berwn	b8bea27a9c	Update runbook docs for web01 reverse proxy and per-host ACME keys Reflect web01 in the machines table and monitoring scrape list, note Grafana is now also published publicly via web01's reverse proxy, add the CNX Uptime dashboard, and document the dedicated acme_mx1/acme_web01 DNS-01 keys.	2026-06-21 03:17:51 +07:00
Berwn	415a050f6a	Scrape web01 node_exporter into VictoriaMetrics Add web01 to the mesh map and the node scrape job so it appears on the uptime/host dashboards alongside the other hosts.	2026-06-21 03:08:56 +07:00
Berwn	3f3f4118c1	Use Singapore time (UTC+8) for mx1 and web01 Both hosts are in the Singapore region, not UTC+3.	2026-06-21 03:07:57 +07:00
Berwn	dfdeb84ab8	Set time.timeZone on mx1 and web01 Both had NTP (timesyncd) enabled but no timezone, unlike control/ns1/ns2. Default to Etc/GMT-3 to match the majority of hosts.	2026-06-21 03:07:31 +07:00
Berwn	48bf7fb250	Add web01 public reverse proxy with DNS-01 wildcard TLS web01 terminates TLS for grafana.cnx.network and proxies to Grafana on control over the mesh. Caddy serves a *.cnx.network wildcard cert obtained via ACME DNS-01, using a dedicated acme_web01 TSIG key scoped on ns1 to _acme-challenge on the cnx.network zone only. Ports 80/443 are the only public exposure (80 just redirects); admin and the backend ride ZeroTier. Also reload Caddy on cert renewal for both web01 and mx1, since both reference the cert via explicit tls file paths and would otherwise keep serving a stale cert after a silent renewal.	2026-06-21 03:05:54 +07:00
Berwn	86a2928825	update(inventory.json): Installed web01	2026-06-21 02:28:43 +07:00
Berwn	f6da01ba18	Add web01 to secret vars/shared/dns-acme-web01-secret/secret	2026-06-21 02:26:44 +07:00
Berwn	eeed40bcb5	Update vars via generator dns-acme-web01-rfc2136 for machine web01	2026-06-21 02:26:44 +07:00
Berwn	aac8f9d8e6	Update vars via generator dns-acme-web01-knot for machine ns1	2026-06-21 02:26:43 +07:00
Berwn	f5874bc337	Update vars via generator zerotier for machine web01	2026-06-21 02:26:33 +07:00
Berwn	2481d4bf92	Update vars via generator tor_tor for machine web01	2026-06-21 02:26:32 +07:00
Berwn	2d8096ee57	Update vars via generator state-version for machine web01	2026-06-21 02:26:30 +07:00
Berwn	1a4a749d78	Update vars via generator root-password for machine web01	2026-06-21 02:26:30 +07:00
Berwn	1c779d8013	Update vars via generator openssh for machine web01	2026-06-21 02:26:30 +07:00
Berwn	9c4e036b09	Update vars via generator emergency-access for machine web01	2026-06-21 02:26:30 +07:00
Berwn	8139b91fbc	Add machine web01 to secrets	2026-06-21 02:26:30 +07:00
Berwn	c436389619	Update secret web01-age.key	2026-06-21 02:26:29 +07:00
Berwn	9fc97e65b2	Update vars via generator dns-acme-web01-secret for machine ns1	2026-06-21 02:26:29 +07:00
Berwn	bd84bf7c85	Set disk schema of machine: web01 to single-disk	2026-06-21 02:25:24 +07:00
Berwn	848dc0dff7	machines/web01/facter.json: update hardware configuration	2026-06-21 02:23:00 +07:00
Berwn	95aff44f86	Add machine web01	2026-06-21 01:58:59 +07:00
Berwn	f42569e992	Add provisioned Grafana uptime dashboard for all hosts	2026-06-21 01:57:08 +07:00
Berwn	1dd3aadb97	Add mail.cnx.email client alias as a cert SAN A mail.cnx.email CNAME (-> mx1.cnx.email) lets clients (Thunderbird etc.) use a friendly hostname for submission/IMAP. To avoid a TLS name mismatch the cert now carries mail.cnx.email as a SAN, so the acme_mx1 key is authorized to write _acme-challenge.mail too. The MX still points at mx1.cnx.email and --reuse-key keeps the DANE TLSA digest valid across the re-issue.	2026-06-18 15:01:03 +07:00
Berwn	dc21348727	Format drifted files to satisfy the treefmt flake-check gate Pure formatting (nixfmt/prettier/yamlfmt); no behavior change. These files predate the current treefmt config and were failing nix flake check; reformatting them makes the gate green again.	2026-06-18 14:49:48 +07:00
Berwn	1cb6f39ea2	Add declarative SNM mail stack on mx1 with DNS-01, DANE, MTA-STS mx1 runs Simple NixOS Mailserver (Postfix/Dovecot/Rspamd/OpenDKIM) for cnx.email. The TLS cert is obtained via ACME DNS-01 using a dedicated, scoped TSIG key (acme_mx1) that ns1 authorizes for only _acme-challenge.mx1 and _acme-challenge.mta-sts on the cnx.email zone, so the credential can write nothing else. Mailbox passwords are auto-minted by a clan vars generator (four-word passphrase + number). DANE TLSA (3 1 1) is published for _25._tcp.mx1; --reuse-key keeps the key digest stable across renewals. MTA-STS is enforced via a Caddy vhost serving the policy on :443 from the same cert (mta-sts SAN). Firewall opens 25/587/465/143/993/443; 80 stays closed.	2026-06-18 14:47:20 +07:00
Berwn	026a26dd53	Add ns1 to secret vars/shared/dns-acme-mx1-secret/secret	2026-06-18 14:11:40 +07:00
Berwn	7e5d50b260	Update vars via generator dns-acme-mx1-knot for machine ns1	2026-06-18 14:11:40 +07:00
Berwn	312de984c1	Update vars via generator dns-acme-rfc2136 for machine mx1	2026-06-18 14:11:40 +07:00
Berwn	d76aa8cc8d	Update vars via generator mail-passwd-postmaster-at-cnx-email for machine mx1	2026-06-18 14:11:36 +07:00
Berwn	0a78cad06e	Update vars via generator dns-acme-mx1-secret for machine mx1	2026-06-18 14:11:36 +07:00
Berwn	d1b24017aa	Use no-store for docs: epoch mtimes make revalidation serve stale	2026-06-18 12:24:38 +07:00
Berwn	77a18df257	Stop browsers serving stale docs by forcing revalidation	2026-06-18 12:19:42 +07:00
Berwn	a4fe2a7b3a	Document how to pull registrar DS records from Knot on ns1 Explain that key material is auto-managed in the KASP keystore under /var/lib/knot, and that the registrar DS is generated per zone with `sudo -u knot keymgr <zone> ds`.	2026-06-18 12:12:10 +07:00
Berwn	6e4178df04	Onboard mx1 mail host and factor out per-host public IPs - Register mx1 in the inventory and as a direct-SSH `internet` host; give it a static public IPv6 (2a01:4ff:2f0:1963::1). - Point the cnx.email MX (plus SPF/DMARC) at mx1 and add its A record. - Bring mx1 into monitoring: import exporters, add it to the mesh map and the node scrape job so its host metrics and journald reach control. - Add a clan-mx1 Hetzner firewall: inbound SMTP + ZeroTier + ICMP, no public SSH (admin rides the mesh like the other hosts). 587/465/993 held for now. - Extract per-host public IPv4/IPv6 into modules/hosts.nix, consumed by clan.nix's internet hosts and each machine's cnx.staticIPv6, so each address is declared once instead of being duplicated across configs. - docs: add mx1 to the machines table.	2026-06-18 11:53:14 +07:00
Berwn	2c89ab913c	update(inventory.json): Installed mx1	2026-06-18 11:35:22 +07:00
Berwn	84c3eece58	Update vars via generator zerotier for machine mx1	2026-06-18 11:33:06 +07:00
Berwn	7f5227d2e2	Update vars via generator tor_tor for machine mx1	2026-06-18 11:33:06 +07:00
Berwn	ebf4efe5c9	Update vars via generator state-version for machine mx1	2026-06-18 11:33:04 +07:00
Berwn	64b7eb1934	Update vars via generator root-password for machine mx1	2026-06-18 11:33:04 +07:00
Berwn	e763d76ae9	Update vars via generator openssh for machine mx1	2026-06-18 11:33:03 +07:00
Berwn	b65f526ea2	Update vars via generator emergency-access for machine mx1	2026-06-18 11:33:03 +07:00
Berwn	3a0bc2dba4	Add machine mx1 to secrets	2026-06-18 11:33:03 +07:00
Berwn	6098fe9a3b	Update secret mx1-age.key	2026-06-18 11:33:03 +07:00
Berwn	8d9981ee5a	Set disk schema of machine: mx1 to single-disk	2026-06-18 11:32:33 +07:00
Berwn	afc2e997c0	machines/mx1/facter.json: update hardware configuration	2026-06-18 11:32:22 +07:00
Berwn	faaa7b66c0	Add machine mx1	2026-06-18 11:21:27 +07:00
Berwn	9c8a2abf3f	Bind VictoriaLogs on IPv6 so the mesh can ship journald to it VictoriaLogs, like the VM scraper, is IPv4-only by default: ":9428" binds 0.0.0.0 only, so ns1/ns2 pushing journald over the IPv6 mesh got "connection refused" while control's own loopback (v4) upload worked. Add -enableTCP6 so it binds [::] (dual-stack), matching the flag already used for the scraper. Also simplify the systemd-journal-upload override to just startLimitIntervalSec=0 (retry forever / self-heal) and drop the SuccessExitStatus masking: a persistent sink failure should stay loud rather than be hidden behind a green deploy.	2026-06-17 17:27:56 +07:00
Berwn	0eb883061b	Keep systemd-journal-upload retrying instead of failing a deploy The uploader exits when VictoriaLogs is unreachable. Upstream already sets Restart=always/RestartSec=3sec, but the default start-rate limit lets the unit give up permanently and trip switch-to-configuration when the sink is briefly down. Disable the limit (startLimitIntervalSec=0) so logging stays best-effort and never wedges a host or a deploy.	2026-06-17 17:09:30 +07:00
Berwn	d4a171640b	Add VictoriaLogs for centralized journald across all hosts control runs VictoriaLogs (:9428, 30d, mesh-scoped) with a matching Grafana datasource. Each host ships journald via systemd's own journald.upload to the /insert/journald endpoint -- no extra agent. control uploads over loopback so its logs survive a mesh outage; ns1 and ns2 push over the mesh.	2026-06-17 16:53:52 +07:00
Berwn	c7b0f206c8	Alert on and chart blackbox DNS probe failures DNSResolutionProbeFailed and DNSSECProbeFailed fire when an SOA or DNSKEY probe to a public nameserver address stays down for 5m. The CNX DNS dashboard gains a "DNS probes (outside-in)" row: per-zone/server status table, probe success, and probe latency.	2026-06-17 15:42:13 +07:00
Berwn	54f607d063	Add blackbox exporter for outside-in DNS probes control runs blackbox_exporter on loopback, probing each nameserver's public v4+v6 address for every zone: SOA (zone served) and DNSKEY (still signed, since blackbox has no DO-bit option). Probe definitions are shared between the exporter config and the VictoriaMetrics scrape jobs so they can't drift. Verified live against ns1/ns2 over v4 and v6.	2026-06-17 15:37:45 +07:00
Berwn	0544bf95e5	Add vmalert rules for failed and stale backups BackupJobFailed fires when a borgbackup job enters the systemd failed state; BackupStale fires when the daily timer has not run in over 26h (or has never run). Both read the node_exporter systemd collector on the backup client, matching the CNX Backups dashboard.	2026-06-17 15:17:12 +07:00
Berwn	1ea5bda23f	Add CNX Backups dashboard and document the backup setup Grafana dashboard (auto-provisioned from the dashboards dir) tracks borgbackup job health, time since last run, and per-job systemd state from the node_exporter systemd collector on the client. New docs page covers the ns1 -> control topology, secrets flow, and restore commands.	2026-06-17 15:13:47 +07:00
Berwn	ed746b58c3	Update vars via generator borgbackup for machine ns1	2026-06-17 15:07:13 +07:00
				`@@ -0,0 +1 @@`
				`$6$E/6oFzd0dMewfnBC$ppC/HUSkmazEB93xxR21v0YtEfCZRxPgqth3d0LgSbXsQ87akNMhWss8pPGI5Ez3QT93btGmGLQNY.kYFy465/`
				`@@ -0,0 +1 @@`
				`ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJJWavBJ7+x+VjALDd1UzkAnYkNiHgrkIzzvwT6ZQX4k`
				`@@ -0,0 +1 @@`
				`ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH12h7xNDiJB5JXsKVKbsUYiAxPFqo3Klr2K7LDY5ZWG`