Add internal reverse proxy for gateways (Caddy, wildcard via DNS-01)

cnx.router.proxy serves <service>.<site><n>.cnx.network with a real Let's
Encrypt wildcard obtained via a gateway-scoped TSIG key against ns1; Blocky
resolves the names to the router's LAN address, so they exist only
internally. First user: Omada UI on gw-cnx-1 (omada.cnx1.cnx.network).

docs/src/gateways.md

@@ -20,6 +20,7 @@ Naming: `gw-<city>-<n>`, e.g. `gw-cnx-1`.
 | IPv6         | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC                                                                                                                                                     |
 | Bans         | CrowdSec + nftables bouncer (sshd log parsing)                                                                                                                                                |
 | Omada        | Optional per site: TP-Link Omada controller as a podman container                                                                                                                             |
+| Proxy        | Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1)                                                   |
 | Management   | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet                                                                                                                           |
 | Storage      | Single-disk ZFS (zstd, `/var` its own dataset for snapshots)                                                                                                                                  |
 
@@ -57,17 +58,24 @@ Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet onl
    (dynamic PPPoE IP; clan connects over the mesh).
 3. Add its node_exporter and Blocky scrape targets in
    `modules/monitoring/server.nix`.
-4. `git add` the new machine directory — flake evaluation only sees
+4. If the site uses the internal proxy (`cnx.router.proxy.enable`), wire ns1:
+   import `(import ../../modules/dns/acme-gw-secret.nix "gw-<city>-<n>")` in
+   **both** the gateway's and ns1's configuration, and on ns1 add the
+   `dns-acme-gw-<city>-<n>-knot` generator, its `keyFiles` entry, and an
+   `acl_acme_gw_<city>_<n>` scoped to `_acme-challenge.<city><n>` on the
+   `cnx.network` zone (copy the `gw-cnx-1` blocks). Then
+   `clan vars generate ns1` and redeploy ns1.
+5. `git add` the new machine directory — flake evaluation only sees
    git-tracked files, so an untracked `machines/gw-…/` is silently ignored.
-5. `clan vars generate gw-<city>-<n>` — prompts for the site's PPPoE
+6. `clan vars generate gw-<city>-<n>` — prompts for the site's PPPoE
    credentials, mints the ZeroTier identity etc. (`nix flake check` fails
    until this has run, because mesh-hosts reads the ZeroTier IP var.)
-6. Boot the box from a NixOS installer USB on the local network, then:
+7. Boot the box from a NixOS installer USB on the local network, then:
    `clan machines install gw-<city>-<n> --target-host root@<lan-ip>`
-7. Check `facter.json` for the real NIC names, fix `wan.interface` /
+8. Check `facter.json` for the real NIC names, fix `wan.interface` /
    `trunkPorts` if the enumeration differs, and
    `clan machines update gw-<city>-<n>` (rides the mesh from then on).
-8. Add a row to the site table above and to the machines table in
+9. Add a row to the site table above and to the machines table in
    [Overview](./overview.md).
 
 ## Omada controller
@@ -80,6 +88,33 @@ gateway's mesh address from an admin machine. Controller state is under
 `/var/lib/omada`, declared as clan state (`clan.core.state.omada`); wiring it
 into the borgbackup instance is still a follow-up.
 
+## Internal reverse proxy
+
+`cnx.router.proxy.enable` puts Caddy on the gateway, terminating TLS for
+`<service>.<site><n>.cnx.network` (e.g. `https://omada.cnx1.cnx.network`) and
+forwarding to internal backends:
+
+```nix
+cnx.router.proxy = {
+  enable = true;
+  services.omada = {
+    backend = "https://127.0.0.1:8043";
+    insecureSkipVerify = true; # Omada's cert is self-signed
+  };
+};
+```
+
+- **Cert**: one real Let's Encrypt wildcard `*.<site><n>.cnx.network` per
+  gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind
+  PPPoE with zero inbound reachability, and browsers trust it without a CA
+  install. Each gateway has its own TSIG key (`acme_gw_<city>_<n>`) that ns1
+  scopes to `_acme-challenge.<site><n>` TXT records only.
+- **Resolution**: the names exist only internally — Blocky answers
+  `*.<site><n>.cnx.network` with the router's `lan` address; the public
+  `cnx.network` zone never carries them.
+- **Access**: `proxy.allowVlans` (default `mgmt` + `lan`) get 443 (and 80 for
+  the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh.
+
 ## Runbook
 
 - **PPPoE down**: `systemctl status pppd-wan`, `journalctl -u pppd-wan` on the

machines/gw-cnx-1/configuration.nix

@@ -6,6 +6,7 @@
   imports = [
     ../../modules/router
     ../../modules/monitoring/exporters.nix
+    (import ../../modules/dns/acme-gw-secret.nix "gw-cnx-1")
   ];
 
   clan.core.sops.defaultGroups = [ "admins" ];
@@ -36,6 +37,17 @@
     };
     # This site runs the Omada controller for its APs/switches.
     omada.enable = true;
+
+    # Internal reverse proxy: real wildcard cert *.cnx1.cnx.network; Blocky
+    # resolves the names to the router's LAN address for mgmt+lan clients.
+    proxy = {
+      enable = true;
+      services.omada = {
+        # Omada's UI is HTTPS with a self-signed cert on the host network.
+        backend = "https://127.0.0.1:8043";
+        insecureSkipVerify = true;
+      };
+    };
   };
 
   time.timeZone = "Etc/GMT-7"; # UTC+7 (Thailand, fixed offset, no DST)

machines/ns1/configuration.nix

@@ -14,6 +14,7 @@ in
     ../../modules/dns/authoritative.nix
     ../../modules/dns/acme-mx1-secret.nix
     ../../modules/dns/acme-web01-secret.nix
+    (import ../../modules/dns/acme-gw-secret.nix "gw-cnx-1")
     ../../modules/static-ipv6.nix
     ../../modules/monitoring/exporters.nix
   ];
@@ -93,10 +94,29 @@ in
     '';
   };
 
+  # ACME DNS-01, dedicated gateway keys. Same pattern as web01: each gateway
+  # holds its own TSIG key (acme_gw_<site>_<n>), rendered from the shared
+  # secret generator imported above. acl_acme_gw_<site>_<n> scopes it to TXT
+  # updates at _acme-challenge.<site><n> on cnx.network — the owner its internal
+  # wildcard *.<site><n>.cnx.network challenge uses — and nothing else.
+  clan.core.vars.generators.dns-acme-gw-cnx-1-knot = {
+    files."acme.conf" = {
+      secret = true;
+      owner = "knot";
+      group = "knot";
+    };
+    dependencies = [ "dns-acme-gw-cnx-1-secret" ];
+    script = ''
+      printf 'key:\n  - id: acme_gw_cnx_1\n    algorithm: hmac-sha256\n    secret: %s\n' \
+        "$(cat "$in"/dns-acme-gw-cnx-1-secret/secret)" > "$out"/acme.conf
+    '';
+  };
+
   services.knot.keyFiles = [
     config.clan.core.vars.generators.dns-acme-tsig.files."acme.conf".path
     config.clan.core.vars.generators.dns-acme-mx1-knot.files."acme.conf".path
     config.clan.core.vars.generators.dns-acme-web01-knot.files."acme.conf".path
+    config.clan.core.vars.generators.dns-acme-gw-cnx-1-knot.files."acme.conf".path
   ];
 
   services.knot.settings.acl = [
@@ -133,6 +153,17 @@ in
       # i.e. _acme-challenge at the cnx.network apex (where this acl is attached).
       "update-owner-name" = [ "_acme-challenge" ];
     }
+    {
+      id = "acl_acme_gw_cnx_1";
+      key = "acme_gw_cnx_1";
+      action = [ "update" ];
+      "update-type" = [ "TXT" ];
+      "update-owner" = "name";
+      "update-owner-match" = "sub-or-equal";
+      # The internal wildcard *.cnx1.cnx.network places its challenge at
+      # _acme-challenge.cnx1.cnx.network.
+      "update-owner-name" = [ "_acme-challenge.cnx1" ];
+    }
   ];
 
   # Automatic DNSSEC signing policy (primary only). ECDSA P-256/SHA-256 with
@@ -168,6 +199,9 @@ in
       "acl_acme"
     ]
     ++ lib.optionals (d == "cnx.email") [ "acl_acme_mx1" ]
-    ++ lib.optionals (d == "cnx.network") [ "acl_acme_web01" ];
+    ++ lib.optionals (d == "cnx.network") [
+      "acl_acme_web01"
+      "acl_acme_gw_cnx_1"
+    ];
   }) domains;
 }

modules/dns/acme-gw-secret.nix

@@ -0,0 +1,17 @@
+# Shared TSIG secret for a gateway's dedicated ACME key (function: machine
+# name -> module). The acme_gw_<x> key lets that gateway — and only it — write
+# _acme-challenge.<label> TXT records on ns1 to obtain its internal wildcard
+# cert via DNS-01. ns1 scopes it with a matching acl on the cnx.network zone.
+# Import on BOTH ns1 and the gateway machine, applied with the machine name:
+#   (import ../../modules/dns/acme-gw-secret.nix "gw-cnx-1")
+machine:
+{ pkgs, ... }:
+{
+  clan.core.vars.generators."dns-acme-${machine}-secret" = {
+    share = true;
+    files."secret".secret = true;
+    runtimeInputs = [ pkgs.openssl ];
+    # 32 random bytes, base64 — a valid hmac-sha256 TSIG secret.
+    script = ''openssl rand -base64 32 | tr -d '\n' > "$out"/secret'';
+  };
+}

modules/router/default.nix

@@ -84,6 +84,7 @@ in
     ./dns-dhcp.nix
     ./crowdsec.nix
     ./omada.nix
+    ./proxy.nix
   ];
 
   options.cnx.router = {

modules/router/proxy.nix

@@ -0,0 +1,141 @@
+# Internal reverse proxy for the gateway: Caddy terminates TLS for
+# <service>.<site><siteId>.cnx.network (e.g. omada.cnx1.cnx.network) and
+# forwards to backends by their internal address. The cert is a real Let's
+# Encrypt wildcard (*.<site><siteId>.cnx.network) obtained via ACME DNS-01
+# against ns1 with a gateway-scoped TSIG key, so browsers trust it without
+# any CA install; the names only *resolve* internally — Blocky answers them
+# with the router's LAN address, the public zone never carries them.
+#
+# Requires the machine to also import the shared secret generator:
+#   (import ../../modules/dns/acme-gw-secret.nix "<hostname>")
+# and ns1 to hold the matching key + acl (see machines/ns1/configuration.nix).
+{
+  config,
+  lib,
+  ...
+}:
+let
+  cfg = config.cnx.router;
+  hosts = import ../hosts.nix;
+  hostname = config.networking.hostName;
+  tsigKey = "acme_${lib.replaceStrings [ "-" ] [ "_" ] hostname}";
+  certName = "${cfg.site}${toString cfg.siteId}.cnx.network";
+
+  serviceModule = {
+    options = {
+      backend = lib.mkOption {
+        type = lib.types.str;
+        example = "https://127.0.0.1:8043";
+        description = "URL Caddy forwards to (internal/mesh address).";
+      };
+      insecureSkipVerify = lib.mkOption {
+        type = lib.types.bool;
+        default = false;
+        description = "Skip TLS verification towards the backend (self-signed upstreams like Omada).";
+      };
+    };
+  };
+in
+{
+  options.cnx.router.proxy = {
+    enable = lib.mkEnableOption "internal reverse proxy (Caddy, wildcard cert via DNS-01)";
+
+    services = lib.mkOption {
+      type = lib.types.attrsOf (lib.types.submodule serviceModule);
+      default = { };
+      description = "Proxied services; attr name becomes <name>.${certName}.";
+    };
+
+    allowVlans = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [
+        "mgmt"
+        "lan"
+      ];
+      description = "VLANs whose clients may reach the proxy (443, plus 80 for the redirect).";
+    };
+  };
+
+  config = lib.mkIf (cfg.enable && cfg.proxy.enable) {
+    assertions = [
+      {
+        assertion = lib.all (v: cfg.vlans ? ${v}) cfg.proxy.allowVlans;
+        message = "cnx.router.proxy.allowVlans must name VLANs defined in cnx.router.vlans.";
+      }
+    ];
+
+    # Render the shared per-gateway TSIG secret into a lego rfc2136 env file;
+    # same pattern as web01 (modules/web-proxy.nix), scoped on ns1 to
+    # _acme-challenge.<site><siteId> TXT records only.
+    clan.core.vars.generators."dns-acme-${hostname}-rfc2136" = {
+      files."rfc2136.env".secret = true; # root-owned; systemd reads it as root
+      dependencies = [ "dns-acme-${hostname}-secret" ];
+      script = ''
+        printf 'RFC2136_NAMESERVER=${hosts.ns1.ipv4}:53\nRFC2136_TSIG_ALGORITHM=hmac-sha256.\nRFC2136_TSIG_KEY=${tsigKey}\nRFC2136_TSIG_SECRET=%s\n' \
+          "$(cat "$in"/dns-acme-${hostname}-secret/secret)" > "$out"/rfc2136.env
+      '';
+    };
+
+    security.acme = {
+      acceptTerms = true;
+      defaults.email = "postmaster@cnx.email";
+      # One wildcard for every proxied service; DNS-01 against ns1, so issuance
+      # works behind PPPoE with no inbound reachability at all.
+      certs.${certName} = {
+        domain = "*.${certName}";
+        dnsProvider = "rfc2136";
+        environmentFile =
+          config.clan.core.vars.generators."dns-acme-${hostname}-rfc2136".files."rfc2136.env".path;
+        # ns1 is the only nameserver that accepts this key's UPDATE; check
+        # propagation against it directly rather than a public resolver.
+        dnsResolver = "${hosts.ns1.ipv4}:53";
+        # Caddy reads the cert from explicit file paths (tls directive below),
+        # so it won't notice a renewal on its own.
+        reloadServices = [ "caddy.service" ];
+      };
+    };
+
+    # The lego-issued cert is owned group=acme; Caddy needs to read the key.
+    users.users.caddy.extraGroups = [ "acme" ];
+
+    # The explicit `tls cert key` points Caddy at the wildcard cert and disables
+    # its automatic ACME, so no extra issuance happens.
+    services.caddy = {
+      enable = true;
+      virtualHosts = lib.mapAttrs' (
+        name: svc:
+        lib.nameValuePair "${name}.${certName}" {
+          extraConfig = ''
+            tls /var/lib/acme/${certName}/cert.pem /var/lib/acme/${certName}/key.pem
+            ${
+              if svc.insecureSkipVerify then
+                ''
+                  reverse_proxy ${svc.backend} {
+                    transport http {
+                      tls_insecure_skip_verify
+                    }
+                  }''
+              else
+                "reverse_proxy ${svc.backend}"
+            }
+          '';
+        }
+      ) cfg.proxy.services;
+    };
+
+    # Blocky answers <anything>.<site><siteId>.cnx.network (customDNS covers
+    # subdomains) with the router's LAN address — clients on any allowed VLAN
+    # reach that address through the router's input path.
+    services.blocky.settings.customDNS.mapping.${certName} = cfg.vlans.lan.address;
+
+    # 443 serves the proxy; 80 only carries Caddy's automatic HTTP->HTTPS
+    # redirect. mgmt is already a trusted interface; listed anyway so shrinking
+    # trustedInterfaces later doesn't silently break the proxy.
+    networking.firewall.interfaces = lib.genAttrs (map (v: "vlan-${v}") cfg.proxy.allowVlans) (_: {
+      allowedTCPPorts = [
+        80
+        443
+      ];
+    });
+  };
+}