router: make CrowdSec opt-in (crowdsec.enable, default off)

Not every site wants the ban engine (hub sync needs internet at
activation, and it is one more moving part on a small box). Gate
crowdsec.nix on a new `crowdsec.enable` option like `omada.enable`.

gw-cnx-1 keeps it on; the VM test drops its mkForce overrides.

docs/src/gateways.md

@@ -20,22 +20,22 @@ mgmt-only trust model, SSH exposure and the Wi-Fi bridge ports. Run it with
 
 ## What each gateway runs
 
-| Function     | Implementation                                                                                                                                                                                |
-| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| WAN          | PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned |
-| LAN          | VLAN-filtering bridge `br0` over the trunk ports (networkd); `accessPorts` pin a port untagged to one VLAN — convention: the last copper port is an untagged `mgmt` recovery port             |
-| Firewall/NAT | nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT                                                                                                                                  |
-| DHCP         | Kea, one subnet per VLAN                                                                                                                                                                      |
-| DNS          | Blocky (blocklist resolver), metrics on :4000 scraped by control                                                                                                                              |
-| IPv6         | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC                                                                                                                                                     |
-| Bans         | CrowdSec + nftables bouncer (sshd log parsing)                                                                                                                                                |
-| Omada        | Optional per site: TP-Link Omada controller as a podman container                                                                                                                             |
-| Wi-Fi        | Optional: hostapd on the router's radios; each SSID (`wifi.networks`) is an untagged access port of its VLAN, passphrases via vars prompts — see `modules/clan/router/README.md`              |
-| Proxy        | Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1)                                                   |
-| Diagnostics  | iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c <router-vlan-addr>`); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli                    |
-| Speed test   | Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`)                         |
-| Management   | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet                                                                                                                           |
-| Storage      | Single-disk ZFS (zstd, `/var` its own dataset for snapshots)                                                                                                                                  |
+| Function     | Implementation                                                                                                                                                                                                                                                                                                                |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| WAN          | PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned                                                                                                                                 |
+| LAN          | VLAN-filtering bridge `br0` over the trunk ports (networkd); `accessPorts` pin a port untagged to one VLAN — convention: the last copper port is an untagged `mgmt` recovery port; `stagingPort` turns a spare port into a DHCP uplink into the old LAN while the box runs alongside the router it replaces (drop at cutover) |
+| Firewall/NAT | nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT                                                                                                                                                                                                                                                                  |
+| DHCP         | Kea, one subnet per VLAN                                                                                                                                                                                                                                                                                                      |
+| DNS          | Blocky (blocklist resolver), metrics on :4000 scraped by control                                                                                                                                                                                                                                                              |
+| IPv6         | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC                                                                                                                                                                                                                                                                                     |
+| Bans         | Optional per site (`crowdsec.enable`): CrowdSec + nftables bouncer (sshd log parsing)                                                                                                                                                                                                                                         |
+| Omada        | Optional per site: TP-Link Omada controller as a podman container                                                                                                                                                                                                                                                             |
+| Wi-Fi        | Optional: hostapd on the router's radios; each SSID (`wifi.networks`) is an untagged access port of its VLAN, passphrases via vars prompts — see `modules/clan/router/README.md`                                                                                                                                              |
+| Proxy        | Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1)                                                                                                                                                                                   |
+| Diagnostics  | iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c <router-vlan-addr>`); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli                                                                                                                                                    |
+| Speed test   | Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`)                                                                                                                                                         |
+| Management   | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet                                                                                                                                                                                                                                                           |
+| Storage      | Single-disk ZFS (zstd, `/var` its own dataset for snapshots)                                                                                                                                                                                                                                                                  |
 
 ## Addressing plan
 
@@ -71,7 +71,7 @@ Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet onl
    installer: `ls -l /dev/disk/by-id/`).
 2. Add the machine to `inventory.machines` in `clan.nix`, to the `router`
    instance in `inventory.nix` (`roles.default.machines.gw-<city>-<n>.settings`: `site`,
-   `siteId` (next free number), port names, VLANs, `omada.enable`; keep the
+   `siteId` (next free number), port names, VLANs, `omada.enable`, `crowdsec.enable`; keep the
    `mgmt`/`lan` VLANs), and to the machine list in `modules/mesh-hosts.nix`.
    Do **not** add it to `modules/hosts.nix` (dynamic PPPoE IP; clan connects
    over the mesh).

inventory.nix

@@ -40,10 +40,11 @@ in
       wan.interface = "enp1s0";
       wan.vlanId = null; # this ISP runs PPPoE untagged on the port
       wan.macAddress = "a8:b8:e0:01:06:87";
-      trunkPorts = [
-        "enp2s0"
-        # "enp3s0" # STAGING: serves as the uplink until cutover (see the machine config)
-      ];
+      trunkPorts = [ "enp2s0" ];
+      # STAGING (remove at cutover, move enp3s0 back into trunkPorts):
+      # DHCP-client uplink into the existing OPNsense LAN so the box has
+      # internet + mesh while it runs alongside the old router.
+      stagingPort = "enp3s0";
       # Dedicated on-site recovery port: untagged mgmt, always available even
       # if the switch config is broken.
       accessPorts.enp4s0 = "mgmt";
@@ -76,6 +77,8 @@ in
       };
       # This site runs the Omada controller for its APs/switches.
       omada.enable = true;
+      # sshd ban engine (was unconditional before the option existed).
+      crowdsec.enable = true;
 
       # Internal reverse proxy: real wildcard cert *.cnx1.cnx.network; Blocky
       # resolves the names to the router's LAN address for mgmt+lan clients.

machines/gw-cnx-1/configuration.nix

@@ -16,18 +16,6 @@
     builtins.hashString "sha256" config.networking.hostName
   );
 
-  # STAGING (remove at cutover, and restore enp3s0 to trunkPorts in clan.nix):
-  # DHCP-client uplink into the existing OPNsense LAN so the box has internet +
-  # mesh while it runs alongside the old router. Default-deny firewall on this
-  # interface (it's in no VLAN zone); PPPoE simply retries until the WAN port
-  # is cabled. Do NOT connect the trunk ports to the production switch while
-  # staging — Kea on tag 10 would fight the OPNsense LAN DHCP in one broadcast
-  # domain.
-  systemd.network.networks."05-staging" = {
-    matchConfig.Name = "enp3s0";
-    networkConfig.DHCP = "ipv4";
-  };
-
   time.timeZone = "Etc/GMT-7"; # UTC+7 (Thailand, fixed offset, no DST)
   services.chrony.enable = true;
 }

modules/clan/router/README.md

@@ -3,10 +3,11 @@
 Turns a machine with several NICs into a site gateway: PPPoE WAN (ISP
 credentials via vars prompts), a VLAN-filtering bridge over the LAN ports with
 one L3 interface per VLAN, Kea DHCP and Blocky DNS per VLAN, nftables
-firewall/NAT, DHCPv6-PD, CrowdSec, an iperf3 server and a WAN speed-test
-timer. Optional: a Wi-Fi access point on the router's own radios (hostapd),
-the TP-Link Omada controller (podman) and an internal Caddy reverse proxy
-with a real wildcard certificate (ACME DNS-01).
+firewall/NAT, DHCPv6-PD, an iperf3 server and a WAN speed-test timer.
+Optional: a Wi-Fi access point on the router's own radios (hostapd), CrowdSec
+with the nftables bouncer (sshd log parsing), the TP-Link Omada controller
+(podman) and an internal Caddy reverse proxy with a real wildcard certificate
+(ACME DNS-01).
 
 Addressing convention: a site owns `10.<siteId>.0.0/16`; VLAN `<id>` defaults
 to `10.<siteId>.<id>.0/24`, router at `.1`, DHCP pool `.100-.199`. The `mgmt`
@@ -32,6 +33,7 @@ inventory.instances.router = {
     wan.vlanId = 10;                 # or null for untagged PPPoE
     trunkPorts = [ "enp2s0" ];
     accessPorts.enp4s0 = "mgmt";     # untagged on-site recovery port
+    # stagingPort = "enp3s0";       # DHCP uplink into the old LAN until cutover
     vlans = {
       mgmt.id = 10;
       lan.id = 20;

modules/clan/router/crowdsec.nix

@@ -1,46 +1,49 @@
 # CrowdSec security engine + nftables bouncer: parses sshd auth attempts from
 # the journal and bans offending source IPs at the firewall. Log-based (no
-# inline DPI) so it costs the N300 next to nothing.
+# inline DPI) so it costs the N300 next to nothing. Opt-in per site
+# (`crowdsec.enable`): the hub sync needs internet at activation time.
 { settings }:
-{ ... }:
+{ lib, ... }:
 let
   cfg = settings;
 in
 {
-  services.crowdsec = {
-    enable = true;
-    autoUpdateService = true;
-    hub.collections = [
-      "crowdsecurity/linux"
-      "crowdsecurity/sshd"
-    ];
-    localConfig = {
-      acquisitions = [
-        {
-          source = "journalctl";
-          journalctl_filter = [ "_SYSTEMD_UNIT=sshd.service" ];
-          labels.type = "syslog";
-        }
-      ];
-      # Never ban the ZeroTier mesh — it is the only admin path to these
-      # boxes (no public SSH), so a false positive would lock us out.
-      # Parser-stage whitelist: mesh events are dropped before any scenario.
-      parsers.s02Enrich = [
-        {
-          name = "cnx/mesh-whitelist";
-          description = "Whitelist the ZeroTier management mesh";
-          whitelist = {
-            reason = "ZeroTier mesh is the admin path";
-            cidr = [ cfg.mesh.subnet ];
-          };
-        }
+  config = lib.mkIf cfg.crowdsec.enable {
+    services.crowdsec = {
+      enable = true;
+      autoUpdateService = true;
+      hub.collections = [
+        "crowdsecurity/linux"
+        "crowdsecurity/sshd"
       ];
+      localConfig = {
+        acquisitions = [
+          {
+            source = "journalctl";
+            journalctl_filter = [ "_SYSTEMD_UNIT=sshd.service" ];
+            labels.type = "syslog";
+          }
+        ];
+        # Never ban the ZeroTier mesh — it is the only admin path to these
+        # boxes (no public SSH), so a false positive would lock us out.
+        # Parser-stage whitelist: mesh events are dropped before any scenario.
+        parsers.s02Enrich = [
+          {
+            name = "cnx/mesh-whitelist";
+            description = "Whitelist the ZeroTier management mesh";
+            whitelist = {
+              reason = "ZeroTier mesh is the admin path";
+              cidr = [ cfg.mesh.subnet ];
+            };
+          }
+        ];
+      };
+    };
+
+    services.crowdsec-firewall-bouncer = {
+      enable = true;
+      registerBouncer.enable = true;
+      settings.mode = "nftables";
     };
   };
-
-  services.crowdsec-firewall-bouncer = {
-    enable = true;
-    registerBouncer.enable = true;
-    settings.mode = "nftables";
-  };
 }

modules/clan/router/dns-dhcp.nix

@@ -12,7 +12,19 @@ in
   services.kea.dhcp4 = {
     enable = true;
     settings = {
-      interfaces-config.interfaces = lib.mapAttrsToList (name: _: "vlan-${name}") dhcpVlans;
+      interfaces-config = {
+        interfaces = lib.mapAttrsToList (name: _: "vlan-${name}") dhcpVlans;
+        # The unit orders after network-online.target, which under networkd
+        # only waits for the WAN carrier (the vlan-* links are
+        # RequiredForOnline=no), so Kea can start before vlan-* have their
+        # addresses. By default it then logs the failed bind and runs with no
+        # socket at all: clients' DISCOVERs reach vlan-lan and nobody answers.
+        # Insist on every socket and keep retrying while networkd catches up;
+        # if it still cannot bind, exit and let systemd restart the unit.
+        service-sockets-require-all = true;
+        service-sockets-max-retries = 60;
+        service-sockets-retry-wait-time = 1000;
+      };
       lease-database = {
         type = "memfile";
         persist = true;
@@ -44,6 +56,10 @@ in
     };
   };
 
+  # The nixpkgs unit already has Restart=on-failure; space the restarts out so
+  # a persistent bind failure does not trip the start-rate limit.
+  systemd.services.kea-dhcp4-server.serviceConfig.RestartSec = 5;
+
   services.blocky = {
     enable = true;
     settings = {

modules/clan/router/firewall.nix

@@ -3,6 +3,8 @@
 #   other VLANs -> DNS/DHCP on the router + WAN (if allowWan); no inter-VLAN
 #   WAN (ppp0) -> nothing inbound beyond established/related
 #   mesh       -> admin SSH + metrics scrapes (same trust boundary as the fleet)
+#   staging    -> admin SSH only inbound (pre-cutover uplink into the old LAN);
+#                 allowWan VLANs are NATed out through it while ppp0 is down
 { settings }:
 { lib, ... }:
 let
@@ -13,6 +15,13 @@ let
     lib.filterAttrs (_: vlan: vlan.allowWan) cfg.vlans
   );
   nonMgmtIfs = lib.filter (i: i != "vlan-mgmt") vlanIfs;
+
+  # allowWan VLANs may also leave through the staging uplink. Same set as
+  # networking.nat.internalInterfaces below, so `allowWan` holds on both
+  # exits. The kernel picks the exit: ppp0 (metric 0, see pppoe.nix) while
+  # the session is up, the staging DHCP route (metric 1024) otherwise.
+  stagingExit = cfg.stagingPort != null && wanVlanIfs != [ ];
+  wanVlanSet = "{ ${lib.concatMapStringsSep ", " (i: ''"${i}"'') wanVlanIfs} }";
 in
 {
   networking.nftables.enable = true;
@@ -26,14 +35,20 @@ in
     filterForward = true;
     trustedInterfaces = [ "vlan-mgmt" ];
 
-    # Non-mgmt VLANs may only talk to the router's DNS and DHCP.
-    interfaces = lib.genAttrs nonMgmtIfs (_: {
-      allowedTCPPorts = [ 53 ];
-      allowedUDPPorts = [
-        53
-        67
-      ];
-    });
+    # Non-mgmt VLANs may only talk to the router's DNS and DHCP; the staging
+    # uplink (old LAN, pre-cutover) gets admin SSH so the box can be reached
+    # before the mgmt VLAN or the mesh are up.
+    interfaces =
+      lib.genAttrs nonMgmtIfs (_: {
+        allowedTCPPorts = [ 53 ];
+        allowedUDPPorts = [
+          53
+          67
+        ];
+      })
+      // lib.optionalAttrs (cfg.stagingPort != null) {
+        ${cfg.stagingPort}.allowedTCPPorts = [ 22 ];
+      };
 
     extraInputRules = ''
       ip6 saddr ${cfg.mesh.subnet} tcp dport 22 accept comment "admin ssh over the mesh"
@@ -43,6 +58,9 @@ in
     extraForwardRules = ''
       tcp flags syn tcp option maxseg size set rt mtu comment "MSS clamp for PPPoE mtu 1492"
       iifname "vlan-mgmt" accept comment "mgmt reaches all VLANs and the WAN"
+    ''
+    + lib.optionalString stagingExit ''
+      iifname ${wanVlanSet} oifname "${cfg.stagingPort}" accept comment "allowWan VLANs out via the staging uplink"
     '';
   };
 
@@ -55,4 +73,19 @@ in
     externalInterface = "ppp0";
     internalInterfaces = wanVlanIfs;
   };
+
+  # networking.nat only masquerades on its single externalInterface; the
+  # staging uplink needs its own postrouting chain (nixos-nat's is
+  # oifname-scoped to ppp0, so the two never both apply).
+  networking.nftables.tables = lib.optionalAttrs stagingExit {
+    router-staging-nat = {
+      family = "ip";
+      content = ''
+        chain post {
+          type nat hook postrouting priority srcnat;
+          iifname ${wanVlanSet} oifname "${cfg.stagingPort}" masquerade comment "allowWan VLANs out via the staging uplink"
+        }
+      '';
+    };
+  };
 }

modules/clan/router/interface.nix

@@ -250,6 +250,24 @@ in
       '';
     };
 
+    stagingPort = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      example = "enp3s0";
+      description = ''
+        Temporary DHCPv4-client uplink into the existing LAN while the box
+        runs alongside the router it replaces: gives it (and, NATed, the
+        allowWan VLANs) internet + mesh before the WAN port is cabled; once
+        the PPPoE session is up its default route wins, and the staging
+        route only carries traffic again if the session drops (PPPoE simply
+        retries until then). The port is in no VLAN zone; inbound, the
+        firewall admits only SSH on it. Do NOT connect the trunk ports to
+        the production switch while staging — Kea on the mgmt tag would
+        fight the old router's DHCP in one broadcast domain. Set to null at
+        cutover (and usually hand the port back to `trunkPorts`).
+      '';
+    };
+
     vlans = lib.mkOption {
       type = lib.types.attrsOf (lib.types.submodule vlanModule);
       description = "VLANs served at this site; `mgmt` and `lan` are mandatory.";
@@ -267,6 +285,8 @@ in
 
     omada.enable = lib.mkEnableOption "TP-Link Omada SDN controller (podman container)";
 
+    crowdsec.enable = lib.mkEnableOption "CrowdSec (sshd log parsing) with the nftables bouncer";
+
     proxy = {
       enable = lib.mkEnableOption "internal reverse proxy (Caddy, wildcard cert via DNS-01)";
 

modules/clan/router/network.nix

@@ -21,6 +21,16 @@ in
       assertion = lib.all (p: !(cfg.accessPorts ? ${p})) cfg.trunkPorts;
       message = "router: a port cannot be both a trunk and an access port.";
     }
+    {
+      assertion =
+        cfg.stagingPort == null
+        || !(
+          cfg.stagingPort == cfg.wan.interface
+          || lib.elem cfg.stagingPort cfg.trunkPorts
+          || cfg.accessPorts ? ${cfg.stagingPort}
+        );
+      message = "router: stagingPort ${toString cfg.stagingPort} is also the WAN, a trunk or an access port.";
+    }
   ];
 
   # Router diagnostics toolkit: packets (tcpdump), path (mtr), link
@@ -73,7 +83,15 @@ in
     let
       taggedAll = lib.mapAttrsToList (_: vlan: { VLAN = vlan.id; }) cfg.vlans;
     in
-    {
+    lib.optionalAttrs (cfg.stagingPort != null) {
+      # Staging uplink (see interface.nix): plain DHCPv4 client on a spare
+      # port, no bridge/VLAN membership, so the firewall treats it as untrusted.
+      "05-staging" = {
+        matchConfig.Name = cfg.stagingPort;
+        networkConfig.DHCP = "ipv4";
+      };
+    }
+    // {
       # WAN port carries only the PPPoE session; no IP config of its own.
       "10-wan" = {
         matchConfig.Name = cfg.wan.interface;

modules/clan/router/pppoe.nix

@@ -30,6 +30,11 @@ in
     '';
   };
 
+  # defaultroute-metric 0: pppd refuses `defaultroute` while any other
+  # default route exists (e.g. the staging uplink's DHCP route, metric 1024,
+  # network.nix) unless given a metric; with 0 it only checks for a metric-0
+  # route, installs its own as the preferred exit, and removes it again on
+  # hangup so the staging route takes over.
   services.pppd = {
     enable = true;
     peers.wan = {
@@ -40,6 +45,7 @@ in
         file ${creds.files."user-opts".path}
         noipdefault
         defaultroute
+        defaultroute-metric 0
         noauth
         hide-password
         persist

modules/clan/router/tests/vm/default.nix

@@ -1,14 +1,17 @@
 # End-to-end VM test of the router service: a PPPoE access concentrator plays
 # the ISP on the WAN port, a trunk carries tagged lan/iot VLANs to `client`,
-# and an untagged access port carries mgmt to `admin`.
+# an untagged access port carries mgmt to `admin`, and `oldlan` is the DHCP
+# network the box is staged in before cutover.
 #
 #   isp ---(vlan 1: PPPoE)--- wan [gw] trunk ---(vlan 2: tagged 20/40)--- client
 #                                    access --(vlan 3: untagged mgmt)--- admin
+#                                    staging -(vlan 4: DHCP client)--- oldlan
 #
 # What is proven: PPPoE dial-in with the vars-provided credentials, bridge
 # VLAN tagging/untagging, Kea leases and reservations per VLAN, Blocky
-# answering on the VLAN with the blocklist active, NAT to the WAN, and the
-# firewall trust model (allowWan, mgmt-only SSH, no inter-VLAN forwarding).
+# answering on the VLAN with the blocklist active, NAT to the WAN, the
+# firewall trust model (allowWan, mgmt-only SSH, no inter-VLAN forwarding),
+# and the staging uplink as NATed fallback exit behind ppp0.
 { pkgs, lib, ... }:
 let
   # The vars mock answers every prompt with "mock-prompt-value-<name>"; the
@@ -20,6 +23,10 @@ let
   clientAddress = "10.9.20.50";
   adminMac = "02:00:00:00:00:10";
   adminAddress = "10.9.10.50";
+  oldlanAddress = "192.168.88.1";
+  # Only reachable through oldlan's router role, i.e. via gw's staging
+  # default route (metric 1024); ppp0's metric-0 default must win while up.
+  beyondStaging = "203.0.113.1";
 in
 {
   name = "router";
@@ -36,6 +43,7 @@ in
         isp = { };
         client = { };
         admin = { };
+        oldlan = { };
       };
 
       instances.router = {
@@ -48,6 +56,7 @@ in
           wan.interface = "wan";
           trunkPorts = [ "trunk" ];
           accessPorts.access = "mgmt";
+          stagingPort = "staging";
           vlans = {
             mgmt = {
               id = 10;
@@ -111,6 +120,10 @@ in
           vlan = 3;
           assignIP = false;
         };
+        staging = {
+          vlan = 4;
+          assignIP = false;
+        };
       };
 
       # Something must listen on 22 for the mgmt-only SSH rule to be observable
@@ -118,13 +131,11 @@ in
       services.openssh.enable = true;
 
       # The sandbox has no internet: serve the blocklist from a local file
-      # instead of GitHub, and skip CrowdSec, whose hub sync needs the network
-      # (it is not what this test exercises).
+      # instead of GitHub. (CrowdSec, whose hub sync needs the network too,
+      # is opt-in and stays off.)
       services.blocky.settings.blocking.denylists.ads = lib.mkForce [
         (toString (pkgs.writeText "ads.hosts" "0.0.0.0 ads.example.com\n"))
       ];
-      services.crowdsec.enable = lib.mkForce false;
-      services.crowdsec-firewall-bouncer.enable = lib.mkForce false;
 
       # Two simulated radios: wlan0 is the AP (settings above), wlan1 plays a
       # wireless client. It lives in its own network namespace, like the
@@ -251,6 +262,32 @@ in
       };
       environment.systemPackages = [ pkgs.netcat ];
     };
+
+    # The LAN the box is staged in: a DHCP server handing gw its uplink
+    # lease, plus an address that is only reachable via that uplink's
+    # default route. No route back to 10.9.0.0/16: replies only reach the
+    # clients if gw masquerades them.
+    oldlan = {
+      virtualisation.interfaces.staging = {
+        vlan = 4;
+        assignIP = false;
+      };
+      networking.useDHCP = false;
+      networking.useNetworkd = true;
+      systemd.network.networks."10-staging" = {
+        matchConfig.Name = "staging";
+        address = [
+          "${oldlanAddress}/24"
+          "${beyondStaging}/32"
+        ];
+        networkConfig.DHCPServer = true;
+        dhcpServerConfig = {
+          PoolOffset = 100;
+          PoolSize = 50;
+        };
+      };
+      networking.firewall.allowedUDPPorts = [ 67 ];
+    };
   };
 
   testScript = ''
@@ -302,5 +339,21 @@ in
         gw.wait_until_succeeds("ip netns exec sta wpa_cli -i wlan1 status | grep -q wpa_state=COMPLETED")
         gw.succeed("timeout 60 sta-dhcp")
         gw.succeed("ip netns exec sta ip -4 addr show wlan1 | grep -q 'inet 10.9.20.1[0-9][0-9]/24'")
+
+    with subtest("Staging uplink: NATed exit for allowWan VLANs, behind ppp0 while it is up"):
+        gw.wait_until_succeeds("ip -4 route show default dev staging | grep -q 'via ${oldlanAddress}'")
+        # pppd installs its default route despite the DHCP one (defaultroute-metric 0).
+        gw.succeed("ip route get ${beyondStaging} | grep -q 'dev ppp0'")
+        # On-link old-LAN hosts are reached through the staging port regardless.
+        client.succeed("ping -c1 -W2 -I lan0 ${oldlanAddress}")
+        client.fail("ping -c1 -W2 -I iot0 ${oldlanAddress}")
+        # ppp0 down: the staging route carries the WAN traffic, allowWan still holds.
+        gw.systemctl("stop pppd-wan.service")
+        gw.wait_until_succeeds("ip route get ${beyondStaging} | grep -q 'dev staging'")
+        client.succeed("ping -c1 -W2 -I lan0 ${beyondStaging}")
+        client.fail("ping -c1 -W2 -I iot0 ${beyondStaging}")
+        # ppp0 back: preferred again.
+        gw.systemctl("start pppd-wan.service")
+        gw.wait_until_succeeds("ip route get ${beyondStaging} | grep -q 'dev ppp0'")
   '';
 }

modules/clan/router/wifi.nix

@@ -12,7 +12,6 @@
 {
   config,
   lib,
-  pkgs,
   ...
 }:
 let
@@ -121,26 +120,18 @@ in
         );
         message = "router: wifi.radios.<radio>.macAddress is required for radios broadcasting more than one network.";
       }
+      {
+        assertion =
+          !config.networking.wireless.enable
+          && !config.networking.wireless.iwd.enable
+          && !config.networking.networkmanager.enable;
+        message = "router: wifi.enable needs the radios for hostapd; disable networking.wireless (wpa_supplicant), iwd and NetworkManager.";
+      }
     ];
 
     # Regulatory database for the kernel, so countryCode actually applies.
     hardware.wirelessRegulatoryDatabase = true;
 
-    # The kernel refuses to bridge a wireless interface in station mode, and
-    # networkd stops retrying before hostapd switches the radio to AP mode;
-    # so put it in AP mode the moment it appears (kernel name or the renamed
-    # one, whichever the user configured). hostapd finds it already there.
-    services.udev.extraRules = lib.concatMapStrings (
-      radio:
-      let
-        run = ''RUN+="${pkgs.iw}/bin/iw dev ${radio} set type __ap"'';
-      in
-      ''
-        ACTION=="add", SUBSYSTEM=="net", KERNEL=="${radio}", ${run}
-        ACTION=="add", SUBSYSTEM=="net", NAME=="${radio}", ${run}
-      ''
-    ) (lib.attrNames wifi.radios);
-
     clan.core.vars.generators = lib.genAttrs (map (name: "wifi-${name}-passphrase") secured) (
       gen:
       let
@@ -184,12 +175,22 @@ in
     };
 
     # Each BSS is an untagged access port of its VLAN on br0 (cf. accessPorts
-    # in network.nix); networkd enslaves the interface once hostapd creates it.
+    # in network.nix). The kernel refuses to bridge a wireless interface in
+    # station mode (IFF_DONT_BRIDGE), and a failed enslave parks the link in
+    # networkd's `failed` state for good: networkd only re-evaluates a link
+    # when its matching .network file changes. So match on the AP interface
+    # type as well as the name: the radio's initial station-mode netdev
+    # matches nothing (unmanaged), and once hostapd switches it to AP and
+    # brings the carrier up, networkd matches this file for the first time
+    # and enslaves it. Extra BSSes are created by hostapd in AP mode already.
     systemd.network.networks = lib.listToAttrs (
       map (
         b:
         lib.nameValuePair "27-wifi-${b.iface}" {
-          matchConfig.Name = b.iface;
+          matchConfig = {
+            Name = b.iface;
+            WLANInterfaceType = "ap";
+          };
           networkConfig.Bridge = "br0";
           bridgeVLANs = [
             {