The staging uplink is in no VLAN zone and was fully default-deny; admit
TCP 22 on it so the box is reachable from the old LAN before the mgmt
VLAN or the mesh are up. Everything else on the port stays closed.
The pre-cutover DHCP-client uplink into the old LAN was a hand-written
systemd.network block in the gw-cnx-1 machine config. Make it a router
setting next to trunkPorts/accessPorts so every replacement gateway can
stage the same way, with an assertion that the port is not also the WAN,
a trunk or an access port. gw-cnx-1 sets stagingPort = "enp3s0" in the
inventory; the machine-local block is gone.
Move the `router` service instance out of clan.nix into its own clan
module, inventory.nix, imported by clan.nix. Fleet-wide role settings
(mesh subnet, proxy domain/ACME) and the gw-cnx-1 site layout live there;
clan.nix keeps the machine list and the clan-core services. The gw-cnx-1
toplevel is byte-identical before and after the move.
wifi.* settings turn the gateway's own radios into the site AP (hostapd):
SSIDs are defined once in wifi.networks and act as untagged access ports of
their VLAN (each BSS joins br0 with the VLAN's PVID), radios pick what they
broadcast, passphrases are vars prompts. Extra SSIDs on a radio get BSSIDs
derived from its hardware address. A udev rule puts configured radios in AP
mode on appearance, since networkd cannot bridge a station-mode interface
and gives up before hostapd switches it. wifi.enable gates all of it.
The VM test grows a hwsim radio with two SSIDs and a WPA3 station in its
own netns that must get a Kea lease on the SSID's VLAN. Making the client
deterministic (route metrics, loose rp-filter, a guard against a vacuous
negative check) exposed that allowWan was never enforced: networking.nat
opens forward-to-WAN for all of its internalInterfaces, which listed every
VLAN. It now lists only the allowWan VLANs; the duplicate custom rule is
gone. No behavioural change on gw-cnx-1 (all its VLANs allow WAN).
New cnx.router.accessPorts option pins a port untagged to one VLAN via
bridge PVID/EgressUntagged; convention is the last copper port as an
always-available mgmt recovery port. gw-cnx-1 port roles: enp1s0 WAN,
enp2s0 trunk, enp3s0 temporary DHCP uplink into the old OPNsense LAN
(back to trunk at cutover), enp4s0 untagged mgmt.
The newedge.house site adopts the 10.1.<vlanId>.0/24 convention: mgmt 10
(the old untagged LAN), lan 20, iot 40, voip 50, dmz 60, unit1-5 110-150.
PPPoE is untagged at this site. Static leases move to a new per-VLAN
dhcp.reservations option rendered into Kea host reservations.
Per-gateway download/upload/ping from the hourly librespeed runs, with
each link's 7-day median overlaid so the WANSpeedDegraded alert
condition is visible at a glance. Label-driven — new sites appear
automatically once scraped.
iperf3 serves throughput tests from every VLAN and the mesh; an hourly
librespeed run feeds speedtest_* metrics through node_exporter's textfile
collector, and vmalert flags download rates below half the link's own
7-day median so ISP degradation surfaces without per-site thresholds.
cnx.router.proxy serves <service>.<site><n>.cnx.network with a real Let's
Encrypt wildcard obtained via a gateway-scoped TSIG key against ns1; Blocky
resolves the names to the router's LAN address, so they exist only
internally. First user: Omada UI on gw-cnx-1 (omada.cnx1.cnx.network).
sshd carries the admin SSH keys and now also CA-signed host certificates
(TOFU-less verification via the shared openssh-ca); the users service
generates per-machine root passwords (share = false, no prompts).
Reusable cnx.router.* module for the Topton 1U boxes replacing OPNsense:
PPPoE WAN (optionally VLAN-tagged, AIS: 10, secret credentials incl.
username), VLAN-filtering bridge, nftables NAT/firewall with MSS clamp,
Kea DHCP with per-VLAN lease time, Blocky DNS, DHCPv6-PD, CrowdSec with
the ZeroTier mesh whitelisted, optional Omada controller, ZFS disk.
Fleet baseline rides along: admins sops group is now derived for every
machine in clan.nix (secrets encrypt to it from the first vars generate)
and time sync is chrony everywhere instead of systemd-timesyncd.
kurogeek
Berwn