Site gateways

NixOS routers (OPNsense replacements) on Topton 1U fanless boxes — Intel N300, 4x i226-V 2.5G ports (some units add 2x 10G SFP+), 16GB DDR5, 256GB mSATA. Four sites are planned; the first is gw-cnx-1 (Chiang Mai). Everything lives in the router clan service (modules/clan/router/, registered as modules.router in clan.nix); a site joins by adding its machine to the router inventory instance in inventory.nix with settings for port names, VLANs, and feature flags (roles.default.machines.<gw>.settings).

Naming: gw-<city>-<n>, e.g. gw-cnx-1.

The service has a NixOS VM test (modules/clan/router/tests/vm/): an ISP node runs a PPPoE access concentrator, a client sits on the tagged lan/iot VLANs behind the trunk, an admin on the untagged mgmt access port, and a simulated radio (mac80211_hwsim) carries two SSIDs with a WPA3 station. It checks the PPPoE dial-in, leases/reservations, Blocky, NAT, allowWan, the mgmt-only trust model, SSH exposure and the Wi-Fi bridge ports. Run it with nix build .#checks.x86_64-linux.router (also part of nix flake check).

What each gateway runs

Function	Implementation
WAN	PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned
LAN	VLAN-filtering bridge `br0` over the trunk ports (networkd); `accessPorts` pin a port untagged to one VLAN — convention: the last copper port is an untagged `mgmt` recovery port; `stagingPort` turns a spare port into a DHCP uplink into the old LAN while the box runs alongside the router it replaces (drop at cutover)
Firewall/NAT	nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT
DHCP	Kea, one subnet per VLAN
DNS	Blocky (blocklist resolver), metrics on :4000 scraped by control
IPv6	DHCPv6-PD on ppp0, /64 per VLAN via SLAAC
Bans	CrowdSec + nftables bouncer (sshd log parsing)
Omada	Optional per site: TP-Link Omada controller as a podman container
Wi-Fi	Optional: hostapd on the router's radios; each SSID (`wifi.networks`) is an untagged access port of its VLAN, passphrases via vars prompts — see `modules/clan/router/README.md`
Proxy	Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1)
Diagnostics	iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c <router-vlan-addr>`); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli
Speed test	Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`)
Management	ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet
Storage	Single-disk ZFS (zstd, `/var` its own dataset for snapshots)

Addressing plan

Each site owns 10.<siteId>.0.0/16. Defaults per VLAN: subnet 10.<siteId>.<vlanId>.0/24, router at .1, DHCP pool .100–.199 (.2–.99 static/infra, .200–.254 reserved).

VLAN id convention (fleet-wide): 10 = mgmt, 20 = lan (both mandatory at every site), 30 = guest, 40 = iot (reserved). Sites add their own beyond that.

Larger subnets (e.g. public-WiFi guest networks that outgrow a /24) are carved from the upper half 10.<siteId>.128.0/17 and set explicitly on the VLAN. The lower half stays reserved for /24s indexed by VLAN id. High-churn VLANs should also shorten dhcp.leaseTime (default 86400 s) so the pool recycles. First user: gw-cnx-2 (site 2) runs the public WiFi — guest VLAN 30 at 10.2.128.0/22, pool 10.2.128.100 – 10.2.131.250, dhcp.leaseTime = 3600.

Site	siteId	mgmt	lan	site-specific VLANs
cnx	1	`10.1.10.0/24`	`10.1.20.0/24`	iot 40, voip 50, dmz 60, unit1–5 110/120/130/140/150 (all /24s)

Static DHCP leases are declared per VLAN via dhcp.reservations (attribute name = hostname, plus hwAddress/ipAddress); park them outside the pool, in the .2–.99 infra range or .200–.254.

Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet only (no inter-VLAN); WAN → nothing inbound; mesh → SSH, metrics, Omada UI.

Adding a site

Copy machines/gw-cnx-1/ to machines/gw-<city>-<n>/ (disko, facter, hostId, timezone). Fill the real disk id in disko.nix (from the installer: ls -l /dev/disk/by-id/).
Add the machine to inventory.machines in clan.nix, to the router instance in inventory.nix (roles.default.machines.gw-<city>-<n>.settings: site, siteId (next free number), port names, VLANs, omada.enable; keep the mgmt/lan VLANs), and to the machine list in modules/mesh-hosts.nix. Do not add it to modules/hosts.nix (dynamic PPPoE IP; clan connects over the mesh).
Add its node_exporter and Blocky scrape targets in modules/monitoring/server.nix.
If the site uses the internal proxy (proxy.enable in its settings), wire ns1: import (import ../../modules/clan/router/acme-secret.nix "gw-<city>-<n>") in ns1's configuration (the gateway declares the same shared generator itself), and on ns1 add the dns-acme-gw-<city>-<n>-knot generator, its keyFiles entry, and an acl_acme_gw_<city>_<n> scoped to _acme-challenge.<city><n> on the cnx.network zone (copy the gw-cnx-1 blocks). Then clan vars generate ns1 and redeploy ns1.
git add the new machine directory — flake evaluation only sees git-tracked files, so an untracked machines/gw-…/ is silently ignored.
clan vars generate gw-<city>-<n> — prompts for the site's PPPoE credentials, mints the ZeroTier identity etc. (nix flake check fails until this has run, because mesh-hosts reads the ZeroTier IP var.)
Boot the box from a NixOS installer USB on the local network, then: clan machines install gw-<city>-<n> --target-host root@<lan-ip>
Check facter.json for the real NIC names, fix wan.interface / trunkPorts if the enumeration differs, and clan machines update gw-<city>-<n> (rides the mesh from then on).
Add a row to the site table above and to the machines table in Overview.

Omada controller

Sites with TP-Link Omada APs/switches set omada.enable = true in their router settings. The controller runs as a podman container (mbentley/omada-controller — there is no nixpkgs package) with host networking on the mgmt VLAN, where adoption broadcasts (UDP 29810) live. UI: https://<mgmt-addr>:8043 from mgmt, or the gateway's mesh address from an admin machine. Controller state is under /var/lib/omada, declared as clan state (clan.core.state.omada); wiring it into the borgbackup instance is still a follow-up.

Internal reverse proxy

proxy.enable in a site's router settings puts Caddy on the gateway, terminating TLS for <service>.<site><n>.cnx.network (e.g. https://omada.cnx1.cnx.network) and forwarding to internal backends:

proxy = {
  enable = true;
  services.omada = {
    backend = "https://127.0.0.1:8043";
    insecureSkipVerify = true; # Omada's cert is self-signed
  };
};

Cert: one real Let's Encrypt wildcard *.<site><n>.cnx.network per gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind PPPoE with zero inbound reachability, and browsers trust it without a CA install. Each gateway has its own TSIG key (acme_gw_<city>_<n>) that ns1 scopes to _acme-challenge.<site><n> TXT records only. The zone, ns1's address and the ACME contact are fleet-wide role settings in inventory.nix (proxy.domain, proxy.acme.nameserver, proxy.acme.email).
Resolution: the names exist only internally — Blocky answers *.<site><n>.cnx.network with the router's lan address; the public cnx.network zone never carries them.
Access: proxy.allowVlans (default mgmt + lan) get 443 (and 80 for the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh.

Using the service from another clan

The service is exported as clan.modules.router of this flake; add this repo as a flake input and reference it with module.input = "<input name>". modules/clan/router/README.md is the consumer-facing description (it is also the module's manifest readme, shown by clan modules).

Runbook

PPPoE down: systemctl status pppd-wan, journalctl -u pppd-wan on the gateway (over the mesh). ISP credentials live in the pppoe-credentials vars generator; re-enter with clan vars generate gw-<city>-<n> --regenerate.
A VLAN gets no leases: systemctl status kea-dhcp4-server; check the port's bridgeVLANs tagging and that the switch trunk carries the VLAN.
DNS/blocklist issues: Blocky metrics are in VictoriaMetrics (job blocky); journalctl -u blocky on the gateway.
Slow internet reported / WANSpeedDegraded firing: check the speedtest_download_mbps / speedtest_ping_ms trend in VictoriaMetrics, then librespeed-cli and mtr <isp-gw> on the gateway to separate ISP degradation from local saturation (iftop on ppp0 shows who's eating it).
Banned yourself: cscli decisions list / cscli decisions delete --ip <ip> on the gateway. The ZeroTier mesh /88 is whitelisted at the parser stage (cnx/mesh-whitelist), so admin access over the mesh can never be banned.

Follow-ups

Borgbackup client for Omada + Kea lease state.
Suricata (IDS-only) if CPU headroom allows — deliberately skipped for now.
Remaining three sites.

13 KiB Raw Blame History Unescape Escape