cnx-network-clan/docs/src/gateways.md

Site gateways

NixOS routers (OPNsense replacements) on Topton 1U fanless boxes — Intel N300, 4x i226-V 2.5G ports (some units add 2x 10G SFP+), 16GB DDR5, 256GB mSATA. Four sites are planned; the first is gw-cnx-1 (Chiang Mai). Everything lives in the reusable modules/router/ module (cnx.router.* options); a site's machine config only sets port names, VLANs, and feature flags.

Naming: gw-<city>-<n>, e.g. gw-cnx-1.

What each gateway runs

Function	Implementation
WAN	PPPoE (pppd), per-site ISP credentials via clan vars prompts; wan.vlanId when the ISP tags the session (AIS: 10); wan.macAddress to clone the old router's MAC if the ISP has it pinned
LAN	VLAN-filtering bridge br0 over the trunk ports (networkd); accessPorts pin a port untagged to one VLAN — convention: the last copper port is an untagged mgmt recovery port
Firewall/NAT	nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT
DHCP	Kea, one subnet per VLAN
DNS	Blocky (blocklist resolver), metrics on :4000 scraped by control
IPv6	DHCPv6-PD on ppp0, /64 per VLAN via SLAAC
Bans	CrowdSec + nftables bouncer (sshd log parsing)
Omada	Optional per site: TP-Link Omada controller as a podman container
Proxy	Optional: Caddy reverse proxy for internal services under *.<site><n>.cnx.network with a real Let's Encrypt wildcard (DNS-01 against ns1)
Diagnostics	iperf3 server on 5201, reachable from every VLAN and the mesh (iperf3 -c <router-vlan-addr>); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli
Speed test	Hourly librespeed run (speedtest.timer) → speedtest_* metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (WANSpeedDegraded)
Management	ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet
Storage	Single-disk ZFS (zstd, /var its own dataset for snapshots)

Addressing plan

Each site owns 10.<siteId>.0.0/16. Defaults per VLAN: subnet 10.<siteId>.<vlanId>.0/24, router at .1, DHCP pool .100–.199 (.2–.99 static/infra, .200–.254 reserved).

VLAN id convention (fleet-wide): 10 = mgmt, 20 = lan (both mandatory at every site), 30 = guest, 40 = iot (reserved). Sites add their own beyond that.

Larger subnets (e.g. public-WiFi guest networks that outgrow a /24) are carved from the upper half 10.<siteId>.128.0/17 and set explicitly on the VLAN. The lower half stays reserved for /24s indexed by VLAN id. High-churn VLANs should also shorten dhcp.leaseTime (default 86400 s) so the pool recycles. First user: gw-cnx-2 (site 2) runs the public WiFi — guest VLAN 30 at 10.2.128.0/22, pool 10.2.128.100 – 10.2.131.250, dhcp.leaseTime = 3600.

Site	siteId	mgmt	lan	site-specific VLANs
cnx	1	10.1.10.0/24	10.1.20.0/24	iot 40, voip 50, dmz 60, unit1–5 110/120/130/140/150 (all /24s)

Static DHCP leases are declared per VLAN via dhcp.reservations (attribute name = hostname, plus hwAddress/ipAddress); park them outside the pool, in the .2–.99 infra range or .200–.254.

Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet only (no inter-VLAN); WAN → nothing inbound; mesh → SSH, metrics, Omada UI.

Adding a site

1. Copy machines/gw-cnx-1/ to machines/gw-<city>-<n>/; set site, siteId (next free number), port names, VLANs, and omada.enable. Fill the real disk id in disko.nix (from the installer: ls -l /dev/disk/by-id/). Keep the mgmt/lan VLANs.
2. Add the machine to inventory.machines in clan.nix and to the machine list in modules/mesh-hosts.nix. Do not add it to modules/hosts.nix (dynamic PPPoE IP; clan connects over the mesh).
3. Add its node_exporter and Blocky scrape targets in modules/monitoring/server.nix.
4. If the site uses the internal proxy (cnx.router.proxy.enable), wire ns1: import (import ../../modules/dns/acme-gw-secret.nix "gw-<city>-<n>") in both the gateway's and ns1's configuration, and on ns1 add the dns-acme-gw-<city>-<n>-knot generator, its keyFiles entry, and an acl_acme_gw_<city>_<n> scoped to _acme-challenge.<city><n> on the cnx.network zone (copy the gw-cnx-1 blocks). Then clan vars generate ns1 and redeploy ns1.
5. git add the new machine directory — flake evaluation only sees git-tracked files, so an untracked machines/gw-…/ is silently ignored.
6. clan vars generate gw-<city>-<n> — prompts for the site's PPPoE credentials, mints the ZeroTier identity etc. (nix flake check fails until this has run, because mesh-hosts reads the ZeroTier IP var.)
7. Boot the box from a NixOS installer USB on the local network, then: clan machines install gw-<city>-<n> --target-host root@<lan-ip>
8. Check facter.json for the real NIC names, fix wan.interface / trunkPorts if the enumeration differs, and clan machines update gw-<city>-<n> (rides the mesh from then on).
9. Add a row to the site table above and to the machines table in Overview.

Omada controller

Sites with TP-Link Omada APs/switches set cnx.router.omada.enable = true. The controller runs as a podman container (mbentley/omada-controller — there is no nixpkgs package) with host networking on the mgmt VLAN, where adoption broadcasts (UDP 29810) live. UI: https://<mgmt-addr>:8043 from mgmt, or the gateway's mesh address from an admin machine. Controller state is under /var/lib/omada, declared as clan state (clan.core.state.omada); wiring it into the borgbackup instance is still a follow-up.

Internal reverse proxy

cnx.router.proxy.enable puts Caddy on the gateway, terminating TLS for <service>.<site><n>.cnx.network (e.g. https://omada.cnx1.cnx.network) and forwarding to internal backends:

cnx.router.proxy = {
  enable = true;
  services.omada = {
    backend = "https://127.0.0.1:8043";
    insecureSkipVerify = true; # Omada's cert is self-signed
  };
};

• Cert: one real Let's Encrypt wildcard *.<site><n>.cnx.network per gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind PPPoE with zero inbound reachability, and browsers trust it without a CA install. Each gateway has its own TSIG key (acme_gw_<city>_<n>) that ns1 scopes to _acme-challenge.<site><n> TXT records only.
• Resolution: the names exist only internally — Blocky answers *.<site><n>.cnx.network with the router's lan address; the public cnx.network zone never carries them.
• Access: proxy.allowVlans (default mgmt + lan) get 443 (and 80 for the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh.

Runbook

• PPPoE down: systemctl status pppd-wan, journalctl -u pppd-wan on the gateway (over the mesh). ISP credentials live in the pppoe-credentials vars generator; re-enter with clan vars generate gw-<city>-<n> --regenerate.
• A VLAN gets no leases: systemctl status kea-dhcp4-server; check the port's bridgeVLANs tagging and that the switch trunk carries the VLAN.
• DNS/blocklist issues: Blocky metrics are in VictoriaMetrics (job blocky); journalctl -u blocky on the gateway.
• Slow internet reported / WANSpeedDegraded firing: check the speedtest_download_mbps / speedtest_ping_ms trend in VictoriaMetrics, then librespeed-cli and mtr <isp-gw> on the gateway to separate ISP degradation from local saturation (iftop on ppp0 shows who's eating it).
• Banned yourself: cscli decisions list / cscli decisions delete --ip <ip> on the gateway. The ZeroTier mesh /88 is whitelisted at the parser stage (cnx/mesh-whitelist), so admin access over the mesh can never be banned.

Follow-ups

• Borgbackup client for Omada + Kea lease state.
• Suricata (IDS-only) if CPU headroom allows — deliberately skipped for now.
• Remaining three sites.