# Site gateways NixOS routers (OPNsense replacements) on Topton 1U fanless boxes — Intel N300, 4x i226-V 2.5G ports (some units add 2x 10G SFP+), 16GB DDR5, 256GB mSATA. Four sites are planned; the first is `gw-cnx-1` (Chiang Mai). Everything lives in the `router` clan service (`modules/clan/router/`, registered as `modules.router` in `clan.nix`); a site joins by adding its machine to the `router` inventory instance in `inventory.nix` with settings for port names, VLANs, and feature flags (`roles.default.machines..settings`). Naming: `gw--`, e.g. `gw-cnx-1`. The service has a NixOS VM test (`modules/clan/router/tests/vm/`): an ISP node runs a PPPoE access concentrator, a `client` sits on the tagged lan/iot VLANs behind the trunk, an `admin` on the untagged mgmt access port, and a simulated radio (`mac80211_hwsim`) carries two SSIDs with a WPA3 station. It checks the PPPoE dial-in, leases/reservations, Blocky, NAT, `allowWan`, the mgmt-only trust model, SSH exposure and the Wi-Fi bridge ports. Run it with `nix build .#checks.x86_64-linux.router` (also part of `nix flake check`). ## What each gateway runs | Function | Implementation | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | WAN | PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned | | LAN | VLAN-filtering bridge `br0` over the trunk ports (networkd); `accessPorts` pin a port untagged to one VLAN — convention: the last copper port is an untagged `mgmt` recovery port; `stagingPort` turns a spare port into a DHCP uplink into the old LAN while the box runs alongside the router it replaces (drop at cutover) | | Firewall/NAT | nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT | | DHCP | Kea, one subnet per VLAN | | DNS | Blocky (blocklist resolver), metrics on :4000 scraped by control | | IPv6 | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC | | Bans | CrowdSec + nftables bouncer (sshd log parsing) | | Omada | Optional per site: TP-Link Omada controller as a podman container | | Wi-Fi | Optional: hostapd on the router's radios; each SSID (`wifi.networks`) is an untagged access port of its VLAN, passphrases via vars prompts — see `modules/clan/router/README.md` | | Proxy | Optional: Caddy reverse proxy for internal services under `*..cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1) | | Diagnostics | iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c `); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli | | Speed test | Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`) | | Management | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet | | Storage | Single-disk ZFS (zstd, `/var` its own dataset for snapshots) | ## Addressing plan Each site owns `10..0.0/16`. Defaults per VLAN: subnet `10...0/24`, router at `.1`, DHCP pool `.100–.199` (`.2–.99` static/infra, `.200–.254` reserved). VLAN id convention (fleet-wide): **10 = mgmt**, **20 = lan** (both mandatory at every site), 30 = guest, 40 = iot (reserved). Sites add their own beyond that. Larger subnets (e.g. public-WiFi guest networks that outgrow a /24) are carved from the **upper half** `10..128.0/17` and set explicitly on the VLAN. The lower half stays reserved for /24s indexed by VLAN id. High-churn VLANs should also shorten `dhcp.leaseTime` (default 86400 s) so the pool recycles. First user: `gw-cnx-2` (site 2) runs the public WiFi — guest VLAN 30 at `10.2.128.0/22`, pool `10.2.128.100 – 10.2.131.250`, `dhcp.leaseTime = 3600`. | Site | siteId | mgmt | lan | site-specific VLANs | | ---- | ------ | -------------- | -------------- | --------------------------------------------------------------- | | cnx | 1 | `10.1.10.0/24` | `10.1.20.0/24` | iot 40, voip 50, dmz 60, unit1–5 110/120/130/140/150 (all /24s) | Static DHCP leases are declared per VLAN via `dhcp.reservations` (attribute name = hostname, plus `hwAddress`/`ipAddress`); park them outside the pool, in the `.2–.99` infra range or `.200–.254`. Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet only (no inter-VLAN); WAN → nothing inbound; mesh → SSH, metrics, Omada UI. ## Adding a site 1. Copy `machines/gw-cnx-1/` to `machines/gw--/` (disko, facter, hostId, timezone). Fill the real disk id in `disko.nix` (from the installer: `ls -l /dev/disk/by-id/`). 2. Add the machine to `inventory.machines` in `clan.nix`, to the `router` instance in `inventory.nix` (`roles.default.machines.gw--.settings`: `site`, `siteId` (next free number), port names, VLANs, `omada.enable`; keep the `mgmt`/`lan` VLANs), and to the machine list in `modules/mesh-hosts.nix`. Do **not** add it to `modules/hosts.nix` (dynamic PPPoE IP; clan connects over the mesh). 3. Add its node_exporter and Blocky scrape targets in `modules/monitoring/server.nix`. 4. If the site uses the internal proxy (`proxy.enable` in its settings), wire ns1: import `(import ../../modules/clan/router/acme-secret.nix "gw--")` in ns1's configuration (the gateway declares the same shared generator itself), and on ns1 add the `dns-acme-gw---knot` generator, its `keyFiles` entry, and an `acl_acme_gw__` scoped to `_acme-challenge.` on the `cnx.network` zone (copy the `gw-cnx-1` blocks). Then `clan vars generate ns1` and redeploy ns1. 5. `git add` the new machine directory — flake evaluation only sees git-tracked files, so an untracked `machines/gw-…/` is silently ignored. 6. `clan vars generate gw--` — prompts for the site's PPPoE credentials, mints the ZeroTier identity etc. (`nix flake check` fails until this has run, because mesh-hosts reads the ZeroTier IP var.) 7. Boot the box from a NixOS installer USB on the local network, then: `clan machines install gw-- --target-host root@` 8. Check `facter.json` for the real NIC names, fix `wan.interface` / `trunkPorts` if the enumeration differs, and `clan machines update gw--` (rides the mesh from then on). 9. Add a row to the site table above and to the machines table in [Overview](./overview.md). ## Omada controller Sites with TP-Link Omada APs/switches set `omada.enable = true` in their router settings. The controller runs as a podman container (`mbentley/omada-controller` — there is no nixpkgs package) with host networking on the mgmt VLAN, where adoption broadcasts (UDP 29810) live. UI: `https://:8043` from mgmt, or the gateway's mesh address from an admin machine. Controller state is under `/var/lib/omada`, declared as clan state (`clan.core.state.omada`); wiring it into the borgbackup instance is still a follow-up. ## Internal reverse proxy `proxy.enable` in a site's router settings puts Caddy on the gateway, terminating TLS for `..cnx.network` (e.g. `https://omada.cnx1.cnx.network`) and forwarding to internal backends: ```nix proxy = { enable = true; services.omada = { backend = "https://127.0.0.1:8043"; insecureSkipVerify = true; # Omada's cert is self-signed }; }; ``` - **Cert**: one real Let's Encrypt wildcard `*..cnx.network` per gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind PPPoE with zero inbound reachability, and browsers trust it without a CA install. Each gateway has its own TSIG key (`acme_gw__`) that ns1 scopes to `_acme-challenge.` TXT records only. The zone, ns1's address and the ACME contact are fleet-wide role settings in `inventory.nix` (`proxy.domain`, `proxy.acme.nameserver`, `proxy.acme.email`). - **Resolution**: the names exist only internally — Blocky answers `*..cnx.network` with the router's `lan` address; the public `cnx.network` zone never carries them. - **Access**: `proxy.allowVlans` (default `mgmt` + `lan`) get 443 (and 80 for the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh. ## Using the service from another clan The service is exported as `clan.modules.router` of this flake; add this repo as a flake input and reference it with `module.input = ""`. `modules/clan/router/README.md` is the consumer-facing description (it is also the module's manifest readme, shown by `clan modules`). ## Runbook - **PPPoE down**: `systemctl status pppd-wan`, `journalctl -u pppd-wan` on the gateway (over the mesh). ISP credentials live in the `pppoe-credentials` vars generator; re-enter with `clan vars generate gw-- --regenerate`. - **A VLAN gets no leases**: `systemctl status kea-dhcp4-server`; check the port's `bridgeVLANs` tagging and that the switch trunk carries the VLAN. - **DNS/blocklist issues**: Blocky metrics are in VictoriaMetrics (job `blocky`); `journalctl -u blocky` on the gateway. - **Slow internet reported / `WANSpeedDegraded` firing**: check the `speedtest_download_mbps` / `speedtest_ping_ms` trend in VictoriaMetrics, then `librespeed-cli` and `mtr ` on the gateway to separate ISP degradation from local saturation (`iftop` on ppp0 shows who's eating it). - **Banned yourself**: `cscli decisions list` / `cscli decisions delete --ip ` on the gateway. The ZeroTier mesh `/88` is whitelisted at the parser stage (`cnx/mesh-whitelist`), so admin access over the mesh can never be banned. ## Follow-ups - Borgbackup client for Omada + Kea lease state. - Suricata (IDS-only) if CPU headroom allows — deliberately skipped for now. - Remaining three sites.