# Site gateways NixOS routers (OPNsense replacements) on Topton 1U fanless boxes — Intel N300, 4x i226-V 2.5G ports (some units add 2x 10G SFP+), 16GB DDR5, 256GB mSATA. Four sites are planned; the first is `gw-cnx-1` (Chiang Mai). Everything lives in the reusable `modules/router/` module (`cnx.router.*` options); a site's machine config only sets port names, VLANs, and feature flags. Naming: `gw--`, e.g. `gw-cnx-1`. ## What each gateway runs | Function | Implementation | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | WAN | PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned | | LAN | VLAN-filtering bridge `br0` over the trunk ports (networkd); `accessPorts` pin a port untagged to one VLAN — convention: the last copper port is an untagged `mgmt` recovery port | | Firewall/NAT | nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT | | DHCP | Kea, one subnet per VLAN | | DNS | Blocky (blocklist resolver), metrics on :4000 scraped by control | | IPv6 | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC | | Bans | CrowdSec + nftables bouncer (sshd log parsing) | | Omada | Optional per site: TP-Link Omada controller as a podman container | | Proxy | Optional: Caddy reverse proxy for internal services under `*..cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1) | | Diagnostics | iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c `); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli | | Speed test | Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`) | | Management | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet | | Storage | Single-disk ZFS (zstd, `/var` its own dataset for snapshots) | ## Addressing plan Each site owns `10..0.0/16`. Defaults per VLAN: subnet `10...0/24`, router at `.1`, DHCP pool `.100–.199` (`.2–.99` static/infra, `.200–.254` reserved). VLAN id convention (fleet-wide): **10 = mgmt**, **20 = lan** (both mandatory at every site), 30 = guest, 40 = iot (reserved). Sites add their own beyond that. Larger subnets (e.g. public-WiFi guest networks that outgrow a /24) are carved from the **upper half** `10..128.0/17` and set explicitly on the VLAN. The lower half stays reserved for /24s indexed by VLAN id. High-churn VLANs should also shorten `dhcp.leaseTime` (default 86400 s) so the pool recycles. First user: `gw-cnx-2` (site 2) runs the public WiFi — guest VLAN 30 at `10.2.128.0/22`, pool `10.2.128.100 – 10.2.131.250`, `dhcp.leaseTime = 3600`. | Site | siteId | mgmt | lan | site-specific VLANs | | ---- | ------ | -------------- | -------------- | --------------------------------------------------------------- | | cnx | 1 | `10.1.10.0/24` | `10.1.20.0/24` | iot 40, voip 50, dmz 60, unit1–5 110/120/130/140/150 (all /24s) | Static DHCP leases are declared per VLAN via `dhcp.reservations` (attribute name = hostname, plus `hwAddress`/`ipAddress`); park them outside the pool, in the `.2–.99` infra range or `.200–.254`. Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet only (no inter-VLAN); WAN → nothing inbound; mesh → SSH, metrics, Omada UI. ## Adding a site 1. Copy `machines/gw-cnx-1/` to `machines/gw--/`; set `site`, `siteId` (next free number), port names, VLANs, and `omada.enable`. Fill the real disk id in `disko.nix` (from the installer: `ls -l /dev/disk/by-id/`). Keep the `mgmt`/`lan` VLANs. 2. Add the machine to `inventory.machines` in `clan.nix` and to the machine list in `modules/mesh-hosts.nix`. Do **not** add it to `modules/hosts.nix` (dynamic PPPoE IP; clan connects over the mesh). 3. Add its node_exporter and Blocky scrape targets in `modules/monitoring/server.nix`. 4. If the site uses the internal proxy (`cnx.router.proxy.enable`), wire ns1: import `(import ../../modules/dns/acme-gw-secret.nix "gw--")` in **both** the gateway's and ns1's configuration, and on ns1 add the `dns-acme-gw---knot` generator, its `keyFiles` entry, and an `acl_acme_gw__` scoped to `_acme-challenge.` on the `cnx.network` zone (copy the `gw-cnx-1` blocks). Then `clan vars generate ns1` and redeploy ns1. 5. `git add` the new machine directory — flake evaluation only sees git-tracked files, so an untracked `machines/gw-…/` is silently ignored. 6. `clan vars generate gw--` — prompts for the site's PPPoE credentials, mints the ZeroTier identity etc. (`nix flake check` fails until this has run, because mesh-hosts reads the ZeroTier IP var.) 7. Boot the box from a NixOS installer USB on the local network, then: `clan machines install gw-- --target-host root@` 8. Check `facter.json` for the real NIC names, fix `wan.interface` / `trunkPorts` if the enumeration differs, and `clan machines update gw--` (rides the mesh from then on). 9. Add a row to the site table above and to the machines table in [Overview](./overview.md). ## Omada controller Sites with TP-Link Omada APs/switches set `cnx.router.omada.enable = true`. The controller runs as a podman container (`mbentley/omada-controller` — there is no nixpkgs package) with host networking on the mgmt VLAN, where adoption broadcasts (UDP 29810) live. UI: `https://:8043` from mgmt, or the gateway's mesh address from an admin machine. Controller state is under `/var/lib/omada`, declared as clan state (`clan.core.state.omada`); wiring it into the borgbackup instance is still a follow-up. ## Internal reverse proxy `cnx.router.proxy.enable` puts Caddy on the gateway, terminating TLS for `..cnx.network` (e.g. `https://omada.cnx1.cnx.network`) and forwarding to internal backends: ```nix cnx.router.proxy = { enable = true; services.omada = { backend = "https://127.0.0.1:8043"; insecureSkipVerify = true; # Omada's cert is self-signed }; }; ``` - **Cert**: one real Let's Encrypt wildcard `*..cnx.network` per gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind PPPoE with zero inbound reachability, and browsers trust it without a CA install. Each gateway has its own TSIG key (`acme_gw__`) that ns1 scopes to `_acme-challenge.` TXT records only. - **Resolution**: the names exist only internally — Blocky answers `*..cnx.network` with the router's `lan` address; the public `cnx.network` zone never carries them. - **Access**: `proxy.allowVlans` (default `mgmt` + `lan`) get 443 (and 80 for the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh. ## Runbook - **PPPoE down**: `systemctl status pppd-wan`, `journalctl -u pppd-wan` on the gateway (over the mesh). ISP credentials live in the `pppoe-credentials` vars generator; re-enter with `clan vars generate gw-- --regenerate`. - **A VLAN gets no leases**: `systemctl status kea-dhcp4-server`; check the port's `bridgeVLANs` tagging and that the switch trunk carries the VLAN. - **DNS/blocklist issues**: Blocky metrics are in VictoriaMetrics (job `blocky`); `journalctl -u blocky` on the gateway. - **Slow internet reported / `WANSpeedDegraded` firing**: check the `speedtest_download_mbps` / `speedtest_ping_ms` trend in VictoriaMetrics, then `librespeed-cli` and `mtr ` on the gateway to separate ISP degradation from local saturation (`iftop` on ppp0 shows who's eating it). - **Banned yourself**: `cscli decisions list` / `cscli decisions delete --ip ` on the gateway. The ZeroTier mesh `/88` is whitelisted at the parser stage (`cnx/mesh-whitelist`), so admin access over the mesh can never be banned. ## Follow-ups - Borgbackup client for Omada + Kea lease state. - Suricata (IDS-only) if CPU headroom allows — deliberately skipped for now. - Remaining three sites.