cnx-network-clan/docs/src/gateways.md

# Site gateways

NixOS routers (OPNsense replacements) on Topton 1U fanless boxes — Intel N300,
4x i226-V 2.5G ports (some units add 2x 10G SFP+), 16GB DDR5, 256GB mSATA.
Four sites are planned; the first is `gw-cnx-1` (Chiang Mai). Everything lives
in the reusable `modules/router/` module (`cnx.router.*` options); a site's
machine config only sets port names, VLANs, and feature flags.

Naming: `gw-<city>-<n>`, e.g. `gw-cnx-1`.

## What each gateway runs

| Function     | Implementation                                                                                                                                                                                |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| WAN          | PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned |
| LAN          | VLAN-filtering bridge `br0` over the trunk ports (networkd); `accessPorts` pin a port untagged to one VLAN — convention: the last copper port is an untagged `mgmt` recovery port             |
| Firewall/NAT | nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT                                                                                                                                  |
| DHCP         | Kea, one subnet per VLAN                                                                                                                                                                      |
| DNS          | Blocky (blocklist resolver), metrics on :4000 scraped by control                                                                                                                              |
| IPv6         | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC                                                                                                                                                     |
| Bans         | CrowdSec + nftables bouncer (sshd log parsing)                                                                                                                                                |
| Omada        | Optional per site: TP-Link Omada controller as a podman container                                                                                                                             |
| Proxy        | Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1)                                                   |
| Diagnostics  | iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c <router-vlan-addr>`); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli                    |
| Speed test   | Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`)                         |
| Management   | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet                                                                                                                           |
| Storage      | Single-disk ZFS (zstd, `/var` its own dataset for snapshots)                                                                                                                                  |

## Addressing plan

Each site owns `10.<siteId>.0.0/16`. Defaults per VLAN: subnet
`10.<siteId>.<vlanId>.0/24`, router at `.1`, DHCP pool `.100–.199`
(`.2–.99` static/infra, `.200–.254` reserved).

VLAN id convention (fleet-wide): **10 = mgmt**, **20 = lan** (both mandatory at
every site), 30 = guest, 40 = iot (reserved). Sites add their own beyond that.

Larger subnets (e.g. public-WiFi guest networks that outgrow a /24) are carved
from the **upper half** `10.<siteId>.128.0/17` and set explicitly on the VLAN.
The lower half stays reserved for /24s indexed by VLAN id. High-churn VLANs
should also shorten `dhcp.leaseTime` (default 86400 s) so the pool recycles.
First user: `gw-cnx-2` (site 2) runs the public WiFi — guest VLAN 30 at
`10.2.128.0/22`, pool `10.2.128.100 – 10.2.131.250`, `dhcp.leaseTime = 3600`.

| Site | siteId | mgmt           | lan            | site-specific VLANs                                             |
| ---- | ------ | -------------- | -------------- | --------------------------------------------------------------- |
| cnx  | 1      | `10.1.10.0/24` | `10.1.20.0/24` | iot 40, voip 50, dmz 60, unit1–5 110/120/130/140/150 (all /24s) |

Static DHCP leases are declared per VLAN via `dhcp.reservations` (attribute
name = hostname, plus `hwAddress`/`ipAddress`); park them outside the pool,
in the `.2–.99` infra range or `.200–.254`.

Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet only
(no inter-VLAN); WAN → nothing inbound; mesh → SSH, metrics, Omada UI.

## Adding a site

1. Copy `machines/gw-cnx-1/` to `machines/gw-<city>-<n>/`; set `site`,
   `siteId` (next free number), port names, VLANs, and `omada.enable`.
   Fill the real disk id in `disko.nix` (from the installer:
   `ls -l /dev/disk/by-id/`). Keep the `mgmt`/`lan` VLANs.
2. Add the machine to `inventory.machines` in `clan.nix` and to the machine
   list in `modules/mesh-hosts.nix`. Do **not** add it to `modules/hosts.nix`
   (dynamic PPPoE IP; clan connects over the mesh).
3. Add its node_exporter and Blocky scrape targets in
   `modules/monitoring/server.nix`.
4. If the site uses the internal proxy (`cnx.router.proxy.enable`), wire ns1:
   import `(import ../../modules/dns/acme-gw-secret.nix "gw-<city>-<n>")` in
   **both** the gateway's and ns1's configuration, and on ns1 add the
   `dns-acme-gw-<city>-<n>-knot` generator, its `keyFiles` entry, and an
   `acl_acme_gw_<city>_<n>` scoped to `_acme-challenge.<city><n>` on the
   `cnx.network` zone (copy the `gw-cnx-1` blocks). Then
   `clan vars generate ns1` and redeploy ns1.
5. `git add` the new machine directory — flake evaluation only sees
   git-tracked files, so an untracked `machines/gw-…/` is silently ignored.
6. `clan vars generate gw-<city>-<n>` — prompts for the site's PPPoE
   credentials, mints the ZeroTier identity etc. (`nix flake check` fails
   until this has run, because mesh-hosts reads the ZeroTier IP var.)
7. Boot the box from a NixOS installer USB on the local network, then:
   `clan machines install gw-<city>-<n> --target-host root@<lan-ip>`
8. Check `facter.json` for the real NIC names, fix `wan.interface` /
   `trunkPorts` if the enumeration differs, and
   `clan machines update gw-<city>-<n>` (rides the mesh from then on).
9. Add a row to the site table above and to the machines table in
   [Overview](./overview.md).

## Omada controller

Sites with TP-Link Omada APs/switches set `cnx.router.omada.enable = true`.
The controller runs as a podman container (`mbentley/omada-controller` — there
is no nixpkgs package) with host networking on the mgmt VLAN, where adoption
broadcasts (UDP 29810) live. UI: `https://<mgmt-addr>:8043` from mgmt, or the
gateway's mesh address from an admin machine. Controller state is under
`/var/lib/omada`, declared as clan state (`clan.core.state.omada`); wiring it
into the borgbackup instance is still a follow-up.

## Internal reverse proxy

`cnx.router.proxy.enable` puts Caddy on the gateway, terminating TLS for
`<service>.<site><n>.cnx.network` (e.g. `https://omada.cnx1.cnx.network`) and
forwarding to internal backends:

```nix
cnx.router.proxy = {
  enable = true;
  services.omada = {
    backend = "https://127.0.0.1:8043";
    insecureSkipVerify = true; # Omada's cert is self-signed
  };
};
```

- **Cert**: one real Let's Encrypt wildcard `*.<site><n>.cnx.network` per
  gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind
  PPPoE with zero inbound reachability, and browsers trust it without a CA
  install. Each gateway has its own TSIG key (`acme_gw_<city>_<n>`) that ns1
  scopes to `_acme-challenge.<site><n>` TXT records only.
- **Resolution**: the names exist only internally — Blocky answers
  `*.<site><n>.cnx.network` with the router's `lan` address; the public
  `cnx.network` zone never carries them.
- **Access**: `proxy.allowVlans` (default `mgmt` + `lan`) get 443 (and 80 for
  the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh.

## Runbook

- **PPPoE down**: `systemctl status pppd-wan`, `journalctl -u pppd-wan` on the
  gateway (over the mesh). ISP credentials live in the `pppoe-credentials`
  vars generator; re-enter with `clan vars generate gw-<city>-<n> --regenerate`.
- **A VLAN gets no leases**: `systemctl status kea-dhcp4-server`; check the
  port's `bridgeVLANs` tagging and that the switch trunk carries the VLAN.
- **DNS/blocklist issues**: Blocky metrics are in VictoriaMetrics (job
  `blocky`); `journalctl -u blocky` on the gateway.
- **Slow internet reported / `WANSpeedDegraded` firing**: check the
  `speedtest_download_mbps` / `speedtest_ping_ms` trend in VictoriaMetrics,
  then `librespeed-cli` and `mtr <isp-gw>` on the gateway to separate ISP
  degradation from local saturation (`iftop` on ppp0 shows who's eating it).
- **Banned yourself**: `cscli decisions list` / `cscli decisions delete --ip <ip>`
  on the gateway. The ZeroTier mesh `/88` is whitelisted at the parser stage
  (`cnx/mesh-whitelist`), so admin access over the mesh can never be banned.

## Follow-ups

- Borgbackup client for Omada + Kea lease state.
- Suricata (IDS-only) if CPU headroom allows — deliberately skipped for now.
- Remaining three sites.