services/router: init
This commit is contained in:
+62
-39
@@ -3,28 +3,39 @@
|
||||
NixOS routers (OPNsense replacements) on Topton 1U fanless boxes — Intel N300,
|
||||
4x i226-V 2.5G ports (some units add 2x 10G SFP+), 16GB DDR5, 256GB mSATA.
|
||||
Four sites are planned; the first is `gw-cnx-1` (Chiang Mai). Everything lives
|
||||
in the reusable `modules/router/` module (`cnx.router.*` options); a site's
|
||||
machine config only sets port names, VLANs, and feature flags.
|
||||
in the `router` clan service (`modules/clan/router/`, registered as
|
||||
`modules.router` in `clan.nix`); a site joins by adding its machine to the
|
||||
`router` inventory instance in `inventory.nix` with settings for port names,
|
||||
VLANs, and feature flags (`roles.default.machines.<gw>.settings`).
|
||||
|
||||
Naming: `gw-<city>-<n>`, e.g. `gw-cnx-1`.
|
||||
|
||||
The service has a NixOS VM test (`modules/clan/router/tests/vm/`): an ISP
|
||||
node runs a PPPoE access concentrator, a `client` sits on the tagged lan/iot
|
||||
VLANs behind the trunk, an `admin` on the untagged mgmt access port, and a
|
||||
simulated radio (`mac80211_hwsim`) carries two SSIDs with a WPA3 station. It
|
||||
checks the PPPoE dial-in, leases/reservations, Blocky, NAT, `allowWan`, the
|
||||
mgmt-only trust model, SSH exposure and the Wi-Fi bridge ports. Run it with
|
||||
`nix build .#checks.x86_64-linux.router` (also part of `nix flake check`).
|
||||
|
||||
## What each gateway runs
|
||||
|
||||
| Function | Implementation |
|
||||
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| WAN | PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned |
|
||||
| LAN | VLAN-filtering bridge `br0` over the trunk ports (networkd); `accessPorts` pin a port untagged to one VLAN — convention: the last copper port is an untagged `mgmt` recovery port |
|
||||
| Firewall/NAT | nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT |
|
||||
| DHCP | Kea, one subnet per VLAN |
|
||||
| DNS | Blocky (blocklist resolver), metrics on :4000 scraped by control |
|
||||
| IPv6 | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC |
|
||||
| Bans | CrowdSec + nftables bouncer (sshd log parsing) |
|
||||
| Omada | Optional per site: TP-Link Omada controller as a podman container |
|
||||
| Proxy | Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1) |
|
||||
| Diagnostics | iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c <router-vlan-addr>`); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli |
|
||||
| Speed test | Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`) |
|
||||
| Management | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet |
|
||||
| Storage | Single-disk ZFS (zstd, `/var` its own dataset for snapshots) |
|
||||
| Function | Implementation |
|
||||
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| WAN | PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned |
|
||||
| LAN | VLAN-filtering bridge `br0` over the trunk ports (networkd); `accessPorts` pin a port untagged to one VLAN — convention: the last copper port is an untagged `mgmt` recovery port; `stagingPort` turns a spare port into a DHCP uplink into the old LAN while the box runs alongside the router it replaces (drop at cutover) |
|
||||
| Firewall/NAT | nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT |
|
||||
| DHCP | Kea, one subnet per VLAN |
|
||||
| DNS | Blocky (blocklist resolver), metrics on :4000 scraped by control |
|
||||
| IPv6 | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC |
|
||||
| Bans | CrowdSec + nftables bouncer (sshd log parsing) |
|
||||
| Omada | Optional per site: TP-Link Omada controller as a podman container |
|
||||
| Wi-Fi | Optional: hostapd on the router's radios; each SSID (`wifi.networks`) is an untagged access port of its VLAN, passphrases via vars prompts — see `modules/clan/router/README.md` |
|
||||
| Proxy | Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1) |
|
||||
| Diagnostics | iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c <router-vlan-addr>`); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli |
|
||||
| Speed test | Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`) |
|
||||
| Management | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet |
|
||||
| Storage | Single-disk ZFS (zstd, `/var` its own dataset for snapshots) |
|
||||
|
||||
## Addressing plan
|
||||
|
||||
@@ -55,22 +66,24 @@ Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet onl
|
||||
|
||||
## Adding a site
|
||||
|
||||
1. Copy `machines/gw-cnx-1/` to `machines/gw-<city>-<n>/`; set `site`,
|
||||
`siteId` (next free number), port names, VLANs, and `omada.enable`.
|
||||
Fill the real disk id in `disko.nix` (from the installer:
|
||||
`ls -l /dev/disk/by-id/`). Keep the `mgmt`/`lan` VLANs.
|
||||
2. Add the machine to `inventory.machines` in `clan.nix` and to the machine
|
||||
list in `modules/mesh-hosts.nix`. Do **not** add it to `modules/hosts.nix`
|
||||
(dynamic PPPoE IP; clan connects over the mesh).
|
||||
1. Copy `machines/gw-cnx-1/` to `machines/gw-<city>-<n>/` (disko, facter,
|
||||
hostId, timezone). Fill the real disk id in `disko.nix` (from the
|
||||
installer: `ls -l /dev/disk/by-id/`).
|
||||
2. Add the machine to `inventory.machines` in `clan.nix`, to the `router`
|
||||
instance in `inventory.nix` (`roles.default.machines.gw-<city>-<n>.settings`: `site`,
|
||||
`siteId` (next free number), port names, VLANs, `omada.enable`; keep the
|
||||
`mgmt`/`lan` VLANs), and to the machine list in `modules/mesh-hosts.nix`.
|
||||
Do **not** add it to `modules/hosts.nix` (dynamic PPPoE IP; clan connects
|
||||
over the mesh).
|
||||
3. Add its node_exporter and Blocky scrape targets in
|
||||
`modules/monitoring/server.nix`.
|
||||
4. If the site uses the internal proxy (`cnx.router.proxy.enable`), wire ns1:
|
||||
import `(import ../../modules/dns/acme-gw-secret.nix "gw-<city>-<n>")` in
|
||||
**both** the gateway's and ns1's configuration, and on ns1 add the
|
||||
`dns-acme-gw-<city>-<n>-knot` generator, its `keyFiles` entry, and an
|
||||
`acl_acme_gw_<city>_<n>` scoped to `_acme-challenge.<city><n>` on the
|
||||
`cnx.network` zone (copy the `gw-cnx-1` blocks). Then
|
||||
`clan vars generate ns1` and redeploy ns1.
|
||||
4. If the site uses the internal proxy (`proxy.enable` in its settings), wire
|
||||
ns1: import `(import ../../modules/clan/router/acme-secret.nix
|
||||
"gw-<city>-<n>")` in ns1's configuration (the gateway declares the same
|
||||
shared generator itself), and on ns1 add the `dns-acme-gw-<city>-<n>-knot`
|
||||
generator, its `keyFiles` entry, and an `acl_acme_gw_<city>_<n>` scoped to
|
||||
`_acme-challenge.<city><n>` on the `cnx.network` zone (copy the `gw-cnx-1`
|
||||
blocks). Then `clan vars generate ns1` and redeploy ns1.
|
||||
5. `git add` the new machine directory — flake evaluation only sees
|
||||
git-tracked files, so an untracked `machines/gw-…/` is silently ignored.
|
||||
6. `clan vars generate gw-<city>-<n>` — prompts for the site's PPPoE
|
||||
@@ -86,9 +99,10 @@ Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet onl
|
||||
|
||||
## Omada controller
|
||||
|
||||
Sites with TP-Link Omada APs/switches set `cnx.router.omada.enable = true`.
|
||||
The controller runs as a podman container (`mbentley/omada-controller` — there
|
||||
is no nixpkgs package) with host networking on the mgmt VLAN, where adoption
|
||||
Sites with TP-Link Omada APs/switches set `omada.enable = true` in their
|
||||
router settings. The controller runs as a podman container
|
||||
(`mbentley/omada-controller` — there is no nixpkgs package) with host
|
||||
networking on the mgmt VLAN, where adoption
|
||||
broadcasts (UDP 29810) live. UI: `https://<mgmt-addr>:8043` from mgmt, or the
|
||||
gateway's mesh address from an admin machine. Controller state is under
|
||||
`/var/lib/omada`, declared as clan state (`clan.core.state.omada`); wiring it
|
||||
@@ -96,12 +110,12 @@ into the borgbackup instance is still a follow-up.
|
||||
|
||||
## Internal reverse proxy
|
||||
|
||||
`cnx.router.proxy.enable` puts Caddy on the gateway, terminating TLS for
|
||||
`<service>.<site><n>.cnx.network` (e.g. `https://omada.cnx1.cnx.network`) and
|
||||
forwarding to internal backends:
|
||||
`proxy.enable` in a site's router settings puts Caddy on the gateway,
|
||||
terminating TLS for `<service>.<site><n>.cnx.network` (e.g.
|
||||
`https://omada.cnx1.cnx.network`) and forwarding to internal backends:
|
||||
|
||||
```nix
|
||||
cnx.router.proxy = {
|
||||
proxy = {
|
||||
enable = true;
|
||||
services.omada = {
|
||||
backend = "https://127.0.0.1:8043";
|
||||
@@ -114,13 +128,22 @@ cnx.router.proxy = {
|
||||
gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind
|
||||
PPPoE with zero inbound reachability, and browsers trust it without a CA
|
||||
install. Each gateway has its own TSIG key (`acme_gw_<city>_<n>`) that ns1
|
||||
scopes to `_acme-challenge.<site><n>` TXT records only.
|
||||
scopes to `_acme-challenge.<site><n>` TXT records only. The zone, ns1's
|
||||
address and the ACME contact are fleet-wide role settings in `inventory.nix`
|
||||
(`proxy.domain`, `proxy.acme.nameserver`, `proxy.acme.email`).
|
||||
- **Resolution**: the names exist only internally — Blocky answers
|
||||
`*.<site><n>.cnx.network` with the router's `lan` address; the public
|
||||
`cnx.network` zone never carries them.
|
||||
- **Access**: `proxy.allowVlans` (default `mgmt` + `lan`) get 443 (and 80 for
|
||||
the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh.
|
||||
|
||||
## Using the service from another clan
|
||||
|
||||
The service is exported as `clan.modules.router` of this flake; add this
|
||||
repo as a flake input and reference it with `module.input = "<input name>"`.
|
||||
`modules/clan/router/README.md` is the consumer-facing description (it is
|
||||
also the module's manifest readme, shown by `clan modules`).
|
||||
|
||||
## Runbook
|
||||
|
||||
- **PPPoE down**: `systemctl status pppd-wan`, `journalctl -u pppd-wan` on the
|
||||
|
||||
Reference in New Issue
Block a user