mob next [ci-skip] [ci skip] [skip ci]
lastFile:docs/src/gateways.md
This commit is contained in:
+45
-24
@@ -3,11 +3,20 @@
|
||||
NixOS routers (OPNsense replacements) on Topton 1U fanless boxes — Intel N300,
|
||||
4x i226-V 2.5G ports (some units add 2x 10G SFP+), 16GB DDR5, 256GB mSATA.
|
||||
Four sites are planned; the first is `gw-cnx-1` (Chiang Mai). Everything lives
|
||||
in the reusable `modules/router/` module (`cnx.router.*` options); a site's
|
||||
machine config only sets port names, VLANs, and feature flags.
|
||||
in the `router` clan service (`modules/clan/router/`, registered as
|
||||
`modules.router` in `clan.nix`); a site joins by adding its machine to the
|
||||
`router` inventory instance with settings for port names, VLANs, and feature
|
||||
flags (`inventory.instances.router.roles.default.machines.<gw>.settings`).
|
||||
|
||||
Naming: `gw-<city>-<n>`, e.g. `gw-cnx-1`.
|
||||
|
||||
The service has a NixOS VM test (`modules/clan/router/tests/vm/`): an ISP
|
||||
node runs a PPPoE access concentrator, a `client` sits on the tagged lan/iot
|
||||
VLANs behind the trunk, an `admin` on the untagged mgmt access port. It checks
|
||||
the PPPoE dial-in, leases/reservations, Blocky, NAT, `allowWan`, the mgmt-only
|
||||
trust model and SSH exposure. Run it with `nix build .#checks.x86_64-linux.router`
|
||||
(also part of `nix flake check`).
|
||||
|
||||
## What each gateway runs
|
||||
|
||||
| Function | Implementation |
|
||||
@@ -55,22 +64,24 @@ Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet onl
|
||||
|
||||
## Adding a site
|
||||
|
||||
1. Copy `machines/gw-cnx-1/` to `machines/gw-<city>-<n>/`; set `site`,
|
||||
`siteId` (next free number), port names, VLANs, and `omada.enable`.
|
||||
Fill the real disk id in `disko.nix` (from the installer:
|
||||
`ls -l /dev/disk/by-id/`). Keep the `mgmt`/`lan` VLANs.
|
||||
2. Add the machine to `inventory.machines` in `clan.nix` and to the machine
|
||||
list in `modules/mesh-hosts.nix`. Do **not** add it to `modules/hosts.nix`
|
||||
(dynamic PPPoE IP; clan connects over the mesh).
|
||||
1. Copy `machines/gw-cnx-1/` to `machines/gw-<city>-<n>/` (disko, facter,
|
||||
hostId, timezone). Fill the real disk id in `disko.nix` (from the
|
||||
installer: `ls -l /dev/disk/by-id/`).
|
||||
2. Add the machine to `inventory.machines` in `clan.nix`, to the `router`
|
||||
instance (`roles.default.machines.gw-<city>-<n>.settings`: `site`,
|
||||
`siteId` (next free number), port names, VLANs, `omada.enable`; keep the
|
||||
`mgmt`/`lan` VLANs), and to the machine list in `modules/mesh-hosts.nix`.
|
||||
Do **not** add it to `modules/hosts.nix` (dynamic PPPoE IP; clan connects
|
||||
over the mesh).
|
||||
3. Add its node_exporter and Blocky scrape targets in
|
||||
`modules/monitoring/server.nix`.
|
||||
4. If the site uses the internal proxy (`cnx.router.proxy.enable`), wire ns1:
|
||||
import `(import ../../modules/dns/acme-gw-secret.nix "gw-<city>-<n>")` in
|
||||
**both** the gateway's and ns1's configuration, and on ns1 add the
|
||||
`dns-acme-gw-<city>-<n>-knot` generator, its `keyFiles` entry, and an
|
||||
`acl_acme_gw_<city>_<n>` scoped to `_acme-challenge.<city><n>` on the
|
||||
`cnx.network` zone (copy the `gw-cnx-1` blocks). Then
|
||||
`clan vars generate ns1` and redeploy ns1.
|
||||
4. If the site uses the internal proxy (`proxy.enable` in its settings), wire
|
||||
ns1: import `(import ../../modules/clan/router/acme-secret.nix
|
||||
"gw-<city>-<n>")` in ns1's configuration (the gateway declares the same
|
||||
shared generator itself), and on ns1 add the `dns-acme-gw-<city>-<n>-knot`
|
||||
generator, its `keyFiles` entry, and an `acl_acme_gw_<city>_<n>` scoped to
|
||||
`_acme-challenge.<city><n>` on the `cnx.network` zone (copy the `gw-cnx-1`
|
||||
blocks). Then `clan vars generate ns1` and redeploy ns1.
|
||||
5. `git add` the new machine directory — flake evaluation only sees
|
||||
git-tracked files, so an untracked `machines/gw-…/` is silently ignored.
|
||||
6. `clan vars generate gw-<city>-<n>` — prompts for the site's PPPoE
|
||||
@@ -86,9 +97,10 @@ Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet onl
|
||||
|
||||
## Omada controller
|
||||
|
||||
Sites with TP-Link Omada APs/switches set `cnx.router.omada.enable = true`.
|
||||
The controller runs as a podman container (`mbentley/omada-controller` — there
|
||||
is no nixpkgs package) with host networking on the mgmt VLAN, where adoption
|
||||
Sites with TP-Link Omada APs/switches set `omada.enable = true` in their
|
||||
router settings. The controller runs as a podman container
|
||||
(`mbentley/omada-controller` — there is no nixpkgs package) with host
|
||||
networking on the mgmt VLAN, where adoption
|
||||
broadcasts (UDP 29810) live. UI: `https://<mgmt-addr>:8043` from mgmt, or the
|
||||
gateway's mesh address from an admin machine. Controller state is under
|
||||
`/var/lib/omada`, declared as clan state (`clan.core.state.omada`); wiring it
|
||||
@@ -96,12 +108,12 @@ into the borgbackup instance is still a follow-up.
|
||||
|
||||
## Internal reverse proxy
|
||||
|
||||
`cnx.router.proxy.enable` puts Caddy on the gateway, terminating TLS for
|
||||
`<service>.<site><n>.cnx.network` (e.g. `https://omada.cnx1.cnx.network`) and
|
||||
forwarding to internal backends:
|
||||
`proxy.enable` in a site's router settings puts Caddy on the gateway,
|
||||
terminating TLS for `<service>.<site><n>.cnx.network` (e.g.
|
||||
`https://omada.cnx1.cnx.network`) and forwarding to internal backends:
|
||||
|
||||
```nix
|
||||
cnx.router.proxy = {
|
||||
proxy = {
|
||||
enable = true;
|
||||
services.omada = {
|
||||
backend = "https://127.0.0.1:8043";
|
||||
@@ -114,13 +126,22 @@ cnx.router.proxy = {
|
||||
gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind
|
||||
PPPoE with zero inbound reachability, and browsers trust it without a CA
|
||||
install. Each gateway has its own TSIG key (`acme_gw_<city>_<n>`) that ns1
|
||||
scopes to `_acme-challenge.<site><n>` TXT records only.
|
||||
scopes to `_acme-challenge.<site><n>` TXT records only. The zone, ns1's
|
||||
address and the ACME contact are fleet-wide role settings in `clan.nix`
|
||||
(`proxy.domain`, `proxy.acme.nameserver`, `proxy.acme.email`).
|
||||
- **Resolution**: the names exist only internally — Blocky answers
|
||||
`*.<site><n>.cnx.network` with the router's `lan` address; the public
|
||||
`cnx.network` zone never carries them.
|
||||
- **Access**: `proxy.allowVlans` (default `mgmt` + `lan`) get 443 (and 80 for
|
||||
the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh.
|
||||
|
||||
## Using the service from another clan
|
||||
|
||||
The service is exported as `clan.modules.router` of this flake; add this
|
||||
repo as a flake input and reference it with `module.input = "<input name>"`.
|
||||
`modules/clan/router/README.md` is the consumer-facing description (it is
|
||||
also the module's manifest readme, shown by `clan modules`).
|
||||
|
||||
## Runbook
|
||||
|
||||
- **PPPoE down**: `systemctl status pppd-wan`, `journalctl -u pppd-wan` on the
|
||||
|
||||
Reference in New Issue
Block a user