B4L/cnx-network-clan
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add site gateway role (modules/router) and gw-cnx-1

Browse Source 
Reusable cnx.router.* module for the Topton 1U boxes replacing OPNsense:
PPPoE WAN (optionally VLAN-tagged, AIS: 10, secret credentials incl.
username), VLAN-filtering bridge, nftables NAT/firewall with MSS clamp,
Kea DHCP with per-VLAN lease time, Blocky DNS, DHCPv6-PD, CrowdSec with
the ZeroTier mesh whitelisted, optional Omada controller, ZFS disk.

Fleet baseline rides along: admins sops group is now derived for every
machine in clan.nix (secrets encrypt to it from the first vars generate)
and time sync is chrony everywhere instead of systemd-timesyncd.
This commit is contained in:
Berwn
2026-07-28 17:06:07 +07:00
parent a87b579eb7
commit 158252323f
20 changed files with 857 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/SUMMARY.md
+2
View File
@@ -3,5 +3,7 @@
- [Overview](./overview.md)
- [ZeroTier mesh](./mesh.md)
- [DNS](./dns.md)
- [Mail](./mail.md)
- [Site gateways](./gateways.md)
- [Monitoring](./monitoring.md)
- [Backups](./backups.md)
docs/src/gateways.md
+100
View File
@@ -0,0 +1,100 @@
# Site gateways
NixOS routers (OPNsense replacements) on Topton 1U fanless boxes — Intel N300,
4x i226-V 2.5G ports (some units add 2x 10G SFP+), 16GB DDR5, 256GB mSATA.
Four sites are planned; the first is `gw-cnx-1` (Chiang Mai). Everything lives
in the reusable `modules/router/` module (`cnx.router.*` options); a site's
machine config only sets port names, VLANs, and feature flags.
Naming: `gw-<city>-<n>`, e.g. `gw-cnx-1`.
## What each gateway runs
| Function | Implementation |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| WAN | PPPoE (`pppd`), per-site ISP credentials via clan vars prompts; `wan.vlanId` when the ISP tags the session (AIS: 10); `wan.macAddress` to clone the old router's MAC if the ISP has it pinned |
| LAN | VLAN-filtering bridge `br0` over the trunk ports (networkd) |
| Firewall/NAT | nftables: default-deny WAN, no inter-VLAN, MSS clamp, v4 NAT |
| DHCP | Kea, one subnet per VLAN |
| DNS | Blocky (blocklist resolver), metrics on :4000 scraped by control |
| IPv6 | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC |
| Bans | CrowdSec + nftables bouncer (sshd log parsing) |
| Omada | Optional per site: TP-Link Omada controller as a podman container |
| Management | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet |
| Storage | Single-disk ZFS (zstd, `/var` its own dataset for snapshots) |
## Addressing plan
Each site owns `10.<siteId>.0.0/16`. Defaults per VLAN: subnet
`10.<siteId>.<vlanId>.0/24`, router at `.1`, DHCP pool `.100–.199`
(`.2–.99` static/infra, `.200–.254` reserved).
VLAN id convention (fleet-wide): **10 = mgmt**, **20 = lan** (both mandatory at
every site), 30 = guest, 40 = iot (reserved). Sites add their own beyond that.
Larger subnets (e.g. public-WiFi guest networks that outgrow a /24) are carved
from the **upper half** `10.<siteId>.128.0/17` and set explicitly on the VLAN.
The lower half stays reserved for /24s indexed by VLAN id. High-churn VLANs
should also shorten `dhcp.leaseTime` (default 86400 s) so the pool recycles.
First user: `gw-cnx-2` (site 2) runs the public WiFi — guest VLAN 30 at
`10.2.128.0/22`, pool `10.2.128.100 – 10.2.131.250`, `dhcp.leaseTime = 3600`.
| Site | siteId | mgmt | lan |
| ---- | ------ | -------------- | -------------- |
| cnx | 1 | `10.1.10.0/24` | `10.1.20.0/24` |
Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet only
(no inter-VLAN); WAN → nothing inbound; mesh → SSH, metrics, Omada UI.
## Adding a site
1. Copy `machines/gw-cnx-1/` to `machines/gw-<city>-<n>/`; set `site`,
`siteId` (next free number), port names, VLANs, and `omada.enable`.
Fill the real disk id in `disko.nix` (from the installer:
`ls -l /dev/disk/by-id/`). Keep the `mgmt`/`lan` VLANs.
2. Add the machine to `inventory.machines` in `clan.nix` and to the machine
list in `modules/mesh-hosts.nix`. Do **not** add it to `modules/hosts.nix`
(dynamic PPPoE IP; clan connects over the mesh).
3. Add its node_exporter and Blocky scrape targets in
`modules/monitoring/server.nix`.
4. `git add` the new machine directory — flake evaluation only sees
git-tracked files, so an untracked `machines/gw-…/` is silently ignored.
5. `clan vars generate gw-<city>-<n>` — prompts for the site's PPPoE
credentials, mints the ZeroTier identity etc. (`nix flake check` fails
until this has run, because mesh-hosts reads the ZeroTier IP var.)
6. Boot the box from a NixOS installer USB on the local network, then:
`clan machines install gw-<city>-<n> --target-host root@<lan-ip>`
7. Check `facter.json` for the real NIC names, fix `wan.interface` /
`trunkPorts` if the enumeration differs, and
`clan machines update gw-<city>-<n>` (rides the mesh from then on).
8. Add a row to the site table above and to the machines table in
[Overview](./overview.md).
## Omada controller
Sites with TP-Link Omada APs/switches set `cnx.router.omada.enable = true`.
The controller runs as a podman container (`mbentley/omada-controller` — there
is no nixpkgs package) with host networking on the mgmt VLAN, where adoption
broadcasts (UDP 29810) live. UI: `https://<mgmt-addr>:8043` from mgmt, or the
gateway's mesh address from an admin machine. Controller state is under
`/var/lib/omada`, declared as clan state (`clan.core.state.omada`); wiring it
into the borgbackup instance is still a follow-up.
## Runbook
- **PPPoE down**: `systemctl status pppd-wan`, `journalctl -u pppd-wan` on the
gateway (over the mesh). ISP credentials live in the `pppoe-credentials`
vars generator; re-enter with `clan vars generate gw-<city>-<n> --regenerate`.
- **A VLAN gets no leases**: `systemctl status kea-dhcp4-server`; check the
port's `bridgeVLANs` tagging and that the switch trunk carries the VLAN.
- **DNS/blocklist issues**: Blocky metrics are in VictoriaMetrics (job
`blocky`); `journalctl -u blocky` on the gateway.
- **Banned yourself**: `cscli decisions list` / `cscli decisions delete --ip <ip>`
on the gateway. The ZeroTier mesh `/88` is whitelisted at the parser stage
(`cnx/mesh-whitelist`), so admin access over the mesh can never be banned.
## Follow-ups
- Borgbackup client for Omada + Kea lease state.
- Suricata (IDS-only) if CPU headroom allows — deliberately skipped for now.
- Remaining three sites.
docs/src/overview.md
+16 -7
View File
@@ -6,13 +6,14 @@ this book is built from `docs/` and served on `control` over the ZeroTier mesh.
## Machines
| Machine | Role | Public IPv4 | Public IPv6 |
| --------- | -------------------------------------- | ---------------- | ----------------------- |
| `control` | ZeroTier controller, monitoring, docs | `77.42.68.181` | `2a01:4f9:c013:e6d0::1` |
| `ns1` | Knot DNS **primary** (master) | `46.224.170.206` | `2a01:4f8:c014:b5c5::1` |
| `ns2` | Knot DNS **secondary** (slave) | `157.180.70.82` | `2a01:4f9:c014:6d87::1` |
| `mx1` | Mail server (**MX** for cnx.email) | `5.223.65.38` | `2a01:4ff:2f0:1963::1` |
| `web01` | Public reverse proxy (TLS termination) | `5.223.55.246` | `2a01:4ff:2f0:2d8f::1` |
| Machine | Role | Public IPv4 | Public IPv6 |
| ---------- | -------------------------------------- | ---------------- | ----------------------- |
| `control` | ZeroTier controller, monitoring, docs | `77.42.68.181` | `2a01:4f9:c013:e6d0::1` |
| `ns1` | Knot DNS **primary** (master) | `46.224.170.206` | `2a01:4f8:c014:b5c5::1` |
| `ns2` | Knot DNS **secondary** (slave) | `157.180.70.82` | `2a01:4f9:c014:6d87::1` |
| `mx1` | Mail server (**MX** for cnx.email) | `5.223.65.38` | `2a01:4ff:2f0:1963::1` |
| `web01` | Public reverse proxy (TLS termination) | `5.223.55.246` | `2a01:4ff:2f0:2d8f::1` |
| `gw-cnx-1` | Site gateway Chiang Mai (router) | dynamic (PPPoE) | — |
## Access
@@ -21,6 +22,14 @@ this book is built from `docs/` and served on `control` over the ZeroTier mesh.
- clan reaches machines by their public IPs first (the `internet` instance), with
the mesh and Tor as automatic fallbacks.
## Host baseline
Every machine sets a fixed-offset `time.timeZone` and runs **chrony** for time
sync (`services.chrony.enable`). chrony is the single NTP implementation across
the fleet — do **not** use `systemd-timesyncd`; the two are mutually exclusive
and we standardise on chrony everywhere. New hosts must set both the timezone and
chrony as part of their baseline config.
## Editing these docs
Commit-to-edit: change the markdown under `docs/src/`, commit, and redeploy