Add gateway diagnostics: iperf3, CLI toolkit, periodic WAN speed test

iperf3 serves throughput tests from every VLAN and the mesh; an hourly
librespeed run feeds speedtest_* metrics through node_exporter's textfile
collector, and vmalert flags download rates below half the link's own
7-day median so ISP degradation surfaces without per-site thresholds.
This commit is contained in:
Berwn
2026-07-31 10:44:52 +07:00
parent b11ff75ca6
commit d8d4a686eb
5 changed files with 142 additions and 0 deletions
+6
View File
@@ -21,6 +21,8 @@ Naming: `gw-<city>-<n>`, e.g. `gw-cnx-1`.
| Bans | CrowdSec + nftables bouncer (sshd log parsing) |
| Omada | Optional per site: TP-Link Omada controller as a podman container |
| Proxy | Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1) |
| Diagnostics | iperf3 server on 5201, reachable from every VLAN and the mesh (`iperf3 -c <router-vlan-addr>`); CLI toolkit: tcpdump, mtr, ethtool, conntrack, kdig, iftop, librespeed-cli |
| Speed test | Hourly librespeed run (`speedtest.timer`) → `speedtest_*` metrics via node_exporter; vmalert flags download < 50% of the link's own 7-day median (`WANSpeedDegraded`) |
| Management | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet |
| Storage | Single-disk ZFS (zstd, `/var` its own dataset for snapshots) |
@@ -124,6 +126,10 @@ cnx.router.proxy = {
port's `bridgeVLANs` tagging and that the switch trunk carries the VLAN.
- **DNS/blocklist issues**: Blocky metrics are in VictoriaMetrics (job
`blocky`); `journalctl -u blocky` on the gateway.
- **Slow internet reported / `WANSpeedDegraded` firing**: check the
`speedtest_download_mbps` / `speedtest_ping_ms` trend in VictoriaMetrics,
then `librespeed-cli` and `mtr <isp-gw>` on the gateway to separate ISP
degradation from local saturation (`iftop` on ppp0 shows who's eating it).
- **Banned yourself**: `cscli decisions list` / `cscli decisions delete --ip <ip>`
on the gateway. The ZeroTier mesh `/88` is whitelisted at the parser stage
(`cnx/mesh-whitelist`), so admin access over the mesh can never be banned.