B4L/cnx-network-clan
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update runbook docs for web01 reverse proxy and per-host ACME keys

Browse Source 
Reflect web01 in the machines table and monitoring scrape list, note Grafana is
now also published publicly via web01's reverse proxy, add the CNX Uptime
dashboard, and document the dedicated acme_mx1/acme_web01 DNS-01 keys.
This commit is contained in:
Berwn
2026-06-21 03:17:51 +07:00
parent 415a050f6a
commit b8bea27a9c
3 changed files with 38 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/monitoring.md
+13 -7
View File
@@ -1,6 +1,7 @@
# Monitoring
Metrics and dashboards live on `control`, reachable only over the ZeroTier mesh.
Metrics and logs live on `control` over the ZeroTier mesh; the Grafana dashboards
are also published publicly through `web01` (see [Dashboards](#dashboards)).
## Collection
@@ -18,8 +19,8 @@ Metrics and dashboards live on `control`, reachable only over the ZeroTier mesh.
## Storage & scraping
**VictoriaMetrics** on `control`, bound to `127.0.0.1:8428`, 180-day retention
(`modules/monitoring/server.nix`). It scrapes `control` over loopback and `ns1`/
`ns2` over the mesh.
(`modules/monitoring/server.nix`). It scrapes `control` over loopback and
`ns1`/`ns2`/`mx1`/`web01` over the mesh.
> The scraper dials IPv4-only by default, so mesh (IPv6) targets need
> `extraOptions = [ "-enableTCP6" ]`. Without it, ns1/ns2 are dropped with
@@ -31,8 +32,10 @@ Metrics and dashboards live on `control`, reachable only over the ZeroTier mesh.
## Dashboards
**Grafana** on `control` (`:3000`), mesh-only, anonymous access disabled. The
admin password is a clan var:
**Grafana** on `control` (`:3000`), anonymous access disabled. Reachable directly
over the mesh, and publicly at `https://grafana.cnx.network` via `web01`'s reverse
proxy (TLS termination — see [Overview](./overview.md)). The admin password is a
clan var:
```
clan vars get control grafana-admin/password
@@ -46,6 +49,9 @@ there is picked up):
outside-in DNS probes.
- **CNX Backups** (`backups.json`) — borgbackup job health, time since the last
run, and per-job state. See [Backups](./backups.md).
- **CNX Uptime** (`uptime.json`) — per-host up/down status, current uptime,
availability over the selected window, and up/down history. Label-driven, so
every scraped host appears automatically.
## Logs
@@ -53,8 +59,8 @@ there is picked up):
(`modules/monitoring/server.nix`). All three hosts ship journald to it via
systemd's own `services.journald.upload` → the `/insert/journald` endpoint
(`modules/monitoring/exporters.nix`); no extra agent. `control` uploads over
loopback so its logs survive a mesh outage, `ns1`/`ns2` push over the mesh, and
9428 is firewall-scoped to the mesh like everything else.
loopback so its logs survive a mesh outage, the other hosts push over the mesh,
and 9428 is firewall-scoped to the mesh like everything else.
> Same IPv4-only default as the scraper: VictoriaLogs binds `0.0.0.0:9428` for a
> bare `:9428`, so mesh (IPv6) pushes from ns1/ns2 are refused until you pass