Add VictoriaLogs for centralized journald across all hosts

control runs VictoriaLogs (:9428, 30d, mesh-scoped) with a matching
Grafana datasource. Each host ships journald via systemd's own
journald.upload to the /insert/journald endpoint -- no extra agent.
control uploads over loopback so its logs survive a mesh outage; ns1
and ns2 push over the mesh.
This commit is contained in:
Berwn
2026-06-17 16:53:52 +07:00
parent c7b0f206c8
commit d4a171640b
4 changed files with 72 additions and 7 deletions
+20 -1
View File
@@ -42,6 +42,25 @@ Dashboards are provisioned from `modules/monitoring/dashboards/` (any JSON file
there is picked up):
- **CNX DNS** (`dns.json`) — firing alerts, per-nameserver SOA serials, zone
expiry countdowns, query/response rates, and host CPU/memory/disk/load.
expiry countdowns, query/response rates, host CPU/memory/disk/load, and the
outside-in DNS probes.
- **CNX Backups** (`backups.json`) — borgbackup job health, time since the last
run, and per-job state. See [Backups](./backups.md).
## Logs
**VictoriaLogs** on `control` (`:9428`), 30-day retention
(`modules/monitoring/server.nix`). All three hosts ship journald to it via
systemd's own `services.journald.upload` → the `/insert/journald` endpoint
(`modules/monitoring/exporters.nix`); no extra agent. `control` uploads over
loopback so its logs survive a mesh outage, `ns1`/`ns2` push over the mesh, and
9428 is firewall-scoped to the mesh like everything else.
Query logs from Grafana via the provisioned **VictoriaLogs** datasource (Explore
view, LogsQL), or directly in the built-in UI at `http://[control]:9428/select/vmui`.
Logs are tagged with `_HOSTNAME` and `_SYSTEMD_UNIT`, so to follow one service
across hosts:
```
_SYSTEMD_UNIT:"knot.service"
```