B4L/cnx-network-clan
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add internal reverse proxy for gateways (Caddy, wildcard via DNS-01)

Browse Source 
cnx.router.proxy serves <service>.<site><n>.cnx.network with a real Let's
Encrypt wildcard obtained via a gateway-scoped TSIG key against ns1; Blocky
resolves the names to the router's LAN address, so they exist only
internally. First user: Omada UI on gw-cnx-1 (omada.cnx1.cnx.network).
This commit is contained in:
Berwn
2026-07-31 10:09:08 +07:00
parent 4735968433
commit b11ff75ca6
6 changed files with 246 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/gateways.md
+40 -5
View File
@@ -20,6 +20,7 @@ Naming: `gw-<city>-<n>`, e.g. `gw-cnx-1`.
| IPv6 | DHCPv6-PD on ppp0, /64 per VLAN via SLAAC |
| Bans | CrowdSec + nftables bouncer (sshd log parsing) |
| Omada | Optional per site: TP-Link Omada controller as a podman container |
| Proxy | Optional: Caddy reverse proxy for internal services under `*.<site><n>.cnx.network` with a real Let's Encrypt wildcard (DNS-01 against ns1) |
| Management | ZeroTier mesh: SSH, node_exporter, journald upload — like the fleet |
| Storage | Single-disk ZFS (zstd, `/var` its own dataset for snapshots) |
@@ -57,17 +58,24 @@ Trust model: mgmt → everything; other VLANs → router DNS/DHCP + internet onl
(dynamic PPPoE IP; clan connects over the mesh).
3. Add its node_exporter and Blocky scrape targets in
`modules/monitoring/server.nix`.
4. `git add` the new machine directory — flake evaluation only sees
4. If the site uses the internal proxy (`cnx.router.proxy.enable`), wire ns1:
import `(import ../../modules/dns/acme-gw-secret.nix "gw-<city>-<n>")` in
**both** the gateway's and ns1's configuration, and on ns1 add the
`dns-acme-gw-<city>-<n>-knot` generator, its `keyFiles` entry, and an
`acl_acme_gw_<city>_<n>` scoped to `_acme-challenge.<city><n>` on the
`cnx.network` zone (copy the `gw-cnx-1` blocks). Then
`clan vars generate ns1` and redeploy ns1.
5. `git add` the new machine directory — flake evaluation only sees
git-tracked files, so an untracked `machines/gw-…/` is silently ignored.
5. `clan vars generate gw-<city>-<n>` — prompts for the site's PPPoE
6. `clan vars generate gw-<city>-<n>` — prompts for the site's PPPoE
credentials, mints the ZeroTier identity etc. (`nix flake check` fails
until this has run, because mesh-hosts reads the ZeroTier IP var.)
6. Boot the box from a NixOS installer USB on the local network, then:
7. Boot the box from a NixOS installer USB on the local network, then:
`clan machines install gw-<city>-<n> --target-host root@<lan-ip>`
7. Check `facter.json` for the real NIC names, fix `wan.interface` /
8. Check `facter.json` for the real NIC names, fix `wan.interface` /
`trunkPorts` if the enumeration differs, and
`clan machines update gw-<city>-<n>` (rides the mesh from then on).
8. Add a row to the site table above and to the machines table in
9. Add a row to the site table above and to the machines table in
[Overview](./overview.md).
## Omada controller
@@ -80,6 +88,33 @@ gateway's mesh address from an admin machine. Controller state is under
`/var/lib/omada`, declared as clan state (`clan.core.state.omada`); wiring it
into the borgbackup instance is still a follow-up.
## Internal reverse proxy
`cnx.router.proxy.enable` puts Caddy on the gateway, terminating TLS for
`<service>.<site><n>.cnx.network` (e.g. `https://omada.cnx1.cnx.network`) and
forwarding to internal backends:
```nix
cnx.router.proxy = {
enable = true;
services.omada = {
backend = "https://127.0.0.1:8043";
insecureSkipVerify = true; # Omada's cert is self-signed
};
};
```
- **Cert**: one real Let's Encrypt wildcard `*.<site><n>.cnx.network` per
gateway, issued via ACME DNS-01 (lego/RFC2136) against ns1 — works behind
PPPoE with zero inbound reachability, and browsers trust it without a CA
install. Each gateway has its own TSIG key (`acme_gw_<city>_<n>`) that ns1
scopes to `_acme-challenge.<site><n>` TXT records only.
- **Resolution**: the names exist only internally — Blocky answers
`*.<site><n>.cnx.network` with the router's `lan` address; the public
`cnx.network` zone never carries them.
- **Access**: `proxy.allowVlans` (default `mgmt` + `lan`) get 443 (and 80 for
the HTTP→HTTPS redirect). Not exposed to WAN, guest VLANs, or the mesh.
## Runbook
- **PPPoE down**: `systemctl status pppd-wan`, `journalctl -u pppd-wan` on the
machines/gw-cnx-1/configuration.nix
+12
View File
@@ -6,6 +6,7 @@
imports = [
../../modules/router
../../modules/monitoring/exporters.nix
(import ../../modules/dns/acme-gw-secret.nix "gw-cnx-1")
];
clan.core.sops.defaultGroups = [ "admins" ];
@@ -36,6 +37,17 @@
};
# This site runs the Omada controller for its APs/switches.
omada.enable = true;
# Internal reverse proxy: real wildcard cert *.cnx1.cnx.network; Blocky
# resolves the names to the router's LAN address for mgmt+lan clients.
proxy = {
enable = true;
services.omada = {
# Omada's UI is HTTPS with a self-signed cert on the host network.
backend = "https://127.0.0.1:8043";
insecureSkipVerify = true;
};
};
};
time.timeZone = "Etc/GMT-7"; # UTC+7 (Thailand, fixed offset, no DST)
machines/ns1/configuration.nix
+35 -1
View File
@@ -14,6 +14,7 @@ in
../../modules/dns/authoritative.nix
../../modules/dns/acme-mx1-secret.nix
../../modules/dns/acme-web01-secret.nix
(import ../../modules/dns/acme-gw-secret.nix "gw-cnx-1")
../../modules/static-ipv6.nix
../../modules/monitoring/exporters.nix
];
@@ -93,10 +94,29 @@ in
'';
};
# ACME DNS-01, dedicated gateway keys. Same pattern as web01: each gateway
# holds its own TSIG key (acme_gw_<site>_<n>), rendered from the shared
# secret generator imported above. acl_acme_gw_<site>_<n> scopes it to TXT
# updates at _acme-challenge.<site><n> on cnx.network — the owner its internal
# wildcard *.<site><n>.cnx.network challenge uses — and nothing else.
clan.core.vars.generators.dns-acme-gw-cnx-1-knot = {
files."acme.conf" = {
secret = true;
owner = "knot";
group = "knot";
};
dependencies = [ "dns-acme-gw-cnx-1-secret" ];
script = ''
printf 'key:\n - id: acme_gw_cnx_1\n algorithm: hmac-sha256\n secret: %s\n' \
"$(cat "$in"/dns-acme-gw-cnx-1-secret/secret)" > "$out"/acme.conf
'';
};
services.knot.keyFiles = [
config.clan.core.vars.generators.dns-acme-tsig.files."acme.conf".path
config.clan.core.vars.generators.dns-acme-mx1-knot.files."acme.conf".path
config.clan.core.vars.generators.dns-acme-web01-knot.files."acme.conf".path
config.clan.core.vars.generators.dns-acme-gw-cnx-1-knot.files."acme.conf".path
];
services.knot.settings.acl = [
@@ -133,6 +153,17 @@ in
# i.e. _acme-challenge at the cnx.network apex (where this acl is attached).
"update-owner-name" = [ "_acme-challenge" ];
}
{
id = "acl_acme_gw_cnx_1";
key = "acme_gw_cnx_1";
action = [ "update" ];
"update-type" = [ "TXT" ];
"update-owner" = "name";
"update-owner-match" = "sub-or-equal";
# The internal wildcard *.cnx1.cnx.network places its challenge at
# _acme-challenge.cnx1.cnx.network.
"update-owner-name" = [ "_acme-challenge.cnx1" ];
}
];
# Automatic DNSSEC signing policy (primary only). ECDSA P-256/SHA-256 with
@@ -168,6 +199,9 @@ in
"acl_acme"
]
++ lib.optionals (d == "cnx.email") [ "acl_acme_mx1" ]
++ lib.optionals (d == "cnx.network") [ "acl_acme_web01" ];
++ lib.optionals (d == "cnx.network") [
"acl_acme_web01"
"acl_acme_gw_cnx_1"
];
}) domains;
}
modules/dns/acme-gw-secret.nix
+17
View File
@@ -0,0 +1,17 @@
# Shared TSIG secret for a gateway's dedicated ACME key (function: machine
# name -> module). The acme_gw_<x> key lets that gateway — and only it — write
# _acme-challenge.<label> TXT records on ns1 to obtain its internal wildcard
# cert via DNS-01. ns1 scopes it with a matching acl on the cnx.network zone.
# Import on BOTH ns1 and the gateway machine, applied with the machine name:
# (import ../../modules/dns/acme-gw-secret.nix "gw-cnx-1")
machine:
{ pkgs, ... }:
{
clan.core.vars.generators."dns-acme-${machine}-secret" = {
share = true;
files."secret".secret = true;
runtimeInputs = [ pkgs.openssl ];
# 32 random bytes, base64 — a valid hmac-sha256 TSIG secret.
script = ''openssl rand -base64 32 | tr -d '\n' > "$out"/secret'';
};
}
modules/router/default.nix
+1
View File
@@ -84,6 +84,7 @@ in
./dns-dhcp.nix
./crowdsec.nix
./omada.nix
./proxy.nix
];
options.cnx.router = {
modules/router/proxy.nix
+141
View File
@@ -0,0 +1,141 @@
# Internal reverse proxy for the gateway: Caddy terminates TLS for
# <service>.<site><siteId>.cnx.network (e.g. omada.cnx1.cnx.network) and
# forwards to backends by their internal address. The cert is a real Let's
# Encrypt wildcard (*.<site><siteId>.cnx.network) obtained via ACME DNS-01
# against ns1 with a gateway-scoped TSIG key, so browsers trust it without
# any CA install; the names only *resolve* internally — Blocky answers them
# with the router's LAN address, the public zone never carries them.
#
# Requires the machine to also import the shared secret generator:
# (import ../../modules/dns/acme-gw-secret.nix "<hostname>")
# and ns1 to hold the matching key + acl (see machines/ns1/configuration.nix).
{
config,
lib,
...
}:
let
cfg = config.cnx.router;
hosts = import ../hosts.nix;
hostname = config.networking.hostName;
tsigKey = "acme_${lib.replaceStrings [ "-" ] [ "_" ] hostname}";
certName = "${cfg.site}${toString cfg.siteId}.cnx.network";
serviceModule = {
options = {
backend = lib.mkOption {
type = lib.types.str;
example = "https://127.0.0.1:8043";
description = "URL Caddy forwards to (internal/mesh address).";
};
insecureSkipVerify = lib.mkOption {
type = lib.types.bool;
default = false;
description = "Skip TLS verification towards the backend (self-signed upstreams like Omada).";
};
};
};
in
{
options.cnx.router.proxy = {
enable = lib.mkEnableOption "internal reverse proxy (Caddy, wildcard cert via DNS-01)";
services = lib.mkOption {
type = lib.types.attrsOf (lib.types.submodule serviceModule);
default = { };
description = "Proxied services; attr name becomes <name>.${certName}.";
};
allowVlans = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [
"mgmt"
"lan"
];
description = "VLANs whose clients may reach the proxy (443, plus 80 for the redirect).";
};
};
config = lib.mkIf (cfg.enable && cfg.proxy.enable) {
assertions = [
{
assertion = lib.all (v: cfg.vlans ? ${v}) cfg.proxy.allowVlans;
message = "cnx.router.proxy.allowVlans must name VLANs defined in cnx.router.vlans.";
}
];
# Render the shared per-gateway TSIG secret into a lego rfc2136 env file;
# same pattern as web01 (modules/web-proxy.nix), scoped on ns1 to
# _acme-challenge.<site><siteId> TXT records only.
clan.core.vars.generators."dns-acme-${hostname}-rfc2136" = {
files."rfc2136.env".secret = true; # root-owned; systemd reads it as root
dependencies = [ "dns-acme-${hostname}-secret" ];
script = ''
printf 'RFC2136_NAMESERVER=${hosts.ns1.ipv4}:53\nRFC2136_TSIG_ALGORITHM=hmac-sha256.\nRFC2136_TSIG_KEY=${tsigKey}\nRFC2136_TSIG_SECRET=%s\n' \
"$(cat "$in"/dns-acme-${hostname}-secret/secret)" > "$out"/rfc2136.env
'';
};
security.acme = {
acceptTerms = true;
defaults.email = "postmaster@cnx.email";
# One wildcard for every proxied service; DNS-01 against ns1, so issuance
# works behind PPPoE with no inbound reachability at all.
certs.${certName} = {
domain = "*.${certName}";
dnsProvider = "rfc2136";
environmentFile =
config.clan.core.vars.generators."dns-acme-${hostname}-rfc2136".files."rfc2136.env".path;
# ns1 is the only nameserver that accepts this key's UPDATE; check
# propagation against it directly rather than a public resolver.
dnsResolver = "${hosts.ns1.ipv4}:53";
# Caddy reads the cert from explicit file paths (tls directive below),
# so it won't notice a renewal on its own.
reloadServices = [ "caddy.service" ];
};
};
# The lego-issued cert is owned group=acme; Caddy needs to read the key.
users.users.caddy.extraGroups = [ "acme" ];
# The explicit `tls cert key` points Caddy at the wildcard cert and disables
# its automatic ACME, so no extra issuance happens.
services.caddy = {
enable = true;
virtualHosts = lib.mapAttrs' (
name: svc:
lib.nameValuePair "${name}.${certName}" {
extraConfig = ''
tls /var/lib/acme/${certName}/cert.pem /var/lib/acme/${certName}/key.pem
${
if svc.insecureSkipVerify then
''
reverse_proxy ${svc.backend} {
transport http {
tls_insecure_skip_verify
}
}''
else
"reverse_proxy ${svc.backend}"
}
'';
}
) cfg.proxy.services;
};
# Blocky answers <anything>.<site><siteId>.cnx.network (customDNS covers
# subdomains) with the router's LAN address — clients on any allowed VLAN
# reach that address through the router's input path.
services.blocky.settings.customDNS.mapping.${certName} = cfg.vlans.lan.address;
# 443 serves the proxy; 80 only carries Caddy's automatic HTTP->HTTPS
# redirect. mgmt is already a trusted interface; listed anyway so shrinking
# trustedInterfaces later doesn't silently break the proxy.
networking.firewall.interfaces = lib.genAttrs (map (v: "vlan-${v}") cfg.proxy.allowVlans) (_: {
allowedTCPPorts = [
80
443
];
});
};
}