fix: ntfy listens on the Tailscale address; monitor publishes to it directly

The reverse proxy is a separate VPS on the tailnet, so a loopback-only
ntfy was unreachable from it. install_ntfy.sh now binds the host's Tailscale
IP (NTFY_LISTEN overrides). New NTFY_PUBLISH_URL: where the monitor POSTs,
separate from the public NTFY_SERVER subscribers see, so an alert never
waits on DNS or the proxy (first cycle logged 502s from Cloudflare while
the domain was not yet proxied).

.env.example

@@ -89,6 +89,10 @@ SMTP_PASSWORD=
 # <prefix>-<station>-danger, <prefix>-warning, <prefix>-danger, <prefix>-p1-outlook,
 # <prefix>-status. See docs/NOTIFICATIONS.md.
 NTFY_SERVER=
+# Where the monitor POSTs (defaults to NTFY_SERVER). Use the local ntfy
+# address (loopback or Tailscale IP) so publishing does not depend on
+# DNS / the reverse proxy being up.
+NTFY_PUBLISH_URL=
 NTFY_TOPIC_PREFIX=ping
 NTFY_TOKEN=
 PUBLIC_URL=https://water.buildfor.life/

docs/NOTIFICATIONS.md

@@ -60,12 +60,14 @@ cd /opt/thailand-water-monitor
 NTFY_DOMAIN=ntfy.buildfor.life bash scripts/install_ntfy.sh
 ```
 
-This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on
+This installs the ntfy .deb, writes `/etc/ntfy/server.yml` (listen on the
-`127.0.0.1:2586`, anonymous read, token-only write, 72 h message cache,
+host's Tailscale address, port 2586; anonymous read, token-only write, 72 h
-signup/login/metrics off, tight visitor limits), enables the systemd unit,
+message cache, signup/login/metrics off, tight visitor limits), enables the
+systemd unit,
 creates the `monitor` user with **write-only access to `ping-*`**, mints a
-token, and appends `NTFY_SERVER` / `NTFY_TOPIC_PREFIX` / `NTFY_TOKEN` to
+token, and appends `NTFY_SERVER` (public URL for subscribers),
-`.env` if they are not there yet. Then:
+`NTFY_PUBLISH_URL` (loopback, what the monitor POSTs to), `NTFY_TOPIC_PREFIX`
+and `NTFY_TOKEN` to `.env` if they are not there yet. Then:
 
 ```bash
 systemctl restart water-monitor
@@ -73,30 +75,31 @@ journalctl -u water-monitor -n 20 | grep ntfy      # "ntfy notifications: https:
 curl -s 'https://ntfy.buildfor.life/ping-status/json?poll=1'   # anonymous read works
 ```
 
-Put `https://ntfy.buildfor.life` in front of `127.0.0.1:2586` with whatever
+The reverse proxy is a separate VPS on the same tailnet, so ntfy listens on
-already terminates TLS for `water.buildfor.life`. Subscribers hold a
+the monitor host's Tailscale address and nothing is exposed on a public
-long-lived connection, so the proxy needs websockets on and no short read
+interface. On the Caddy machine:
-timeout:
 
 ```caddyfile
 ntfy.buildfor.life {
-    reverse_proxy 127.0.0.1:2586
+    reverse_proxy <monitor tailscale ip>:2586
 }
 ```
 
-Cloudflare tunnel: add a public hostname `ntfy.buildfor.life` →
+Caddy proxies websockets and keeps long-poll connections open by default;
-`http://127.0.0.1:2586`. Cloudflare proxies websockets by default; nothing
+subscribers hold one open. `behind-proxy: true` makes ntfy rate-limit on
-else to set.
+`X-Forwarded-For` rather than treating every subscriber as the proxy.
 
-Nothing about the message pipeline needs the domain to be public before you
+Publishing does not depend on the domain: `NTFY_PUBLISH_URL` points the
-test: with `NTFY_SERVER=http://127.0.0.1:2586` in `.env` the monitor
+monitor at the Tailscale address directly, so a DNS or proxy problem never
-publishes locally and `curl .../ping-status/json?poll=1` shows what went out.
+holds back an alert. Test the pipeline before the domain is live with
+`curl -s 'http://<tailscale ip>:2586/ping-status/json?poll=1'`.
 
 ## Configuration
 
 | Variable | Default | Meaning |
 |---|---|---|
-| `NTFY_SERVER` | *(empty = off)* | base URL of the ntfy server the monitor publishes to |
+| `NTFY_SERVER` | *(empty = off)* | public base URL subscribers use; shown on the dashboard |
+| `NTFY_PUBLISH_URL` | = `NTFY_SERVER` | where the monitor POSTs; the local ntfy address (`http://<tailscale ip>:2586`), so publishing never waits on DNS/proxy |
 | `NTFY_TOPIC_PREFIX` | `ping` | first segment of every topic |
 | `NTFY_TOKEN` | *(empty)* | bearer token if the server requires auth to publish (it does, see above) |
 | `PUBLIC_URL` | `https://water.buildfor.life/` | click-through target in messages |

scripts/install_ntfy.sh

@@ -7,22 +7,28 @@
 # What it does:
 #   - installs the ntfy .deb from the official GitHub release (single Go
 #     binary, ~30 MB RSS, sqlite message cache)
-#   - writes /etc/ntfy/server.yml: listens on 127.0.0.1:2586 only (put it
+#   - writes /etc/ntfy/server.yml: listens on the Tailscale address only
-#     behind your existing reverse proxy / Cloudflare tunnel), anonymous
+#     (the reverse proxy is another VPS on the tailnet; nothing is exposed
-#     READ on all topics, WRITE only with a token
+#     on a public interface), anonymous READ on all topics, WRITE only with
+#     a token. Override with NTFY_LISTEN=host:port.
 #   - creates the `monitor` publishing user + token, writes NTFY_SERVER /
 #     NTFY_TOKEN into /opt/thailand-water-monitor/.env if not present
 #
-# Reverse proxy: forward https://$NTFY_DOMAIN -> http://127.0.0.1:2586 with
+# Reverse proxy (on the Caddy VPS, over Tailscale):
-# websockets enabled and a long/no read timeout (subscribers hold the
+#     ntfy.buildfor.life {
-# connection open). Caddy: `reverse_proxy 127.0.0.1:2586`. Cloudflare
+#         reverse_proxy <this host's tailscale ip>:2586
-# tunnel: add a public hostname pointing at http://127.0.0.1:2586.
+#     }
+# Caddy passes websockets and keeps long-poll connections open by default;
+# subscribers hold one open. ntfy runs with behind-proxy: true so rate
+# limits key on X-Forwarded-For, not on the proxy's address.
 set -euo pipefail
 
 NTFY_DOMAIN="${NTFY_DOMAIN:?set NTFY_DOMAIN, e.g. ntfy.buildfor.life}"
 NTFY_VERSION="${NTFY_VERSION:-2.28.0}"
 MONITOR_DIR="${MONITOR_DIR:-/opt/thailand-water-monitor}"
-LISTEN="${NTFY_LISTEN:-127.0.0.1:2586}"
+TS_IP="$(tailscale ip -4 2>/dev/null | head -1 || true)"
+LISTEN="${NTFY_LISTEN:-${TS_IP:-127.0.0.1}:2586}"
+echo "ntfy will listen on ${LISTEN}"
 
 if ! command -v ntfy >/dev/null || [[ "$(ntfy --version 2>/dev/null | awk '{print $3}')" != "$NTFY_VERSION" ]]; then
     tmp=$(mktemp -d)
@@ -82,6 +88,7 @@ if [[ -f "$env_file" ]] && ! grep -q '^NTFY_SERVER=' "$env_file"; then
         echo ""
         echo "# ntfy public notifications (scripts/install_ntfy.sh)"
         echo "NTFY_SERVER=https://${NTFY_DOMAIN}"
+        echo "NTFY_PUBLISH_URL=http://${LISTEN}"
         echo "NTFY_TOPIC_PREFIX=ping"
         echo "NTFY_TOKEN=${token}"
     } >> "$env_file"

src/config.py

@@ -40,7 +40,12 @@ class Config:
     THAIWATER_API_KEY = os.getenv("THAIWATER_API_KEY")
 
     # Public flood notifications (ntfy). Off unless NTFY_SERVER is set.
+    # NTFY_SERVER is what subscribers use (public https URL, shown on the
+    # dashboard). NTFY_PUBLISH_URL is where the monitor POSTs; defaults to
+    # NTFY_SERVER, set it to http://127.0.0.1:2586 when ntfy runs on the same
+    # host so publishing never depends on DNS/proxy/tunnel being up.
     NTFY_SERVER = os.getenv("NTFY_SERVER", "").strip()
+    NTFY_PUBLISH_URL = os.getenv("NTFY_PUBLISH_URL", "").strip() or NTFY_SERVER
     NTFY_TOPIC_PREFIX = os.getenv("NTFY_TOPIC_PREFIX", "ping").strip()
     NTFY_TOKEN = os.getenv("NTFY_TOKEN", "").strip()  # publish token if ACL enabled
     PUBLIC_URL = os.getenv("PUBLIC_URL", "https://water.buildfor.life/").strip()

src/web_api.py

@@ -323,13 +323,14 @@ def _init_notifications():
                 "(a restart may re-send the current level)"
             )
         publisher = notify_mod.NtfyPublisher(
-            Config.NTFY_SERVER,
+            Config.NTFY_PUBLISH_URL,
             prefix=Config.NTFY_TOPIC_PREFIX,
             token=Config.NTFY_TOKEN or None,
             dashboard_url=Config.PUBLIC_URL,
         )
         logger.info(
-            f"ntfy notifications: {Config.NTFY_SERVER} topics {Config.NTFY_TOPIC_PREFIX}-*"
+            f"ntfy notifications: publish to {Config.NTFY_PUBLISH_URL}, "
+            f"subscribers use {Config.NTFY_SERVER}, topics {Config.NTFY_TOPIC_PREFIX}-*"
         )
         return publisher, state
     except Exception as e: