From ffcce42a51d636fbd023c566c6c1e48d7f30d594 Mon Sep 17 00:00:00 2001 From: grabowski Date: Mon, 21 Sep 2026 21:41:49 +0200 Subject: [PATCH] Digital factory plan: build on the existing cnx-network clan Rewrite docs/digital-factory.md against the fleet that already exists at git.b4l.co.th/B4L/cnx-network-clan instead of a plant-only clan: - The plant is one more site: gw-lpn-1 from the cnx.router module, plant machines in the same inventory, reached over the existing ZeroTier mesh (no VPS, no moon, no port forwards). - Fleet pieces reused as-is: control for host metrics, journals and host alerts; ns1/ns2 for the gateway wildcard cert; web01 as the public proxy for the customer portal; clan borgbackup with a plant server. - Process time series move from TimescaleDB to a plant-local VictoriaMetrics (same modules and rules as control); PostgreSQL keeps everything with identity. Object storage dropped for ZFS datasets. - VLAN table on the fleet addressing convention (siteId 3 assumed, ids above 40); OT segments as dual-homed islands behind the collectors; stations reach the app only through the gateway Caddy proxy. - New section listing every file that changes in the clan repo, the modules/vinyl/* to write, and three router-module extensions (NTP serving, static VLANs, second WAN). - Gaps the plant closes for the fleet: Alertmanager delivery, off-site backup destination, TSDB backup. - Repository layout decided: infra in cnx-network-clan, app in cnx-factory as a flake input, this repo stays documents. README bullet updated to match. --- README.md | 11 +- docs/digital-factory.md | 678 ++++++++++++++++++++++++---------------- 2 files changed, 413 insertions(+), 276 deletions(-) diff --git a/README.md b/README.md index dff1a5b..13a1c3b 100644 --- a/README.md +++ b/README.md @@ -88,11 +88,12 @@ The detail lives in dedicated documents: shipment, and the requirements this sets for the factory software - [Digital factory systems plan](docs/digital-factory.md): every system that has to exist for environment, machine, QC and genealogy data to land - in one database keyed to one record identity: the NixOS machine fleet - managed as one Clan, networks and VLAN zoning, Telegraf/MQTT acquisition - source by source, QC instrument integration, the factory application, - alerting, backup, security, the data clauses for every RFQ, build order - and rough cost + in one database keyed to one record identity: the plant as one more site + in B4L's existing NixOS fleet (the cnx-network clan: mesh, DNS, fleet + monitoring, site gateway module, public proxy), what changes in that + repo, VLAN zoning, Telegraf/MQTT acquisition source by source, QC + instrument integration, the factory application, alerting, backup, + security, the data clauses for every RFQ, build order and rough cost - [Quality automation](docs/quality-automation.md): stamper metrology (Keyence-class confocal, profiler, XRF), layered pressed-record QC (100% optical inspection, per-cycle press telemetry, sampled automated audio diff --git a/docs/digital-factory.md b/docs/digital-factory.md index b31c892..4c86dbb 100644 --- a/docs/digital-factory.md +++ b/docs/digital-factory.md @@ -9,44 +9,84 @@ genealogy all land in one place, keyed to one record identity, and can be shown to a customer. It is a planning position as of September 2026; every cost figure is an estimate until a quote replaces it. -**Standing decision: every service runs on NixOS, and the whole fleet is -declared and deployed as one [Clan](https://clan.lol).** Servers, edge -collectors, station kiosks, the router and the off-site VPS are all NixOS -machines in a single inventory. Vendor instrument PCs (Keyence, vision -controllers, PLC HMIs) are appliances, not servers: they export into the -NixOS estate and never host a service of ours. +**Standing decision: every service runs on NixOS, and the plant is one more +site in the existing B4L fleet, the +[cnx-network clan](https://git.b4l.co.th/B4L/cnx-network-clan).** That +clan already runs the ZeroTier mesh, authoritative DNS, mail, fleet +monitoring, backups, a public reverse proxy and the first site gateway, all +declared in one repository and deployed with the clan CLI. The plant adds +a gateway, a core server, a backup server, edge collectors and station +kiosks to that inventory and reuses everything the fleet already provides. +Nothing is built twice. Vendor instrument PCs (Keyence, vision controllers, +PLC HMIs) are appliances, not servers: they export into the NixOS estate +and never host a service of ours. + +## What the fleet already provides + +Read the clan's own runbook (`docs/` in the repo, served on `control`) for +detail; this is what the plant plan leans on. + +| Fleet piece | Where | What the plant gets from it | +|---|---|---| +| Clan repo, clan-core 26.05, sops vars gated by YubiKey, `admins` group applied to every machine automatically | git.b4l.co.th/B4L/cnx-network-clan | one inventory, one deploy path (`clan machines install` / `update`), secrets minted with `clan vars generate` | +| ZeroTier mesh, controller on `control`, IPv6 ULA `/88`, `modules/mesh-hosts.nix` reads every peer's address from vars | `control` | admin SSH, deploys and scrapes for plant machines behind PPPoE NAT, exactly as `gw-cnx-1` is reached today; no port forwards, no VPS needed | +| `modules/router/`: `cnx.router.*` on Topton 1U boxes: PPPoE WAN, VLAN-filtering bridge, nftables with no inter-VLAN traffic, Kea, Blocky, CrowdSec, Omada, iperf3, hourly speed test, Caddy internal proxy with a real wildcard `*..cnx.network` via DNS-01 | site gateways | the plant router is a copy of `machines/gw-cnx-1/` with its own VLANs; the internal proxy gives every station a browser-trusted `https://factory.1.cnx.network` | +| Addressing convention `10...0/24`, VLAN ids 10 mgmt and 20 lan mandatory, 30 guest, 40 iot reserved | fleet-wide | the plant takes the next free `siteId` and adds its own VLANs above 40 | +| VictoriaMetrics (180 d), VictoriaLogs (30 d), vmalert, Grafana, blackbox probes | `control` | node_exporter on every plant machine scraped over the mesh, journald shipped to VictoriaLogs, host alerts from the existing rules; process data stays on site (below) | +| Knot DNS (`cnx.network`, `buildfor.life`, `cnx.email`) with per-consumer TSIG keys for ACME | `ns1`, `ns2` | the plant's public names and the gateway's wildcard cert; a brand domain for the customer portal is one zone file away | +| `web01`: public Caddy with `*.cnx.network`, backends dialed over the mesh | Hetzner | the customer portal is one `virtualHosts` entry pointing at the plant core's mesh address, the same way `grafana.cnx.network` works now | +| clan `borgbackup` service, `control` as server, repokey encryption, backups over the mesh | `control` | the pattern; the plant needs its own server because plant data is far larger than the DNSSEC keystore | +| chrony everywhere, fixed-offset time zones (`Etc/GMT-7` for Thailand) | fleet baseline | plant machines inherit; edge collectors additionally serve NTP to the PLCs | +| mdbook runbook in `docs/src/`, built and served on `control` | `control` | the plant's operational runbook goes there as a chapter; this document stays the planning source | + +Known gaps in the fleet that the plant needs closed (all on the clan's own +TODO list or small extensions to existing modules): + +- **Alert delivery**: vmalert currently uses `notifier.blackhole`. A boiler + alarm needs a phone. The plant brings an Alertmanager with Telegram or + LINE and SMS receivers; `control`'s vmalert can point at it too. +- **Off-site backup destination**: `control`'s disk is not sized for plant + images and audio. Add a Hetzner Storage Box or a storage VPS as a second + `borgbackup` destination (the clan service supports several per client). +- **Router module**: three small extensions, listed under Networks. ## Design decisions -1. **One clan, one repo, rebuildable from git.** Machine configuration, - service placement, mesh networking, backups, host monitoring and - secrets (clan vars, sops-encrypted) live in one flake. A failed machine - is replaced with `clan machines install ` from a fresh disk; a - configuration change is `clan machines update`. Nothing is configured by - hand on a running box. -2. **One relational database for everything that has identity.** - PostgreSQL with TimescaleDB holds the genealogy, orders, gates, QC - results, per-cycle press data, environment history and energy history. - Joining "this record" to "the mold temperature on that cycle" and to - "the humidity in the press hall that hour" is a query, not an export. - Host and infrastructure metrics (CPU, disk, systemd, journals) go to - the separate Prometheus/Loki pair that the clan monitoring service - provides; they are not process data. +1. **One clan, one repo, rebuildable from git.** Plant machines are entries + in `clan.nix`'s `fleet` set with a `machines//` directory each + (`configuration.nix`, `disko.nix`, `facter.json`), like every machine + already there. A failed box is replaced from a fresh disk with + `clan machines install`; a change is `clan machines update`. Nothing is + configured by hand on a running machine. +2. **Two stores, by shape of the data.** Everything with an identity + (orders, gates, genealogy, lots, press runs, per-cycle rows, QC + verdicts, events) lives in PostgreSQL on the plant core. Everything + that is a time series (1 Hz machine tags, zone environment, energy, + water) lives in a plant-local VictoriaMetrics, the same engine and + modules the fleet already runs on `control`, so dashboards, PromQL and + vmalert rules carry over unchanged. Per-cycle rows carry a snapshot of + the zone environment and the key press values, so the relational join + "this record, that cycle, those conditions" never needs the time-series + store; the curve behind the snapshot is one PromQL range query away. + (TimescaleDB in PostgreSQL is the fallback if that split proves + awkward; it is not the starting point.) 3. **MQTT is the plant bus, Telegraf is the universal collector.** Every - machine, meter and sensor is read by a Telegraf instance on an edge - collector near it (Modbus, OPC UA, Siemens S7 and MQTT inputs are - stock plugins) and published to the broker on the core server. The - core side persists everything to TimescaleDB with the same tool. Custom - code exists only where no plugin does (vision results, instrument CSV - ingestion, the audio QC pipeline). + machine, meter and sensor is read by a Telegraf instance on the edge + collector nearest it (Modbus, OPC UA, Siemens S7 and MQTT inputs are + stock plugins) and published to Mosquitto on the core. On the core, + Telegraf writes the stream to VictoriaMetrics (InfluxDB line protocol, + native to VM) and the factory application subscribes to the event + topics (cycle complete, alarm, instrument result) and writes rows. + Custom code exists only where no plugin does (vision results, + instrument CSV ingestion, the audio QC pipeline). 4. **Read-only toward machines.** The data platform never writes to a PLC or an instrument. Control, safety and interlocks (fume extraction, steam shutoff, boiler safety chain) stay in the vendor PLCs and hard wiring, as [facility security](facility-security.md) already requires. 5. **Join keys are counters, not clocks.** Every press cycle carries the machine's own cycle counter; every instrument result carries the part - serial it was read from. Wall-clock time is recorded (UTC, NTP from the - router) but is never the thing two systems are joined on. + serial it was read from. Wall-clock time is recorded (UTC, chrony from + the fleet baseline) but is never what two systems are joined on. 6. **Append-only, hash-chained.** Process events, gate approvals and QC verdicts are written once and never updated; corrections are new events that reference the old. Each event carries the hash of the previous one @@ -55,55 +95,65 @@ NixOS estate and never host a service of ours. 7. **The customer portal is a view, not a copy.** It reads the same database through the same application with a customer role; there is no exported report to keep in sync. +8. **The plant runs without the internet.** Everything that pressing, + QC and traceability need is on site. A WAN outage stops only the + customer portal, off-site backup, fleet monitoring and remote support, + and all four catch up when it returns. ## The systems on one page | System | What it does | Runs on | Phase | |---|---|---|---| -| Network and firewall | VLAN zoning (OT, sensors, servers, stations, office, CCTV, guest), dual WAN, NTP, DHCP/DNS | `router-1` (NixOS, nftables) plus managed switches | Build | -| Clan fleet | Inventory of every NixOS machine, ZeroTier mesh, SSH and user management, secrets, backups, host monitoring | git repo, applied to all machines | Now (dev), Build (plant) | -| Plant data platform | PostgreSQL + TimescaleDB, MQTT broker, Telegraf persister, Grafana, S3-compatible object storage | `core-1` | Build | -| Factory application | Orders, gates, genealogy, materials, press runs, QC verdicts, packaging, fulfillment, customer portal, audit log | `core-1` | Build (v1), Launch (v2) | -| Edge collectors | Telegraf per machine zone, local store-and-forward, vision result receiver | `edge-press`, `edge-utility`, `edge-galvanics` | Build, Expand | -| Environmental monitoring | Zone temperature, humidity, PM, differential pressure; alarms; correlation with production | sensors on RS485 loops into `edge-utility` and `edge-press` | Build | +| Site gateway | PPPoE WAN, VLAN zoning, DHCP/DNS, NTP for the site, internal TLS proxy, CrowdSec | `gw--1` (existing `cnx.router` module) | Build | +| Plant core | PostgreSQL, Mosquitto, Telegraf persister, VictoriaMetrics + vmalert + Grafana, Alertmanager, factory application, file datasets on ZFS | `vinyl-core` | Build | +| Factory application | Orders, gates, genealogy, materials, press runs, QC verdicts, packaging, fulfillment, customer portal, audit log | `vinyl-core` | Build (v1), Launch (v2) | +| Edge collectors | Telegraf per machine zone, disk-backed store-and-forward, NTP to the OT segment, vision result receiver | `vinyl-edge-press`, `vinyl-edge-util`, `vinyl-edge-galv` | Build, Expand | +| Environmental monitoring | Zone temperature, humidity, PM, differential pressure; alarms; correlation with production | RS485 loops into the edge collectors | Build | | Machine telemetry | Presses, extruders, hydraulics, boiler, chiller, compressor, heat pump, HVAC, PV, BESS, energy and water meters | edge collectors | Build | -| QC instrument integration | Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans | stations plus `edge-press` | Launch, Expand | -| Station kiosks | Badge check-in and scanning at each manual step; kiosk browser on the factory application | `st-*` machines | Build | -| Observability and alerting | Process alarms (Grafana on TimescaleDB), host alarms (clan monitoring), routing to phones, data-silence alarms | `core-1` | Build | -| Backup and DR | ZFS snapshots, Borg to `backup-1` and off-site, PostgreSQL point-in-time recovery, restore drills | `backup-1`, `vps-1` | Build | -| Identity | OIDC single sign-on for staff, badge IDs for operators at stations, role model for customers | `core-1` (Pocket ID) | Build | -| Client audio vault | Encrypted, isolated storage for pre-release audio with checksums and access logging | `core-1` (separate dataset) | Build | -| Site systems integration | Access-control door events, fire panel and BESS detection contacts, UPS state, CCTV | `edge-utility`, optional `nvr-1` | Build | -| Off-site node | ZeroTier moon with a public IP, off-site backup target, public reverse proxy for the portal | `vps-1` | Build | +| QC instrument integration | Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans | stations plus `vinyl-edge-press` | Launch, Expand | +| Station kiosks | Badge check-in and scanning at each manual step; kiosk browser on the factory application | `vinyl-st-*` | Build | +| Process alerting | Environment, utilities, reject rate, stamper life, PV/BESS, fire and BESS detection contacts, UPS, data silence; delivered to phones | `vinyl-core` (vmalert + Alertmanager) | Build | +| Fleet monitoring | Host metrics, journals, host alerts for every plant machine | `control` (existing) | Build | +| Backup and DR | ZFS snapshots and replication, Borg to the plant backup server and off site, PostgreSQL point-in-time recovery, restore drills | `vinyl-backup`, off-site destination | Build | +| Identity | Staff sign-on (OIDC), badge IDs for operators at stations, customer role | `vinyl-core` | Build | +| Client audio vault | Encrypted, isolated dataset for pre-release audio with checksums and access logging | `vinyl-core` | Build | +| Site systems integration | Access-control door events, fire panel and BESS detection contacts, UPS state | `vinyl-edge-util` | Build | +| Customer portal, public | TLS termination and forwarding over the mesh to the plant core | `web01` (existing) | Launch | +| CCTV | Recording on its own VLAN | vendor NVR appliance, or `vinyl-nvr` (Frigate) if wanted | Build | ## Architecture ```mermaid flowchart LR - subgraph L0[Field] + subgraph L0[Field, isolated OT segments] PLC[Press and extruder PLCs
OPC UA / S7] UT[Boiler, chiller, compressor,
heat pump, HVAC
Modbus / BACnet] EN[Energy meters, Deye inverters,
BESS, water meters
Modbus / pulse] ENV[Zone T / RH / PM / dP sensors
Modbus RTU] GV[Rectifiers, bath sensors
Modbus] - VIS[Vision controller
TCP results, FTP images] + VIS[Vision controller
TCP results, SFTP images] INST[Keyence, XRF, lab instruments
CSV export] end - subgraph L1[Edge, NixOS] - EP[edge-press
Telegraf + vision receiver] - EU[edge-utility
Telegraf] - EG[edge-galvanics
Telegraf] - ST[st-* stations
kiosk, scanners, ingestion] + subgraph L1[Edge, NixOS, dual-homed] + EP[vinyl-edge-press
Telegraf + vision receiver] + EU[vinyl-edge-util
Telegraf + dry-contact I/O] + EG[vinyl-edge-galv
Telegraf] + ST[vinyl-st-* stations
kiosk, scanners, ingestion] end - subgraph L2[Core, NixOS] - MQ[MQTT broker] - DB[(PostgreSQL + TimescaleDB)] - OBJ[(Object storage)] + subgraph L2[Plant core, NixOS, servers VLAN] + MQ[Mosquitto] + VM[(VictoriaMetrics
vmalert, Grafana)] + DB[(PostgreSQL)] APP[Factory application] - GF[Grafana + alerting] + AM[Alertmanager] end - subgraph L3[Outside] - VPS[vps-1: moon, backup, proxy] + subgraph GW[gw-site-1] + PX[Caddy internal proxy
factory.site1.cnx.network] + end + subgraph L3[Fleet, Hetzner, over the mesh] + CTL[control: node metrics,
journals, host alerts] + WEB[web01: public proxy] + BK[off-site borg destination] CUST[Customer portal] PH[Phones: alerts] end @@ -117,146 +167,208 @@ flowchart LR EP --> MQ EU --> MQ EG --> MQ - ST --> APP - MQ --> DB + MQ --> VM MQ --> APP APP --> DB - APP --> OBJ - EP --> OBJ - DB --> GF - GF --> PH - APP --> VPS --> CUST + ST --> PX --> APP + VM --> AM --> PH + APP -.mesh.-> WEB --> CUST + L2 -.mesh.-> CTL + L2 -.mesh.-> BK ``` -Three rules keep this simple: field devices talk only to their edge -collector (they are on VLANs with no route anywhere else); edge collectors -talk only to the broker and object storage; the application and Grafana -talk only to the database, the broker and object storage. The VPS reaches -the core over the ZeroTier mesh, never over an open port. +Three rules keep this simple: field devices sit on isolated OT segments +whose only other member is their edge collector; edge collectors publish to +the core and nothing else; stations and office clients reach the +application only through the gateway's proxy. Everything that leaves the +site rides the ZeroTier mesh, never an open port. ## Machines -All NixOS, all in the clan inventory. Hardware classes are planning -positions; models are chosen at purchase. +All NixOS, all entries in the clan inventory. Names follow the fleet +convention (`gw--` for gateways, short role names for servers) +with a `vinyl-` prefix so the plant's machines are recognisable in a +multi-site inventory. `` is the city code chosen at site selection; +Mueang Chi is in Lamphun province, so `lpn` is the working assumption and +the gateway is `gw-lpn-1`, giving internal names under `*.lpn1.cnx.network`. +Hardware classes are planning positions; models are chosen at purchase. -| Machine | Role | Hardware class | Where | Clan tags | -|---|---|---|---|---| -| `router-1` | Firewall, VLANs, DHCP/DNS, NTP (chrony), dual WAN failover | fanless x86, 4+ NICs | IT room | `core` | -| `core-1` | PostgreSQL + TimescaleDB, Mosquitto, Telegraf persister, factory app, Grafana, Garage object storage, Pocket ID, Caddy | 1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD pool for objects, dual PSU on UPS | IT room | `core`, `db` | -| `backup-1` | Borg server, PostgreSQL streaming replica, second Garage node, ZFS receive target | tower or 2U, large HDD pool | separate room from `core-1` (archive side of the building) | `core`, `backup` | -| `vps-1` | ZeroTier moon (public IP), off-site Borg destination, third Garage node, reverse proxy for the customer portal | small cloud VPS with block storage, outside Thailand or in Bangkok, decision open | cloud | `offsite` | -| `edge-press` | Telegraf (OPC UA / S7 to the presses and extruders, Modbus to hydraulics and label oven), vision result receiver, disk-backed buffer | fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485 | press hall control cabinet | `edge` | -| `edge-utility` | Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS, water; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS | same class, more RS485 | electrical room or utility yard cabinet | `edge` | -| `edge-galvanics` | Telegraf (rectifiers, bath sensors, extraction status) | same class, corrosion-protected enclosure outside the wet area | galvanics service corridor | `edge` (Expand phase) | -| `st-intake` | Office workstation for audio intake and gates (plain browser; not a kiosk) | any NixOS desktop | office | `station` | -| `st-cutting` | Session log entry, lacquer registration, scribe ID | mini PC, touch screen, badge reader | cutting room | `station` | -| `st-press-1`, `st-press-2` | Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons | mini PC, touch screen, badge reader, 2D scanner | at each press | `station` | -| `st-lab` | PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment | mini PC, RS232/USB to instruments | material lab | `station` | -| `st-metrology` | SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan | mini PC | metrology bench (Expand phase) | `station` | -| `st-audioqc` | Needle-drop capture and analysis pipeline | mini PC with audio interface, turntable, phono stage | listening/QC room (Expand phase) | `station` | -| `st-pack-1` | Runout code camera read, jacket barcode scan, BOM check, barcode verifier | mini PC, camera, scanner, verifier | packaging | `station` | -| `st-ship` | Consignment assembly, carrier label and tracking capture | mini PC, label printer | shipping | `station` | -| `nvr-1` | Frigate CCTV recording on its own VLAN (optional; a vendor NVR appliance is the alternative) | small server with GPU or Coral accelerator | IT room | `cctv` | +| Machine | Role | Hardware class | Where | +|---|---|---|---| +| `gw-lpn-1` | Site gateway: `cnx.router` with the plant VLANs, internal proxy for the factory application and Grafana, NTP for the site | Topton 1U N300, 4x 2.5G (fleet standard); a 10G SFP+ variant if the core server link warrants it | IT room | +| `vinyl-core` | PostgreSQL, Mosquitto, Telegraf persister, VictoriaMetrics + vmalert + Grafana, Alertmanager, factory application, ZFS datasets for files (audio captures, inspection images, metrology raw data, CoA scans, client audio) | 1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD mirror or raidz for files, dual PSU on UPS | IT room | +| `vinyl-backup` | Borg server for the plant clients, ZFS receive target for `vinyl-core`'s datasets, PostgreSQL streaming replica, pgBackRest repository | tower or 2U, large HDD pool | separate room from `vinyl-core` (archive side of the building) | +| `vinyl-edge-press` | Telegraf (OPC UA / S7 to presses and extruders, Modbus to hydraulics and label oven), vision result receiver, SFTP target for inspection images, chrony serving the OT segment, disk-backed MQTT buffer | fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485 | press hall control cabinet | +| `vinyl-edge-util` | Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS | same class, more RS485 | electrical room or utility yard cabinet | +| `vinyl-edge-galv` | Telegraf (rectifiers, bath sensors, extraction status) | same class, corrosion-protected enclosure outside the wet area | galvanics service corridor (Expand phase) | +| `vinyl-st-cutting` | Session log entry, lacquer registration, scribe ID | mini PC, touch screen, badge reader | cutting room | +| `vinyl-st-press1`, `vinyl-st-press2` | Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons | mini PC, touch screen, badge reader, 2D scanner | at each press | +| `vinyl-st-lab` | PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment | mini PC, RS232/USB to instruments | material lab | +| `vinyl-st-metrology` | SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan | mini PC | metrology bench (Expand phase) | +| `vinyl-st-audioqc` | Needle-drop capture and analysis pipeline | mini PC with audio interface, turntable, phono stage | listening/QC room (Expand phase) | +| `vinyl-st-pack1` | Runout code camera read, jacket barcode scan, BOM check, barcode verifier | mini PC, camera, scanner, verifier | packaging | +| `vinyl-st-ship` | Consignment assembly, carrier label and tracking capture | mini PC, label printer | shipping | +| `vinyl-nvr` | Frigate CCTV recording on the cctv VLAN (optional; a vendor NVR appliance is the alternative) | small server with GPU or Coral accelerator | IT room | -Stations run a kiosk browser (`cage` plus Firefox or Chromium) pointed at -the factory application; badge readers and scanners are USB HID keyboard -devices, so there is no driver work. Fifteen machines at launch, about -twenty at maturity; every one is a few lines in the inventory. +Office audio intake and gate approvals happen from ordinary staff laptops +on the `lan` VLAN through the same proxy; no dedicated machine. Stations +run a kiosk browser (`cage` plus Firefox or Chromium) pointed at the +application; badge readers and scanners are USB HID keyboard devices, so +there is no driver work. Fifteen plant machines at launch, about twenty at +maturity; every one is a directory and a line in the inventory. ## Networks -| VLAN | Members | Rule | -|---|---|---| -| `ot-press` | press and extruder PLCs and HMIs, vision controller, `edge-press` | no internet, no route to other VLANs; `edge-press` is the only gateway | -| `ot-utility` | boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, `edge-utility` | same | -| `ot-galvanics` | rectifiers, bath transmitters, `edge-galvanics` | same | -| `sensors` | RS485 loops terminate on the edge collectors, so zone sensors have no IP presence; any IP sensor that is added lives here | no internet | -| `servers` | `core-1`, `backup-1`, `router-1` management | reachable from `stations`, `office`, edge collectors and the mesh | -| `stations` | `st-*` kiosks, instrument PCs (Keyence, vision setup laptop) | reach `core-1` only | -| `office` | staff laptops, printers | internet, `core-1` | -| `cctv` | cameras, `nvr-1` | no internet; NVR reachable from `office` by role | -| `guest` | visitors, vendor laptops | internet only | -| `mgmt` | switch and IPMI management interfaces | `router-1` and `core-1` admins only | +The plant is site `` (next free number in the fleet's site table; +`cnx` is 1 and the public-WiFi site is 2, so 3 is the working assumption) +and owns `10..0.0/16`. VLANs follow the fleet convention; the +ids above 40 are proposals. -Vendor remote support (Newbilt and Pheenix both offer remote monitoring) goes -through a jump account on `router-1` over the ZeroTier mesh, time-boxed and -logged, with a port forward to the one PLC HMI they need. No machine ever -gets a vendor's own cellular router or cloud agent on the OT VLAN. +| VLAN | id | Members | `allowWan` | Rule | +|---|---|---|---|---| +| `mgmt` | 10 | switches, APs, Omada, IPMI, admin laptops on site | yes | fleet convention: trusted, reaches everything | +| `lan` | 20 | office laptops, printers | yes | proxy to factory app and Grafana, internet | +| `guest` | 30 | visitors, vendor laptops | yes | internet only | +| `servers` | 100 | `vinyl-core`, `vinyl-backup`, the servers-side NIC of every edge collector | yes (mesh, nix, journald, off-site backup) | no inbound from other VLANs; the gateway's own Caddy dials the app here | +| `stations` | 110 | `vinyl-st-*`, instrument PCs (Keyence, vision setup laptop), label printer | yes (mesh, deploys, journald) | proxy to the factory app only; `dhcp.reservations` for every station | +| `cctv` | 120 | cameras, NVR | no | reachable from `mgmt`; NVR UI via the proxy if wanted | +| `ot-press` | 210 | press and extruder PLCs and HMIs, vision controller, OT NIC of `vinyl-edge-press` | no | isolated segment, see below | +| `ot-util` | 220 | boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, OT NIC of `vinyl-edge-util` | no | isolated segment | +| `ot-galv` | 230 | rectifiers, bath transmitters, OT NIC of `vinyl-edge-galv` | no | isolated segment | -Dual WAN: fibre plus a 4G/5G modem on `router-1`, failover only. The plant -does not depend on the internet to run: everything in the tables above is -on site except `vps-1`, and the only things that stop when the internet -drops are the customer portal and off-site backup. +**OT segments are islands, not routes.** Each edge collector is dual-homed: +one NIC on `servers` (an access port on the cabinet switch), one NIC on +its OT segment. The OT segment lives on the cabinet's industrial switch; +if it must span cabinets it rides the building trunk as a VLAN with +`dhcp.enable = false` and `allowWan = false`, and the router's default +no-inter-VLAN policy keeps it closed. Either way the gateway forwards +nothing into or out of an OT segment; the collector is the only bridge, +and it only publishes. Zone sensors on RS485 loops terminate directly on +the collectors and have no IP presence at all. Cheap ESP32/MQTT sensor +nodes are acceptable for extra density in non-critical zones (on +`stations`, publishing to the core), never for alarmed zones. -## Clan inventory (sketch) +**Stations never route to `servers`.** They reach +`https://factory.lpn1.cnx.network`, which Blocky resolves to the gateway +and Caddy forwards to `vinyl-core` on the servers VLAN, with the real +wildcard certificate the router module already issues. `proxy.allowVlans` +gains `stations`. The same proxy serves `grafana.lpn1.cnx.network` to the +office. This keeps the fleet's trust model intact: no VLAN talks to +another VLAN; the router talks to its backends. -The plant clan in outline. Option names follow the clan release current at -writing (26.05); check them against the pinned release before use, and -expect the monitoring service to change, since it is still marked -experimental upstream. +Router module extensions the plant needs (small, and useful for the other +sites too): -```nix -{ - meta.name = "cnx-vinyl"; - meta.domain = "cnx"; # mesh-only TLD; hosts resolve as .cnx +1. **NTP served by the gateway** to every VLAN (chrony `allow` for the + site's `/16`, UDP 123 in the per-VLAN firewall set) so stations and + the NVR sync locally; today the gateway is only a chrony client. +2. **Static-only VLANs**: a `dhcp.enable = false` VLAN should still + accept `reservations` semantics for documentation, or simply be + documented as address-by-hand; either is fine, decide when writing it. +3. **Second WAN**: a 4G/5G modem as failover for the PPPoE session + (`wan.backupInterface` with a metric and a health check). Not needed + for launch because the plant runs without the internet; needed before + the customer portal is promised as always-on. - inventory.machines = { - router-1 = { tags = [ "core" ]; }; - core-1 = { tags = [ "core" "db" "monitoring-server" ]; }; - backup-1 = { tags = [ "core" "backup" ]; }; - vps-1 = { tags = [ "offsite" ]; }; - edge-press = { tags = [ "edge" ]; }; - edge-utility = { tags = [ "edge" ]; }; - edge-galvanics = { tags = [ "edge" ]; }; - st-cutting = { tags = [ "station" ]; }; - st-press-1 = { tags = [ "station" ]; }; - st-lab = { tags = [ "station" ]; }; - st-pack-1 = { tags = [ "station" ]; }; - st-ship = { tags = [ "station" ]; }; - }; +Vendor remote support (Newbilt and Pheenix both offer remote monitoring) +goes through an admin's SSH port-forward across the mesh into the relevant +edge collector and from there to the one PLC HMI they need, time-boxed and +logged. No machine ever gets a vendor's own cellular router or cloud agent +on an OT segment. - inventory.instances = { - sshd.roles.server.tags.all = { }; - admin.roles.default.tags.all = { }; # admin users and keys +## What changes in the clan repo - zerotier = { - roles.controller.machines.core-1 = { }; - roles.moon.machines.vps-1.settings.stableEndpoints = [ "/9993" ]; - roles.peer.tags.all = { }; +Following the "Adding a site" procedure in the clan runbook, plus the +plant's own modules. Everything below is one pull request per machine +class, in this order. + +``` +clan.nix + fleet: + gw-lpn-1, vinyl-core, vinyl-backup, vinyl-edge-press, + vinyl-edge-util, vinyl-st-press1, vinyl-st-lab, + vinyl-st-pack1, vinyl-st-ship (then the rest as they come) + borgbackup: + roles.server.machines.vinyl-backup = { }; + roles.client.machines.vinyl-core.settings.destinations = { + plant = { repo = "borg@vinyl-backup:vinyl-core"; }; + offsite = { repo = ":vinyl-core"; }; }; + roles.client.machines.vinyl-edge-press = { ... plant only ... }; + # side benefit: control can now back up its VictoriaMetrics TSDB to + # vinyl-backup, closing the open item in the clan TODO + (no `internet` entries: plant machines sit behind PPPoE like gw-cnx-1 + and are reached over the mesh) - borgbackup = { - roles.server.machines.backup-1 = { }; - roles.server.machines.vps-1 = { }; - roles.client.tags.core = { }; - roles.client.tags.edge = { }; - }; +modules/mesh-hosts.nix + every plant machine in the name list +modules/monitoring/server.nix + + node_exporter targets for every plant machine + + a second Grafana datasource "vinyl-vm" pointing + at [vinyl-core mesh]:8428 so fleet dashboards + can show plant series +modules/monitoring/alerts.nix + unchanged; host rules already cover new targets +machines/ns1/configuration.nix + + dns-acme-gw-lpn-1-knot generator, keyFiles entry, + acl_acme_gw_lpn_1 scoped to _acme-challenge.lpn1 + (copy the gw-cnx-1 blocks) +modules/dns/acme-gw-secret.nix + reused as-is with "gw-lpn-1" +modules/web-proxy.nix + virtualHosts."portal." -> [vinyl-core mesh]:8080 + (second cert + scoped TSIG key if is + not cnx.network, following the acme_web01 pattern) - monitoring = { - roles.server.machines.core-1.settings.grafana.enable = true; - roles.client.tags.all = { }; - }; - }; +machines/gw-lpn-1/ copy of gw-cnx-1: site "lpn", siteId 3, the VLAN + table above, proxy.services.factory and .grafana, + proxy.allowVlans = [ "mgmt" "lan" "stations" ] +machines/vinyl-core/ imports modules/vinyl/{core,metrics,factory-app, + alertmanager}.nix + monitoring/exporters.nix +machines/vinyl-backup/ imports modules/vinyl/backup.nix + exporters.nix +machines/vinyl-edge-*/ imports modules/vinyl/edge.nix with the cabinet's + Telegraf inputs and the OT NIC config +machines/vinyl-st-*/ imports modules/vinyl/station.nix with the kiosk URL - # Our own NixOS modules, applied by tag or by machine: - # modules/edge-collector.nix Telegraf with the tag lists for that cabinet, - # Mosquitto bridge with disk buffer - # modules/core-data.nix PostgreSQL + TimescaleDB, Mosquitto, Telegraf - # persister, Garage, Grafana, pgBackRest - # modules/factory-app.nix the application service and its migrations - # modules/station-kiosk.nix cage + browser, HID readers, auto-login - # modules/router.nix nftables zoning, VLAN interfaces, chrony, - # dual-WAN failover -} +modules/vinyl/core.nix PostgreSQL (ZFS dataset, pg_hba for the app and + the replica), Mosquitto (per-client passwords from + vars generators), Telegraf persister (MQTT in, + VictoriaMetrics out), ZFS datasets and sanoid +modules/vinyl/metrics.nix VictoriaMetrics + vmalert + Grafana for the plant, + a sibling of monitoring/server.nix with the + process rules and dashboards under + modules/vinyl/dashboards/ +modules/vinyl/alertmanager.nix + receivers: Telegram or LINE for the shift group, + SMS gateway for the critical set, email digest; + tokens from vars generators +modules/vinyl/factory-app.nix + the cnx-factory package as a flake input (like + nixos-mailserver is today), systemd service, + migrations on start, DB password from vars, + OIDC client secret from vars +modules/vinyl/edge.nix Telegraf with per-machine input lists, MQTT output + with disk buffer, chrony `allow` on the OT NIC, + networkd config for the two NICs, SFTP chroot for + vision images (press collector only) +modules/vinyl/station.nix cage + browser kiosk, auto-login, HID readers, + SMB share and ingestion watcher where needed +modules/vinyl/backup.nix borg server role settings, syncoid receive, + PostgreSQL replica, pgBackRest repo +modules/router/… the three extensions above +docs/src/vinyl.md the plant chapter of the runbook, + SUMMARY.md ``` -What clan provides out of the box and we do not write ourselves: SSH and -admin user management, the ZeroTier mesh (controller on `core-1`, the VPS -as a publicly reachable moon so the plant and remote consultants find each -other behind NAT), Borg backups with generated keys, host metrics and -journal shipping with Grafana on the server role, and sops-encrypted -secrets per machine. What we write: the five modules listed in the sketch -and the factory application itself. +Operational notes that already bit the fleet and apply here: `git add` a +new `machines/` directory before evaluating (untracked files are invisible +to the flake); run `clan vars generate ` (YubiKey) before the +first install and again after adding a generator; check `facter.json` for +the real NIC names after install; the `clan` wrapper in the dev shell +updates machines sequentially so YubiKey prompts are not buried. + +Repository layout, decided: infrastructure lives in `cnx-network-clan`; +the factory application gets its own repository `cnx-factory` in the B4L +organisation and is consumed as a flake input; this repository stays +documents and links to both. That satisfies the README's promise that the +project repository "hosts the software" without mixing Nix, Rust and +planning prose in one history. ## Data acquisition, source by source @@ -268,12 +380,12 @@ is bought, not retrofitted. Rates are planning values. | Source | Signals | Interface | Rate | Requirement to write into the RFQ | |---|---|---|---|---| -| Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA) | cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes | OPC UA server on the PLC, read by Telegraf `inputs.opcua` | per cycle plus 1 Hz | OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty | +| Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA) | cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes | OPC UA server on the PLC, read by Telegraf `inputs.opcua` | per cycle plus 1 Hz | OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty, PLC clock accepting NTP | | Newbilt HRP 12 semi-automatic and VEX 50 extruder (PLC type unconfirmed) | as above where the PLC has them; on a relay-logic machine at least press closed/open, steam on/off, cycle count | OPC UA or Siemens S7 (`inputs.s7comm`); fallback: our own 4 to 20 mA sensors on a Modbus TCP I/O module and a cycle count from the mold-closed limit switch | per cycle plus 1 Hz | state what the PLC is and which of the two paths applies; the fallback sensor set is priced separately | -| Pheenix AD12 (Beckhoff PLC, remote monitoring supported) | same tag family as the Classic | OPC UA (Beckhoff TF6100) or ADS; `inputs.opcua` | per cycle plus 1 Hz | OPC UA license included, tag list delivered, remote monitoring routed through our jump host | +| Pheenix AD12 (Beckhoff PLC, remote monitoring supported) | same tag family as the Classic | OPC UA (Beckhoff TF6100) or ADS; `inputs.opcua` | per cycle plus 1 Hz | OPC UA license included, tag list delivered, remote monitoring routed through our mesh, not the vendor's | | Hydraulic aggregate | system pressure, oil temperature, tank level, filter differential pressure | via the press PLC if exposed, else own transmitters on the I/O module | 1 Hz | ask whether the aggregate's sensors are on the PLC | | Label dry oven | chamber temperature, timer state | Modbus if the controller has it, else a thermocouple on the I/O module | 1/min | | -| PVC grinder and regrind scale | run state, batch weight | scale with RS232 to `st-press`, run contact to I/O | per batch | scale must have a serial output | +| PVC grinder and regrind scale | run state, batch weight | scale with RS232 to `vinyl-st-press1`, run contact to I/O | per batch | scale must have a serial output | ### Utilities and energy @@ -305,10 +417,9 @@ is bought, not retrofitted. Rates are planning values. Industrial RS485 Modbus RTU transmitters on wired loops to the nearest edge collector, one reading per minute, with a reference-grade PM instrument in -the archive or cutting room to check the cheaper sensors against. Cheap -ESP32/MQTT sensor nodes are acceptable for extra density in non-critical -zones, never for the alarmed ones. Every zone alarm has two thresholds -(warning, action) and a rate-of-change alarm for the archive. +the archive or cutting room to check the cheaper sensors against. Every +zone alarm has two thresholds (warning, action) and a rate-of-change alarm +for the archive. ### Galvanics (Expand phase) @@ -324,9 +435,9 @@ results are entered at a station against the bath ID. Access-control door events (badge ID, door, direction, time) from the controller's API or export; fire alarm panel zone and fault contacts, BESS off-gas and early-smoke detection, UPS on-battery and low-battery (NUT on -`core-1` talking to the UPS) as dry contacts on `edge-utility`. CCTV stays -on its own VLAN with its own retention; it is never joined to production -data. +`vinyl-core` talking to the UPS) as dry contacts on `vinyl-edge-util`. +CCTV stays on its own VLAN with its own retention; it is never joined to +production data. ### What produces no telemetry @@ -340,21 +451,22 @@ logged by the sensors above. | Instrument | How results reach the database | Key | Open integration item | |---|---|---|---| -| Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration) | vision controller sends one result string per record over TCP to a receiver on `edge-press`; images pushed by FTP/SFTP to object storage | press ID + cycle counter, passed from the PLC to the vision trigger | the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test | -| Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer) | vendor Windows PCs export CSV and raw 3D to an SMB share on `st-metrology`; a watcher parses and posts | back-side Data Matrix serial, scanned at the bench before measurement | export format per instrument confirmed with Keyence Thailand | -| PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR) | instruments with RS232/USB export are read at `st-lab`; the rest are typed in with the printout photographed; CoA PDF attached | compound lot ID (printed label scanned at intake) | which benchtop models have serial export | -| Audio QC station | capture service on `st-audioqc` records the needle-drop (PipeWire, FLAC to object storage); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted | record key (stamper + press + cycle + side), typed or picked from the run list | first-of-kind pipeline, developed in Expand phase against launch-phase recordings | +| Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration) | vision controller sends one result string per record over TCP to a receiver on `vinyl-edge-press`; images pushed by SFTP to the collector, synced to the core's image dataset | press ID + cycle counter, passed from the PLC to the vision trigger | the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test | +| Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer) | vendor Windows PCs export CSV and raw 3D to an SMB share on `vinyl-st-metrology`; a watcher parses and posts | back-side Data Matrix serial, scanned at the bench before measurement | export format per instrument confirmed with Keyence Thailand | +| PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR) | instruments with RS232/USB export are read at `vinyl-st-lab`; the rest are typed in with the printout photographed; CoA PDF attached | compound lot ID (printed label scanned at intake) | which benchtop models have serial export | +| Audio QC station | capture service on `vinyl-st-audioqc` records the needle-drop (PipeWire, FLAC to the core's capture dataset); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted | record key (stamper + press + cycle + side), typed or picked from the run list | first-of-kind pipeline, developed in Expand phase against launch-phase recordings | | Packaging assembly check | camera with darkfield lighting reads the scribed dead-wax code (a handheld scanner will not read a scribed code on black vinyl reliably); handheld scanner reads the jacket barcode; barcode verifier grades it | order + record key | confirm dead-wax code readability with a sample before the packaging station is designed | | Label print inspection | same vision family as the record label check; result per label lot | label lot ID | shared with the record inspection budget | ## The factory application -Working name `cnx-factory`. Server-rendered web application in Rust (axum, -SQLx, minijinja templates, HTMX) on PostgreSQL, the stack B4L already uses -for its other in-house tooling, with the same append-only ledger and -hash-chained audit log pattern. It subscribes to the MQTT event topics -(cycle complete, machine alarm, instrument result) and owns every write to -the identity tables; Telegraf owns the writes to the time-series tables. +Working name `cnx-factory`, its own repository. Server-rendered web +application in Rust (axum, SQLx, minijinja templates, HTMX) on PostgreSQL, +the stack B4L already uses for its other in-house tooling, with the same +append-only ledger and hash-chained audit log pattern. It subscribes to +the MQTT event topics (cycle complete, machine alarm, instrument result) +and owns every write to the identity tables; Telegraf owns the writes to +VictoriaMetrics. Packaged as a flake so the clan repo pins it as an input. Modules, in build order: @@ -377,8 +489,9 @@ Modules, in build order: lots with incoming QC. 7. **Press runs and cycles**: a run binds order, stamper pair, press, operator, shift, PVC lot in the hopper, regrind batch, label lot; each - cycle event from the PLC becomes a row keyed by press + counter; the - record key is derived from it per side. + cycle event from the PLC becomes a row keyed by press + counter with + the key press values and the zone environment snapshot; the record key + is derived from it per side. 8. **QC verdicts**: optical per record, audio per sampled record, lab per lot, packaging per unit; reject reasons; test-pressing hold state. 9. **Finishing, packaging, fulfillment**: assembly checks, consignment @@ -388,16 +501,15 @@ Modules, in build order: one customer sees only its own orders; PDF export of the trace. 11. **Reports and signals**: yield per stamper, reject rate drift, audio score trend versus play count (the automatic retirement signal), - energy per record, environment excursions overlaid on runs. + energy per record, environment excursions overlaid on runs (the + curves fetched from VictoriaMetrics for the run's window). Core entities: `customer`, `order`, `release`, `side`, `audio_asset`, `gate_approval`, `session`, `lacquer`, `father`, `mother`, `stamper`, `stamper_measurement`, `press`, `press_run`, `press_cycle`, `record`, `optical_result`, `audio_result`, `pvc_lot`, `regrind_batch`, `label_lot`, `carton_lot`, `lab_result`, `packaging_check`, `shipment`, `operator`, -`shift`, `event` (the hash chain). Time-series hypertables: `telemetry` -(machine, tag, value), `environment` (zone, metric, value), `energy` -(meter, kWh, kW), `water`. +`shift`, `event` (the hash chain). Stations use the same application in kiosk mode: an operator taps a badge (the same card the access-control system issues, so one ID) and scans @@ -405,55 +517,64 @@ whatever the step asks for. No forms longer than one screen. ## Observability and alerting -Two tiers, one Grafana: +Two tiers, split by where the data is: -- **Process alarms** from TimescaleDB: zone environment excursions and - rate of change, boiler pressure and level, chiller return temperature, - compressed-air pressure, hydraulic pressure, reject rate per run above - baseline, stamper approaching its play-count limit, PV or BESS faults, - fire and BESS detection contacts, UPS on battery. -- **Infrastructure alarms** from the clan monitoring stack: host down, - disk, service failed, backup missed, certificate expiry. +- **Process, on site** (`vinyl-core`: VictoriaMetrics, vmalert, + Grafana, Alertmanager): zone environment excursions and rate of change, + boiler pressure and level, chiller return temperature, compressed-air + pressure, hydraulic pressure, reject rate per run above baseline, + stamper approaching its play-count limit, PV or BESS faults, fire and + BESS detection contacts, UPS on battery. Works with the WAN down. +- **Hosts, on the fleet** (`control`): node_exporter and journald from + every plant machine over the mesh, the existing `ScrapeTargetDown`, + `RootFilesystemFull`, `BackupJobFailed`, `BackupStale` rules apply to + them without changes. - **Data-silence alarms** in both tiers: a collector that has published nothing for five minutes is an alarm, whatever the last value was. -Routing: Telegram or the LINE Messaging API for the shift group, email for -the daily digest, SMS through a Thai gateway for the critical set (boiler, -fire, BESS, archive climate) with an on-call rota. The animated +Routing: Telegram or the LINE Messaging API for the shift group, email +digest daily, SMS through a Thai gateway for the critical set (boiler, +fire, BESS, archive climate) with an on-call rota. `control`'s vmalert can +drop `notifier.blackhole` and point at the same Alertmanager over the +mesh, which closes the fleet's own alert-delivery item. The animated [plant operation page](plant-operation.html) is the design target for the -live overview: once the telemetry tables exist it is fed from Grafana or -directly from the database instead of its simulation. +live overview: once the series exist it is fed from VictoriaMetrics +instead of its simulation. ## Backup, retention and recovery -- **Hourly ZFS snapshots** on `core-1` (sanoid), replicated to `backup-1` - (syncoid). -- **Nightly Borg** from every `core` and `edge` machine to `backup-1` and - to `vps-1` (clan borgbackup, two destinations). +- **Hourly ZFS snapshots** on `vinyl-core` (sanoid), replicated to + `vinyl-backup` (syncoid) for the file datasets and the PostgreSQL data + directory. +- **Nightly Borg** (clan `borgbackup`) from `vinyl-core` and the edge + collectors to `vinyl-backup`, and from `vinyl-core` to the off-site + destination as a second entry in `destinations`; repokey encryption as + the fleet already does. - **PostgreSQL point-in-time recovery**: pgBackRest with WAL archiving to - object storage, replicated to the off-site node; a streaming replica on - `backup-1` for fast failover. -- **Object storage** (Garage) with three nodes: `core-1`, `backup-1`, - `vps-1`; audio captures, inspection images, metrology raw data and CoA - scans survive the loss of any one node. -- **Configuration**: the clan repo in Gitea, mirrored off-site; any machine - is rebuilt from it. + `vinyl-backup`, its repository included in the off-site Borg set; a + streaming replica on `vinyl-backup` for fast failover. +- **VictoriaMetrics**: `vmbackup` snapshots to `vinyl-backup` nightly; the + fleet's own open item (backing up `control`'s TSDB) gets a destination + at the same time. +- **Configuration**: the clan repo and `cnx-factory` in Gitea; any machine + is rebuilt from them. - **Restore drill quarterly**: restore the database to a scratch VM and open a random order's trace in the portal. A backup that has never been - restored does not count. + restored does not count. The `BackupStale` rule already watches the + timers. | Data | Retention | |---|---| | genealogy, orders, gates, QC verdicts, event log | life of the customer relationship, never deleted while the metal parts exist | -| per-cycle press data | forever (small) | -| 1 Hz machine telemetry | raw 2 years, then 1-minute aggregates forever | -| zone environment | raw 5 years | -| energy and water | forever (small) | +| per-cycle press rows | forever (small) | +| 1 Hz machine telemetry | 2 years in VictoriaMetrics (`retentionPeriod`); per-cycle rows carry the summary forever | +| zone environment | 5 years (separate VM retention filter, or a second small instance) | +| energy and water | forever (small; also summarised per shift into PostgreSQL) | | audio captures and inspection images | 2 years, longer for any record with a complaint | | client audio | per contract; default deleted 12 months after final delivery, or on request | | access-control events | 1 year | | CCTV | 30 days | -| host logs and metrics | 90 days | +| host logs and metrics | fleet defaults: 30 d logs, 180 d metrics on `control` | PDPA applies to badge events, CCTV and customer contact data: a written retention policy, role-based access, and export/deletion on request. The @@ -461,18 +582,24 @@ customer portal shows operator roles, never operator names. ## Security -- OT VLANs have no internet and no inbound routes; edge collectors are the - only bridge and they only publish. -- Staff sign-on through Pocket ID (OIDC, passkeys) with roles: admin, - engineer, operator, sales, customer. Stations authenticate by badge for - attribution, not for privilege. -- Secrets only in clan vars; nothing in plain text in the repo. +- OT segments have no gateway interface and no internet; edge collectors + are the only bridge and they only publish. +- Stations and office reach the application only through the gateway's + Caddy, with the wildcard certificate the fleet already issues; the + servers VLAN accepts nothing from other VLANs. +- Staff sign-on through an OIDC provider on `vinyl-core` (Pocket ID, + passkeys; the fleet has no identity provider yet, so this is the first) + with roles: admin, engineer, operator, sales, customer. Stations + authenticate by badge for attribution, not for privilege. +- Secrets only in clan vars (YubiKey-gated, `admins` group); nothing in + plain text in either repo. - Client audio: separate encrypted ZFS dataset, SFTP or portal upload with checksums, every read logged, a written policy sales can show a label. -- UPS on `core-1`, `router-1`, the switches, the edge collectors and the - press PLCs (already required by facility security); NUT triggers a clean - shutdown before the battery is exhausted. -- Signed commits on the clan repo; deployments only from a reviewed branch. +- UPS on `vinyl-core`, `gw-lpn-1`, the switches, the edge collectors and + the press PLCs (already required by facility security); NUT triggers a + clean shutdown before the battery is exhausted. +- Admin access rides the mesh, as for every fleet machine; public SSH is + never opened. ## Procurement clauses @@ -481,7 +608,7 @@ To go into every RFQ that leaves from [roadmap](roadmap.md) items 4 to 8: 1. **Presses (Newbilt, Pheenix)**: OPC UA (or S7/ADS) read access licensed and enabled; the full tag list with units and the cycle counter as a tag; a defined hand-off of the cycle counter to the inspection trigger; - remote support through our jump host; no cloud agent as a warranty + remote support through our mesh; no cloud agent as a warranty condition; PLC clock accepting NTP. 2. **Boiler, chiller, compressor, heat pump (Thai vendors)**: Modbus RTU or TCP with the register map delivered; alarm contacts wired to a terminal @@ -491,13 +618,16 @@ To go into every RFQ that leaves from [roadmap](roadmap.md) items 4 to 8: loops terminated in the edge cabinets. 4. **Electrical contractor**: Modbus energy meters on the feeders listed above; RS485 to the inverters and BESS BMS without cloud dongles; - structured cabling and a VLAN-capable switch in every cabinet. + structured cabling, a VLAN-capable industrial switch in every cabinet, + and Omada-managed switches and APs for the building so the gateway's + existing Omada controller covers the site. 5. **Galvanics (Digital Matrix)**: what the rectifiers and control rings expose, and how; Modbus preferred. 6. **Metrology (Keyence Thailand)**: confirmed CSV/3D export paths and whether the instrument software can write to a network share unattended. -7. **Vision integrator**: result string format, image push, cycle counter - correlation, and an acceptance test defined in records not in features. +7. **Vision integrator**: result string format, image push over SFTP, + cycle counter correlation, and an acceptance test defined in records not + in features. 8. **Access control and fire**: API or export for door events; dry contacts for every alarm class; BESS detection outputs. @@ -509,23 +639,26 @@ Aligned with the [roadmap](roadmap.md) phases. - Put the procurement clauses above into the Newbilt re-quote, the Pheenix, Digital Matrix and Thai utility RFQs. -- Create the clan repo and a development clan of three small machines at - the office: a `core-1` stand-in, one edge collector with a Modbus and - OPC UA simulator, one station. This proves the tooling before a single - plant machine exists. -- Build the factory application v0 against a press simulator (the state - machine in the plant operation page is the seed: same states, same - cycle timing) so orders, gates, genealogy and press cycles can be - exercised end to end without hardware. -- Decide the repository layout: this repository stays documents; the clan - flake and the application get their own repositories in the same Gitea - organisation, linked from here (see open questions). +- In the clan repo: reserve the site id and VLAN table, write + `modules/vinyl/{core,metrics,edge,station}.nix`, and stand up a dev + trio at the Chiang Mai site behind `gw-cnx-1` (a `vinyl-core` stand-in, + one edge collector with a Modbus and OPC UA simulator, one station). + This proves the modules, the mesh path and the proxy before a single + plant machine exists, and the machines move to the plant when it does. +- Land the three router-module extensions on `gw-cnx-1` first, where they + are also useful. +- Start `cnx-factory` v0 against a press simulator (the state machine in + the plant operation page is the seed: same states, same cycle timing) so + orders, gates, genealogy and press cycles can be exercised end to end + without hardware; package it as a flake from the first commit. +- Wire the Alertmanager and one real receiver; point `control`'s vmalert + at it. **Build and commission (roadmap Phase 3).** -- Network, `router-1`, `core-1`, `backup-1` and `vps-1` installed before +- `gw-lpn-1`, switches, `vinyl-core` and `vinyl-backup` installed before the machinery arrives, so vendor commissioning engineers plug into a - finished network. + finished network; ns1 gets the gateway's ACME key at the same time. - Zone sensor loops, meters and edge cabinets installed by the electrical and HVAC contractors during fit-out; environment logging is live before pressing starts, which gives a baseline. @@ -535,18 +668,19 @@ Aligned with the [roadmap](roadmap.md) phases. - Application v1 live: intake, gates, bridge-stamper genealogy, press runs and cycles, PVC lot gate, packaging scan, shipments; stations at press, lab, packaging and shipping. +- Off-site backup destination live and a first restore drill done. **Launch (roadmap Phase 4).** - Optical inspection integrated and correlated; reject reasons flowing. -- Customer portal opened to the first labels. +- Customer portal opened to the first labels through `web01`. - Energy and environment dashboards used for the first PV and HVAC tuning. **Expand (roadmap Phase 5).** -- `edge-galvanics`, bath sensors, rectifier data, metrology ingestion at - `st-metrology`; genealogy switches from bridged to in-house. +- `vinyl-edge-galv`, bath sensors, rectifier data, metrology ingestion at + `vinyl-st-metrology`; genealogy switches from bridged to in-house. - Audio QC station and pipeline; retirement signal live. - Automated press line joins the same collector with the same tag family. - Biomass boiler on the same Modbus path as its predecessor. @@ -555,21 +689,23 @@ Aligned with the [roadmap](roadmap.md) phases. Planning estimates, September 2026, excluding the QC instruments (those are in quality automation) and excluding the vendors' OPC UA license -options, which come back with the quotes. +options, which come back with the quotes. Fleet pieces already paid for +(`control`, `ns1`, `ns2`, `web01`, the mesh) cost the plant nothing extra. | Block | Estimate (USD) | |---|---| -| `core-1` and `backup-1` servers, UPS, rack | 8,000 to 15,000 | -| `router-1`, managed switches (industrial in the cabinets), cabling allowance | 4,000 to 8,000 | +| `gw-lpn-1` (Topton 1U, fleet standard) | 300 to 500 | +| `vinyl-core` and `vinyl-backup` servers, UPS, rack | 8,000 to 15,000 | +| Omada switches and APs for the building, industrial switches in the cabinets, cabling allowance | 4,000 to 8,000 | | 3 edge collectors with I/O modules and enclosures | 3,000 to 6,000 | | 8 station kiosks with readers, scanners, one barcode verifier, one darkfield camera | 8,000 to 15,000 | | zone environment sensors (12 to 16 points), room dP, reference PM instrument | 4,000 to 8,000 | | energy meters (10 feeders), water meters, level transmitters | 3,000 to 6,000 | | audio QC hardware (turntable, arm, cartridge, phono stage, interface) | 2,000 to 5,000 | -| `vps-1` and off-site storage, per year | 500 to 1,500 | +| off-site backup storage, per year | 100 to 500 | | **Hardware total** | **~32,000 to 64,000** | -| Application development to v1 (orders through shipments, stations, portal) | 4 to 6 person-months | -| Clan fleet, modules and collectors to commissioning | 1 to 2 person-months | +| `cnx-factory` to v1 (orders through shipments, stations, portal) | 4 to 6 person-months | +| Clan modules, router extensions, collectors, dashboards and rules to commissioning | 1 to 2 person-months | | Vision, metrology and audio pipeline integration (Launch and Expand) | 3 to 5 person-months, plus integrator fees in the vision quote | Headcount note: [scale](scale.md) lists no IT or automation role in the @@ -578,22 +714,22 @@ Build onward, part-time at launch; add it to the headcount. ## Open questions +- Site code and id: `lpn` / 3 assumed; confirm when the site is chosen and + the fleet site table is updated. - Semi-automatic press PLC: does the HRP 12 expose data, or is the sensor fallback needed? (Ask in the Newbilt re-quote.) - Cycle counter hand-off to the vision system: PLC output, network message, or time correlation at the unloader? Decided with the vision integrator, tested at acceptance. - HVAC BMS protocol and who owns the zone sensors: HVAC contractor or us. -- Where `vps-1` lives: outside Thailand (latency to the mesh, data - residency for customer data) or a Bangkok provider. -- Portal availability: served through the mesh from `core-1` (down when - the plant's internet is down) or a read replica on `vps-1` (more moving - parts). Start with the former. +- Off-site backup destination: Hetzner Storage Box (cheapest) or a storage + VPS that is also a mesh peer (uniform tooling). Data residency for + customer audio may argue for a Bangkok provider instead. +- Customer portal domain: the plant's brand domain (new zone on ns1/ns2, + second cert on `web01`) or a name under `cnx.network` for launch. +- Portal availability during a plant WAN outage: accept it (the portal is + a view of live data) or add a read replica on the fleet side later. - Accounting and ERP: out of scope for the application; if an ERP is adopted later it references the same order IDs. Decide after launch. -- Repository layout: documents here, `cnx-infra` (clan) and `cnx-factory` - (application) as separate repositories, or everything in one. The - README currently promises this repository will hold the software; a - link satisfies that. - Dead-wax code readability on black and coloured vinyl with a camera: test with samples before the packaging station is designed.