B4L/cnx_vinyl
Private
Public Access
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Digital factory plan: build on the existing cnx-network clan

Browse Source 
Rewrite docs/digital-factory.md against the fleet that already exists at
git.b4l.co.th/B4L/cnx-network-clan instead of a plant-only clan:

- The plant is one more site: gw-lpn-1 from the cnx.router module, plant
  machines in the same inventory, reached over the existing ZeroTier mesh
  (no VPS, no moon, no port forwards).
- Fleet pieces reused as-is: control for host metrics, journals and host
  alerts; ns1/ns2 for the gateway wildcard cert; web01 as the public
  proxy for the customer portal; clan borgbackup with a plant server.
- Process time series move from TimescaleDB to a plant-local
  VictoriaMetrics (same modules and rules as control); PostgreSQL keeps
  everything with identity. Object storage dropped for ZFS datasets.
- VLAN table on the fleet addressing convention (siteId 3 assumed, ids
  above 40); OT segments as dual-homed islands behind the collectors;
  stations reach the app only through the gateway Caddy proxy.
- New section listing every file that changes in the clan repo, the
  modules/vinyl/* to write, and three router-module extensions (NTP
  serving, static VLANs, second WAN).
- Gaps the plant closes for the fleet: Alertmanager delivery, off-site
  backup destination, TSDB backup.
- Repository layout decided: infra in cnx-network-clan, app in
  cnx-factory as a flake input, this repo stays documents.

README bullet updated to match.
This commit is contained in:
grabowski committed 2026-09-21 21:41:49 +02:00
1 parent 0fb60bcbb5
commit ffcce42a51
2 files changed
+413 -276
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+6 -5
View File
@@ -88,11 +88,12 @@ The detail lives in dedicated documents:
shipment, and the requirements this sets for the factory software shipment, and the requirements this sets for the factory software
- [Digital factory systems plan](docs/digital-factory.md): every system - [Digital factory systems plan](docs/digital-factory.md): every system
that has to exist for environment, machine, QC and genealogy data to land that has to exist for environment, machine, QC and genealogy data to land
in one database keyed to one record identity: the NixOS machine fleet in one database keyed to one record identity: the plant as one more site
managed as one Clan, networks and VLAN zoning, Telegraf/MQTT acquisition in B4L's existing NixOS fleet (the cnx-network clan: mesh, DNS, fleet
source by source, QC instrument integration, the factory application, monitoring, site gateway module, public proxy), what changes in that
alerting, backup, security, the data clauses for every RFQ, build order repo, VLAN zoning, Telegraf/MQTT acquisition source by source, QC
and rough cost instrument integration, the factory application, alerting, backup,
security, the data clauses for every RFQ, build order and rough cost
- [Quality automation](docs/quality-automation.md): stamper metrology - [Quality automation](docs/quality-automation.md): stamper metrology
(Keyence-class confocal, profiler, XRF), layered pressed-record QC (100% (Keyence-class confocal, profiler, XRF), layered pressed-record QC (100%
optical inspection, per-cycle press telemetry, sampled automated audio optical inspection, per-cycle press telemetry, sampled automated audio
docs/digital-factory.md
+407 -271
View File
@@ -9,44 +9,84 @@ genealogy all land in one place, keyed to one record identity, and can be
shown to a customer. It is a planning position as of September 2026; every shown to a customer. It is a planning position as of September 2026; every
cost figure is an estimate until a quote replaces it. cost figure is an estimate until a quote replaces it.
**Standing decision: every service runs on NixOS, and the whole fleet is **Standing decision: every service runs on NixOS, and the plant is one more
declared and deployed as one [Clan](https://clan.lol).** Servers, edge site in the existing B4L fleet, the
collectors, station kiosks, the router and the off-site VPS are all NixOS [cnx-network clan](https://git.b4l.co.th/B4L/cnx-network-clan).** That
machines in a single inventory. Vendor instrument PCs (Keyence, vision clan already runs the ZeroTier mesh, authoritative DNS, mail, fleet
controllers, PLC HMIs) are appliances, not servers: they export into the monitoring, backups, a public reverse proxy and the first site gateway, all
NixOS estate and never host a service of ours. declared in one repository and deployed with the clan CLI. The plant adds
a gateway, a core server, a backup server, edge collectors and station
kiosks to that inventory and reuses everything the fleet already provides.
Nothing is built twice. Vendor instrument PCs (Keyence, vision controllers,
PLC HMIs) are appliances, not servers: they export into the NixOS estate
and never host a service of ours.
## What the fleet already provides
Read the clan's own runbook (`docs/` in the repo, served on `control`) for
detail; this is what the plant plan leans on.
| Fleet piece | Where | What the plant gets from it |
|---|---|---|
| Clan repo, clan-core 26.05, sops vars gated by YubiKey, `admins` group applied to every machine automatically | git.b4l.co.th/B4L/cnx-network-clan | one inventory, one deploy path (`clan machines install` / `update`), secrets minted with `clan vars generate` |
| ZeroTier mesh, controller on `control`, IPv6 ULA `/88`, `modules/mesh-hosts.nix` reads every peer's address from vars | `control` | admin SSH, deploys and scrapes for plant machines behind PPPoE NAT, exactly as `gw-cnx-1` is reached today; no port forwards, no VPS needed |
| `modules/router/`: `cnx.router.*` on Topton 1U boxes: PPPoE WAN, VLAN-filtering bridge, nftables with no inter-VLAN traffic, Kea, Blocky, CrowdSec, Omada, iperf3, hourly speed test, Caddy internal proxy with a real wildcard `*.<site><n>.cnx.network` via DNS-01 | site gateways | the plant router is a copy of `machines/gw-cnx-1/` with its own VLANs; the internal proxy gives every station a browser-trusted `https://factory.<site>1.cnx.network` |
| Addressing convention `10.<siteId>.<vlanId>.0/24`, VLAN ids 10 mgmt and 20 lan mandatory, 30 guest, 40 iot reserved | fleet-wide | the plant takes the next free `siteId` and adds its own VLANs above 40 |
| VictoriaMetrics (180 d), VictoriaLogs (30 d), vmalert, Grafana, blackbox probes | `control` | node_exporter on every plant machine scraped over the mesh, journald shipped to VictoriaLogs, host alerts from the existing rules; process data stays on site (below) |
| Knot DNS (`cnx.network`, `buildfor.life`, `cnx.email`) with per-consumer TSIG keys for ACME | `ns1`, `ns2` | the plant's public names and the gateway's wildcard cert; a brand domain for the customer portal is one zone file away |
| `web01`: public Caddy with `*.cnx.network`, backends dialed over the mesh | Hetzner | the customer portal is one `virtualHosts` entry pointing at the plant core's mesh address, the same way `grafana.cnx.network` works now |
| clan `borgbackup` service, `control` as server, repokey encryption, backups over the mesh | `control` | the pattern; the plant needs its own server because plant data is far larger than the DNSSEC keystore |
| chrony everywhere, fixed-offset time zones (`Etc/GMT-7` for Thailand) | fleet baseline | plant machines inherit; edge collectors additionally serve NTP to the PLCs |
| mdbook runbook in `docs/src/`, built and served on `control` | `control` | the plant's operational runbook goes there as a chapter; this document stays the planning source |
Known gaps in the fleet that the plant needs closed (all on the clan's own
TODO list or small extensions to existing modules):
- **Alert delivery**: vmalert currently uses `notifier.blackhole`. A boiler
alarm needs a phone. The plant brings an Alertmanager with Telegram or
LINE and SMS receivers; `control`'s vmalert can point at it too.
- **Off-site backup destination**: `control`'s disk is not sized for plant
images and audio. Add a Hetzner Storage Box or a storage VPS as a second
`borgbackup` destination (the clan service supports several per client).
- **Router module**: three small extensions, listed under Networks.
## Design decisions ## Design decisions
1. **One clan, one repo, rebuildable from git.** Machine configuration, 1. **One clan, one repo, rebuildable from git.** Plant machines are entries
service placement, mesh networking, backups, host monitoring and in `clan.nix`'s `fleet` set with a `machines/<name>/` directory each
secrets (clan vars, sops-encrypted) live in one flake. A failed machine (`configuration.nix`, `disko.nix`, `facter.json`), like every machine
is replaced with `clan machines install <name>` from a fresh disk; a already there. A failed box is replaced from a fresh disk with
configuration change is `clan machines update`. Nothing is configured by `clan machines install`; a change is `clan machines update`. Nothing is
hand on a running box. configured by hand on a running machine.
2. **One relational database for everything that has identity.** 2. **Two stores, by shape of the data.** Everything with an identity
PostgreSQL with TimescaleDB holds the genealogy, orders, gates, QC (orders, gates, genealogy, lots, press runs, per-cycle rows, QC
results, per-cycle press data, environment history and energy history. verdicts, events) lives in PostgreSQL on the plant core. Everything
Joining "this record" to "the mold temperature on that cycle" and to that is a time series (1 Hz machine tags, zone environment, energy,
"the humidity in the press hall that hour" is a query, not an export. water) lives in a plant-local VictoriaMetrics, the same engine and
Host and infrastructure metrics (CPU, disk, systemd, journals) go to modules the fleet already runs on `control`, so dashboards, PromQL and
the separate Prometheus/Loki pair that the clan monitoring service vmalert rules carry over unchanged. Per-cycle rows carry a snapshot of
provides; they are not process data. the zone environment and the key press values, so the relational join
"this record, that cycle, those conditions" never needs the time-series
store; the curve behind the snapshot is one PromQL range query away.
(TimescaleDB in PostgreSQL is the fallback if that split proves
awkward; it is not the starting point.)
3. **MQTT is the plant bus, Telegraf is the universal collector.** Every 3. **MQTT is the plant bus, Telegraf is the universal collector.** Every
machine, meter and sensor is read by a Telegraf instance on an edge machine, meter and sensor is read by a Telegraf instance on the edge
collector near it (Modbus, OPC UA, Siemens S7 and MQTT inputs are collector nearest it (Modbus, OPC UA, Siemens S7 and MQTT inputs are
stock plugins) and published to the broker on the core server. The stock plugins) and published to Mosquitto on the core. On the core,
core side persists everything to TimescaleDB with the same tool. Custom Telegraf writes the stream to VictoriaMetrics (InfluxDB line protocol,
code exists only where no plugin does (vision results, instrument CSV native to VM) and the factory application subscribes to the event
ingestion, the audio QC pipeline). topics (cycle complete, alarm, instrument result) and writes rows.
Custom code exists only where no plugin does (vision results,
instrument CSV ingestion, the audio QC pipeline).
4. **Read-only toward machines.** The data platform never writes to a PLC 4. **Read-only toward machines.** The data platform never writes to a PLC
or an instrument. Control, safety and interlocks (fume extraction, steam or an instrument. Control, safety and interlocks (fume extraction, steam
shutoff, boiler safety chain) stay in the vendor PLCs and hard wiring, shutoff, boiler safety chain) stay in the vendor PLCs and hard wiring,
as [facility security](facility-security.md) already requires. as [facility security](facility-security.md) already requires.
5. **Join keys are counters, not clocks.** Every press cycle carries the 5. **Join keys are counters, not clocks.** Every press cycle carries the
machine's own cycle counter; every instrument result carries the part machine's own cycle counter; every instrument result carries the part
serial it was read from. Wall-clock time is recorded (UTC, NTP from the serial it was read from. Wall-clock time is recorded (UTC, chrony from
router) but is never the thing two systems are joined on. the fleet baseline) but is never what two systems are joined on.
6. **Append-only, hash-chained.** Process events, gate approvals and QC 6. **Append-only, hash-chained.** Process events, gate approvals and QC
verdicts are written once and never updated; corrections are new events verdicts are written once and never updated; corrections are new events
that reference the old. Each event carries the hash of the previous one that reference the old. Each event carries the hash of the previous one
@@ -55,55 +95,65 @@ NixOS estate and never host a service of ours.
7. **The customer portal is a view, not a copy.** It reads the same 7. **The customer portal is a view, not a copy.** It reads the same
database through the same application with a customer role; there is database through the same application with a customer role; there is
no exported report to keep in sync. no exported report to keep in sync.
8. **The plant runs without the internet.** Everything that pressing,
QC and traceability need is on site. A WAN outage stops only the
customer portal, off-site backup, fleet monitoring and remote support,
and all four catch up when it returns.
## The systems on one page ## The systems on one page
| System | What it does | Runs on | Phase | | System | What it does | Runs on | Phase |
|---|---|---|---| |---|---|---|---|
| Network and firewall | VLAN zoning (OT, sensors, servers, stations, office, CCTV, guest), dual WAN, NTP, DHCP/DNS | `router-1` (NixOS, nftables) plus managed switches | Build | | Site gateway | PPPoE WAN, VLAN zoning, DHCP/DNS, NTP for the site, internal TLS proxy, CrowdSec | `gw-<site>-1` (existing `cnx.router` module) | Build |
| Clan fleet | Inventory of every NixOS machine, ZeroTier mesh, SSH and user management, secrets, backups, host monitoring | git repo, applied to all machines | Now (dev), Build (plant) | | Plant core | PostgreSQL, Mosquitto, Telegraf persister, VictoriaMetrics + vmalert + Grafana, Alertmanager, factory application, file datasets on ZFS | `vinyl-core` | Build |
| Plant data platform | PostgreSQL + TimescaleDB, MQTT broker, Telegraf persister, Grafana, S3-compatible object storage | `core-1` | Build | | Factory application | Orders, gates, genealogy, materials, press runs, QC verdicts, packaging, fulfillment, customer portal, audit log | `vinyl-core` | Build (v1), Launch (v2) |
| Factory application | Orders, gates, genealogy, materials, press runs, QC verdicts, packaging, fulfillment, customer portal, audit log | `core-1` | Build (v1), Launch (v2) | | Edge collectors | Telegraf per machine zone, disk-backed store-and-forward, NTP to the OT segment, vision result receiver | `vinyl-edge-press`, `vinyl-edge-util`, `vinyl-edge-galv` | Build, Expand |
| Edge collectors | Telegraf per machine zone, local store-and-forward, vision result receiver | `edge-press`, `edge-utility`, `edge-galvanics` | Build, Expand | | Environmental monitoring | Zone temperature, humidity, PM, differential pressure; alarms; correlation with production | RS485 loops into the edge collectors | Build |
| Environmental monitoring | Zone temperature, humidity, PM, differential pressure; alarms; correlation with production | sensors on RS485 loops into `edge-utility` and `edge-press` | Build |
| Machine telemetry | Presses, extruders, hydraulics, boiler, chiller, compressor, heat pump, HVAC, PV, BESS, energy and water meters | edge collectors | Build | | Machine telemetry | Presses, extruders, hydraulics, boiler, chiller, compressor, heat pump, HVAC, PV, BESS, energy and water meters | edge collectors | Build |
| QC instrument integration | Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans | stations plus `edge-press` | Launch, Expand | | QC instrument integration | Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans | stations plus `vinyl-edge-press` | Launch, Expand |
| Station kiosks | Badge check-in and scanning at each manual step; kiosk browser on the factory application | `st-*` machines | Build | | Station kiosks | Badge check-in and scanning at each manual step; kiosk browser on the factory application | `vinyl-st-*` | Build |
| Observability and alerting | Process alarms (Grafana on TimescaleDB), host alarms (clan monitoring), routing to phones, data-silence alarms | `core-1` | Build | | Process alerting | Environment, utilities, reject rate, stamper life, PV/BESS, fire and BESS detection contacts, UPS, data silence; delivered to phones | `vinyl-core` (vmalert + Alertmanager) | Build |
| Backup and DR | ZFS snapshots, Borg to `backup-1` and off-site, PostgreSQL point-in-time recovery, restore drills | `backup-1`, `vps-1` | Build | | Fleet monitoring | Host metrics, journals, host alerts for every plant machine | `control` (existing) | Build |
| Identity | OIDC single sign-on for staff, badge IDs for operators at stations, role model for customers | `core-1` (Pocket ID) | Build | | Backup and DR | ZFS snapshots and replication, Borg to the plant backup server and off site, PostgreSQL point-in-time recovery, restore drills | `vinyl-backup`, off-site destination | Build |
| Client audio vault | Encrypted, isolated storage for pre-release audio with checksums and access logging | `core-1` (separate dataset) | Build | | Identity | Staff sign-on (OIDC), badge IDs for operators at stations, customer role | `vinyl-core` | Build |
| Site systems integration | Access-control door events, fire panel and BESS detection contacts, UPS state, CCTV | `edge-utility`, optional `nvr-1` | Build | | Client audio vault | Encrypted, isolated dataset for pre-release audio with checksums and access logging | `vinyl-core` | Build |
| Off-site node | ZeroTier moon with a public IP, off-site backup target, public reverse proxy for the portal | `vps-1` | Build | | Site systems integration | Access-control door events, fire panel and BESS detection contacts, UPS state | `vinyl-edge-util` | Build |
| Customer portal, public | TLS termination and forwarding over the mesh to the plant core | `web01` (existing) | Launch |
| CCTV | Recording on its own VLAN | vendor NVR appliance, or `vinyl-nvr` (Frigate) if wanted | Build |
## Architecture ## Architecture
```mermaid ```mermaid
flowchart LR flowchart LR
subgraph L0[Field] subgraph L0[Field, isolated OT segments]
PLC[Press and extruder PLCs<br/>OPC UA / S7] PLC[Press and extruder PLCs<br/>OPC UA / S7]
UT[Boiler, chiller, compressor,<br/>heat pump, HVAC<br/>Modbus / BACnet] UT[Boiler, chiller, compressor,<br/>heat pump, HVAC<br/>Modbus / BACnet]
EN[Energy meters, Deye inverters,<br/>BESS, water meters<br/>Modbus / pulse] EN[Energy meters, Deye inverters,<br/>BESS, water meters<br/>Modbus / pulse]
ENV[Zone T / RH / PM / dP sensors<br/>Modbus RTU] ENV[Zone T / RH / PM / dP sensors<br/>Modbus RTU]
GV[Rectifiers, bath sensors<br/>Modbus] GV[Rectifiers, bath sensors<br/>Modbus]
VIS[Vision controller<br/>TCP results, FTP images] VIS[Vision controller<br/>TCP results, SFTP images]
INST[Keyence, XRF, lab instruments<br/>CSV export] INST[Keyence, XRF, lab instruments<br/>CSV export]
end end
subgraph L1[Edge, NixOS] subgraph L1[Edge, NixOS, dual-homed]
EP[edge-press<br/>Telegraf + vision receiver] EP[vinyl-edge-press<br/>Telegraf + vision receiver]
EU[edge-utility<br/>Telegraf] EU[vinyl-edge-util<br/>Telegraf + dry-contact I/O]
EG[edge-galvanics<br/>Telegraf] EG[vinyl-edge-galv<br/>Telegraf]
ST[st-* stations<br/>kiosk, scanners, ingestion] ST[vinyl-st-* stations<br/>kiosk, scanners, ingestion]
end end
subgraph L2[Core, NixOS] subgraph L2[Plant core, NixOS, servers VLAN]
MQ[MQTT broker] MQ[Mosquitto]
DB[(PostgreSQL + TimescaleDB)] VM[(VictoriaMetrics<br/>vmalert, Grafana)]
OBJ[(Object storage)] DB[(PostgreSQL)]
APP[Factory application] APP[Factory application]
GF[Grafana + alerting] AM[Alertmanager]
end end
subgraph L3[Outside] subgraph GW[gw-site-1]
VPS[vps-1: moon, backup, proxy] PX[Caddy internal proxy<br/>factory.site1.cnx.network]
end
subgraph L3[Fleet, Hetzner, over the mesh]
CTL[control: node metrics,<br/>journals, host alerts]
WEB[web01: public proxy]
BK[off-site borg destination]
CUST[Customer portal] CUST[Customer portal]
PH[Phones: alerts] PH[Phones: alerts]
end end
@@ -117,146 +167,208 @@ flowchart LR
EP --> MQ EP --> MQ
EU --> MQ EU --> MQ
EG --> MQ EG --> MQ
ST --> APP MQ --> VM
MQ --> DB
MQ --> APP MQ --> APP
APP --> DB APP --> DB
APP --> OBJ ST --> PX --> APP
EP --> OBJ VM --> AM --> PH
DB --> GF APP -.mesh.-> WEB --> CUST
GF --> PH L2 -.mesh.-> CTL
APP --> VPS --> CUST L2 -.mesh.-> BK
``` ```
Three rules keep this simple: field devices talk only to their edge Three rules keep this simple: field devices sit on isolated OT segments
collector (they are on VLANs with no route anywhere else); edge collectors whose only other member is their edge collector; edge collectors publish to
talk only to the broker and object storage; the application and Grafana the core and nothing else; stations and office clients reach the
talk only to the database, the broker and object storage. The VPS reaches application only through the gateway's proxy. Everything that leaves the
the core over the ZeroTier mesh, never over an open port. site rides the ZeroTier mesh, never an open port.
## Machines ## Machines
All NixOS, all in the clan inventory. Hardware classes are planning All NixOS, all entries in the clan inventory. Names follow the fleet
positions; models are chosen at purchase. convention (`gw-<city>-<n>` for gateways, short role names for servers)
with a `vinyl-` prefix so the plant's machines are recognisable in a
multi-site inventory. `<site>` is the city code chosen at site selection;
Mueang Chi is in Lamphun province, so `lpn` is the working assumption and
the gateway is `gw-lpn-1`, giving internal names under `*.lpn1.cnx.network`.
Hardware classes are planning positions; models are chosen at purchase.
| Machine | Role | Hardware class | Where | Clan tags | | Machine | Role | Hardware class | Where |
|---|---|---|---|---| |---|---|---|---|
| `router-1` | Firewall, VLANs, DHCP/DNS, NTP (chrony), dual WAN failover | fanless x86, 4+ NICs | IT room | `core` | | `gw-lpn-1` | Site gateway: `cnx.router` with the plant VLANs, internal proxy for the factory application and Grafana, NTP for the site | Topton 1U N300, 4x 2.5G (fleet standard); a 10G SFP+ variant if the core server link warrants it | IT room |
| `core-1` | PostgreSQL + TimescaleDB, Mosquitto, Telegraf persister, factory app, Grafana, Garage object storage, Pocket ID, Caddy | 1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD pool for objects, dual PSU on UPS | IT room | `core`, `db` | | `vinyl-core` | PostgreSQL, Mosquitto, Telegraf persister, VictoriaMetrics + vmalert + Grafana, Alertmanager, factory application, ZFS datasets for files (audio captures, inspection images, metrology raw data, CoA scans, client audio) | 1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD mirror or raidz for files, dual PSU on UPS | IT room |
| `backup-1` | Borg server, PostgreSQL streaming replica, second Garage node, ZFS receive target | tower or 2U, large HDD pool | separate room from `core-1` (archive side of the building) | `core`, `backup` | | `vinyl-backup` | Borg server for the plant clients, ZFS receive target for `vinyl-core`'s datasets, PostgreSQL streaming replica, pgBackRest repository | tower or 2U, large HDD pool | separate room from `vinyl-core` (archive side of the building) |
| `vps-1` | ZeroTier moon (public IP), off-site Borg destination, third Garage node, reverse proxy for the customer portal | small cloud VPS with block storage, outside Thailand or in Bangkok, decision open | cloud | `offsite` | | `vinyl-edge-press` | Telegraf (OPC UA / S7 to presses and extruders, Modbus to hydraulics and label oven), vision result receiver, SFTP target for inspection images, chrony serving the OT segment, disk-backed MQTT buffer | fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485 | press hall control cabinet |
| `edge-press` | Telegraf (OPC UA / S7 to the presses and extruders, Modbus to hydraulics and label oven), vision result receiver, disk-backed buffer | fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485 | press hall control cabinet | `edge` | | `vinyl-edge-util` | Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS | same class, more RS485 | electrical room or utility yard cabinet |
| `edge-utility` | Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS, water; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS | same class, more RS485 | electrical room or utility yard cabinet | `edge` | | `vinyl-edge-galv` | Telegraf (rectifiers, bath sensors, extraction status) | same class, corrosion-protected enclosure outside the wet area | galvanics service corridor (Expand phase) |
| `edge-galvanics` | Telegraf (rectifiers, bath sensors, extraction status) | same class, corrosion-protected enclosure outside the wet area | galvanics service corridor | `edge` (Expand phase) | | `vinyl-st-cutting` | Session log entry, lacquer registration, scribe ID | mini PC, touch screen, badge reader | cutting room |
| `st-intake` | Office workstation for audio intake and gates (plain browser; not a kiosk) | any NixOS desktop | office | `station` | | `vinyl-st-press1`, `vinyl-st-press2` | Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons | mini PC, touch screen, badge reader, 2D scanner | at each press |
| `st-cutting` | Session log entry, lacquer registration, scribe ID | mini PC, touch screen, badge reader | cutting room | `station` | | `vinyl-st-lab` | PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment | mini PC, RS232/USB to instruments | material lab |
| `st-press-1`, `st-press-2` | Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons | mini PC, touch screen, badge reader, 2D scanner | at each press | `station` | | `vinyl-st-metrology` | SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan | mini PC | metrology bench (Expand phase) |
| `st-lab` | PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment | mini PC, RS232/USB to instruments | material lab | `station` | | `vinyl-st-audioqc` | Needle-drop capture and analysis pipeline | mini PC with audio interface, turntable, phono stage | listening/QC room (Expand phase) |
| `st-metrology` | SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan | mini PC | metrology bench (Expand phase) | `station` | | `vinyl-st-pack1` | Runout code camera read, jacket barcode scan, BOM check, barcode verifier | mini PC, camera, scanner, verifier | packaging |
| `st-audioqc` | Needle-drop capture and analysis pipeline | mini PC with audio interface, turntable, phono stage | listening/QC room (Expand phase) | `station` | | `vinyl-st-ship` | Consignment assembly, carrier label and tracking capture | mini PC, label printer | shipping |
| `st-pack-1` | Runout code camera read, jacket barcode scan, BOM check, barcode verifier | mini PC, camera, scanner, verifier | packaging | `station` | | `vinyl-nvr` | Frigate CCTV recording on the cctv VLAN (optional; a vendor NVR appliance is the alternative) | small server with GPU or Coral accelerator | IT room |
| `st-ship` | Consignment assembly, carrier label and tracking capture | mini PC, label printer | shipping | `station` |
| `nvr-1` | Frigate CCTV recording on its own VLAN (optional; a vendor NVR appliance is the alternative) | small server with GPU or Coral accelerator | IT room | `cctv` |
Stations run a kiosk browser (`cage` plus Firefox or Chromium) pointed at Office audio intake and gate approvals happen from ordinary staff laptops
the factory application; badge readers and scanners are USB HID keyboard on the `lan` VLAN through the same proxy; no dedicated machine. Stations
devices, so there is no driver work. Fifteen machines at launch, about run a kiosk browser (`cage` plus Firefox or Chromium) pointed at the
twenty at maturity; every one is a few lines in the inventory. application; badge readers and scanners are USB HID keyboard devices, so
there is no driver work. Fifteen plant machines at launch, about twenty at
maturity; every one is a directory and a line in the inventory.
## Networks ## Networks
| VLAN | Members | Rule | The plant is site `<siteId>` (next free number in the fleet's site table;
|---|---|---| `cnx` is 1 and the public-WiFi site is 2, so 3 is the working assumption)
| `ot-press` | press and extruder PLCs and HMIs, vision controller, `edge-press` | no internet, no route to other VLANs; `edge-press` is the only gateway | and owns `10.<siteId>.0.0/16`. VLANs follow the fleet convention; the
| `ot-utility` | boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, `edge-utility` | same | ids above 40 are proposals.
| `ot-galvanics` | rectifiers, bath transmitters, `edge-galvanics` | same |
| `sensors` | RS485 loops terminate on the edge collectors, so zone sensors have no IP presence; any IP sensor that is added lives here | no internet |
| `servers` | `core-1`, `backup-1`, `router-1` management | reachable from `stations`, `office`, edge collectors and the mesh |
| `stations` | `st-*` kiosks, instrument PCs (Keyence, vision setup laptop) | reach `core-1` only |
| `office` | staff laptops, printers | internet, `core-1` |
| `cctv` | cameras, `nvr-1` | no internet; NVR reachable from `office` by role |
| `guest` | visitors, vendor laptops | internet only |
| `mgmt` | switch and IPMI management interfaces | `router-1` and `core-1` admins only |
Vendor remote support (Newbilt and Pheenix both offer remote monitoring) goes | VLAN | id | Members | `allowWan` | Rule |
through a jump account on `router-1` over the ZeroTier mesh, time-boxed and |---|---|---|---|---|
logged, with a port forward to the one PLC HMI they need. No machine ever | `mgmt` | 10 | switches, APs, Omada, IPMI, admin laptops on site | yes | fleet convention: trusted, reaches everything |
gets a vendor's own cellular router or cloud agent on the OT VLAN. | `lan` | 20 | office laptops, printers | yes | proxy to factory app and Grafana, internet |
| `guest` | 30 | visitors, vendor laptops | yes | internet only |
| `servers` | 100 | `vinyl-core`, `vinyl-backup`, the servers-side NIC of every edge collector | yes (mesh, nix, journald, off-site backup) | no inbound from other VLANs; the gateway's own Caddy dials the app here |
| `stations` | 110 | `vinyl-st-*`, instrument PCs (Keyence, vision setup laptop), label printer | yes (mesh, deploys, journald) | proxy to the factory app only; `dhcp.reservations` for every station |
| `cctv` | 120 | cameras, NVR | no | reachable from `mgmt`; NVR UI via the proxy if wanted |
| `ot-press` | 210 | press and extruder PLCs and HMIs, vision controller, OT NIC of `vinyl-edge-press` | no | isolated segment, see below |
| `ot-util` | 220 | boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, OT NIC of `vinyl-edge-util` | no | isolated segment |
| `ot-galv` | 230 | rectifiers, bath transmitters, OT NIC of `vinyl-edge-galv` | no | isolated segment |
Dual WAN: fibre plus a 4G/5G modem on `router-1`, failover only. The plant **OT segments are islands, not routes.** Each edge collector is dual-homed:
does not depend on the internet to run: everything in the tables above is one NIC on `servers` (an access port on the cabinet switch), one NIC on
on site except `vps-1`, and the only things that stop when the internet its OT segment. The OT segment lives on the cabinet's industrial switch;
drops are the customer portal and off-site backup. if it must span cabinets it rides the building trunk as a VLAN with
`dhcp.enable = false` and `allowWan = false`, and the router's default
no-inter-VLAN policy keeps it closed. Either way the gateway forwards
nothing into or out of an OT segment; the collector is the only bridge,
and it only publishes. Zone sensors on RS485 loops terminate directly on
the collectors and have no IP presence at all. Cheap ESP32/MQTT sensor
nodes are acceptable for extra density in non-critical zones (on
`stations`, publishing to the core), never for alarmed zones.
## Clan inventory (sketch) **Stations never route to `servers`.** They reach
`https://factory.lpn1.cnx.network`, which Blocky resolves to the gateway
and Caddy forwards to `vinyl-core` on the servers VLAN, with the real
wildcard certificate the router module already issues. `proxy.allowVlans`
gains `stations`. The same proxy serves `grafana.lpn1.cnx.network` to the
office. This keeps the fleet's trust model intact: no VLAN talks to
another VLAN; the router talks to its backends.
The plant clan in outline. Option names follow the clan release current at Router module extensions the plant needs (small, and useful for the other
writing (26.05); check them against the pinned release before use, and sites too):
expect the monitoring service to change, since it is still marked
experimental upstream.
```nix 1. **NTP served by the gateway** to every VLAN (chrony `allow` for the
{ site's `/16`, UDP 123 in the per-VLAN firewall set) so stations and
meta.name = "cnx-vinyl"; the NVR sync locally; today the gateway is only a chrony client.
meta.domain = "cnx"; # mesh-only TLD; hosts resolve as <name>.cnx 2. **Static-only VLANs**: a `dhcp.enable = false` VLAN should still
accept `reservations` semantics for documentation, or simply be
documented as address-by-hand; either is fine, decide when writing it.
3. **Second WAN**: a 4G/5G modem as failover for the PPPoE session
(`wan.backupInterface` with a metric and a health check). Not needed
for launch because the plant runs without the internet; needed before
the customer portal is promised as always-on.
inventory.machines = { Vendor remote support (Newbilt and Pheenix both offer remote monitoring)
router-1 = { tags = [ "core" ]; }; goes through an admin's SSH port-forward across the mesh into the relevant
core-1 = { tags = [ "core" "db" "monitoring-server" ]; }; edge collector and from there to the one PLC HMI they need, time-boxed and
backup-1 = { tags = [ "core" "backup" ]; }; logged. No machine ever gets a vendor's own cellular router or cloud agent
vps-1 = { tags = [ "offsite" ]; }; on an OT segment.
edge-press = { tags = [ "edge" ]; };
edge-utility = { tags = [ "edge" ]; };
edge-galvanics = { tags = [ "edge" ]; };
st-cutting = { tags = [ "station" ]; };
st-press-1 = { tags = [ "station" ]; };
st-lab = { tags = [ "station" ]; };
st-pack-1 = { tags = [ "station" ]; };
st-ship = { tags = [ "station" ]; };
};
inventory.instances = { ## What changes in the clan repo
sshd.roles.server.tags.all = { };
admin.roles.default.tags.all = { }; # admin users and keys
zerotier = { Following the "Adding a site" procedure in the clan runbook, plus the
roles.controller.machines.core-1 = { }; plant's own modules. Everything below is one pull request per machine
roles.moon.machines.vps-1.settings.stableEndpoints = [ "<public-ip>/9993" ]; class, in this order.
roles.peer.tags.all = { };
```
clan.nix
fleet: + gw-lpn-1, vinyl-core, vinyl-backup, vinyl-edge-press,
vinyl-edge-util, vinyl-st-press1, vinyl-st-lab,
vinyl-st-pack1, vinyl-st-ship (then the rest as they come)
borgbackup:
roles.server.machines.vinyl-backup = { };
roles.client.machines.vinyl-core.settings.destinations = {
plant = { repo = "borg@vinyl-backup:vinyl-core"; };
offsite = { repo = "<storage-box or bk host>:vinyl-core"; };
}; };
roles.client.machines.vinyl-edge-press = { ... plant only ... };
# side benefit: control can now back up its VictoriaMetrics TSDB to
# vinyl-backup, closing the open item in the clan TODO
(no `internet` entries: plant machines sit behind PPPoE like gw-cnx-1
and are reached over the mesh)
borgbackup = { modules/mesh-hosts.nix + every plant machine in the name list
roles.server.machines.backup-1 = { }; modules/monitoring/server.nix
roles.server.machines.vps-1 = { }; + node_exporter targets for every plant machine
roles.client.tags.core = { }; + a second Grafana datasource "vinyl-vm" pointing
roles.client.tags.edge = { }; at [vinyl-core mesh]:8428 so fleet dashboards
}; can show plant series
modules/monitoring/alerts.nix
unchanged; host rules already cover new targets
machines/ns1/configuration.nix
+ dns-acme-gw-lpn-1-knot generator, keyFiles entry,
acl_acme_gw_lpn_1 scoped to _acme-challenge.lpn1
(copy the gw-cnx-1 blocks)
modules/dns/acme-gw-secret.nix
reused as-is with "gw-lpn-1"
modules/web-proxy.nix + virtualHosts."portal.<domain>" -> [vinyl-core mesh]:8080
(second cert + scoped TSIG key if <domain> is
not cnx.network, following the acme_web01 pattern)
monitoring = { machines/gw-lpn-1/ copy of gw-cnx-1: site "lpn", siteId 3, the VLAN
roles.server.machines.core-1.settings.grafana.enable = true; table above, proxy.services.factory and .grafana,
roles.client.tags.all = { }; proxy.allowVlans = [ "mgmt" "lan" "stations" ]
}; machines/vinyl-core/ imports modules/vinyl/{core,metrics,factory-app,
}; alertmanager}.nix + monitoring/exporters.nix
machines/vinyl-backup/ imports modules/vinyl/backup.nix + exporters.nix
machines/vinyl-edge-*/ imports modules/vinyl/edge.nix with the cabinet's
Telegraf inputs and the OT NIC config
machines/vinyl-st-*/ imports modules/vinyl/station.nix with the kiosk URL
# Our own NixOS modules, applied by tag or by machine: modules/vinyl/core.nix PostgreSQL (ZFS dataset, pg_hba for the app and
# modules/edge-collector.nix Telegraf with the tag lists for that cabinet, the replica), Mosquitto (per-client passwords from
# Mosquitto bridge with disk buffer vars generators), Telegraf persister (MQTT in,
# modules/core-data.nix PostgreSQL + TimescaleDB, Mosquitto, Telegraf VictoriaMetrics out), ZFS datasets and sanoid
# persister, Garage, Grafana, pgBackRest modules/vinyl/metrics.nix VictoriaMetrics + vmalert + Grafana for the plant,
# modules/factory-app.nix the application service and its migrations a sibling of monitoring/server.nix with the
# modules/station-kiosk.nix cage + browser, HID readers, auto-login process rules and dashboards under
# modules/router.nix nftables zoning, VLAN interfaces, chrony, modules/vinyl/dashboards/
# dual-WAN failover modules/vinyl/alertmanager.nix
} receivers: Telegram or LINE for the shift group,
SMS gateway for the critical set, email digest;
tokens from vars generators
modules/vinyl/factory-app.nix
the cnx-factory package as a flake input (like
nixos-mailserver is today), systemd service,
migrations on start, DB password from vars,
OIDC client secret from vars
modules/vinyl/edge.nix Telegraf with per-machine input lists, MQTT output
with disk buffer, chrony `allow` on the OT NIC,
networkd config for the two NICs, SFTP chroot for
vision images (press collector only)
modules/vinyl/station.nix cage + browser kiosk, auto-login, HID readers,
SMB share and ingestion watcher where needed
modules/vinyl/backup.nix borg server role settings, syncoid receive,
PostgreSQL replica, pgBackRest repo
modules/router/… the three extensions above
docs/src/vinyl.md the plant chapter of the runbook, + SUMMARY.md
``` ```
What clan provides out of the box and we do not write ourselves: SSH and Operational notes that already bit the fleet and apply here: `git add` a
admin user management, the ZeroTier mesh (controller on `core-1`, the VPS new `machines/` directory before evaluating (untracked files are invisible
as a publicly reachable moon so the plant and remote consultants find each to the flake); run `clan vars generate <machine>` (YubiKey) before the
other behind NAT), Borg backups with generated keys, host metrics and first install and again after adding a generator; check `facter.json` for
journal shipping with Grafana on the server role, and sops-encrypted the real NIC names after install; the `clan` wrapper in the dev shell
secrets per machine. What we write: the five modules listed in the sketch updates machines sequentially so YubiKey prompts are not buried.
and the factory application itself.
Repository layout, decided: infrastructure lives in `cnx-network-clan`;
the factory application gets its own repository `cnx-factory` in the B4L
organisation and is consumed as a flake input; this repository stays
documents and links to both. That satisfies the README's promise that the
project repository "hosts the software" without mixing Nix, Rust and
planning prose in one history.
## Data acquisition, source by source ## Data acquisition, source by source
@@ -268,12 +380,12 @@ is bought, not retrofitted. Rates are planning values.
| Source | Signals | Interface | Rate | Requirement to write into the RFQ | | Source | Signals | Interface | Rate | Requirement to write into the RFQ |
|---|---|---|---|---| |---|---|---|---|---|
| Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA) | cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes | OPC UA server on the PLC, read by Telegraf `inputs.opcua` | per cycle plus 1 Hz | OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty | | Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA) | cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes | OPC UA server on the PLC, read by Telegraf `inputs.opcua` | per cycle plus 1 Hz | OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty, PLC clock accepting NTP |
| Newbilt HRP 12 semi-automatic and VEX 50 extruder (PLC type unconfirmed) | as above where the PLC has them; on a relay-logic machine at least press closed/open, steam on/off, cycle count | OPC UA or Siemens S7 (`inputs.s7comm`); fallback: our own 4 to 20 mA sensors on a Modbus TCP I/O module and a cycle count from the mold-closed limit switch | per cycle plus 1 Hz | state what the PLC is and which of the two paths applies; the fallback sensor set is priced separately | | Newbilt HRP 12 semi-automatic and VEX 50 extruder (PLC type unconfirmed) | as above where the PLC has them; on a relay-logic machine at least press closed/open, steam on/off, cycle count | OPC UA or Siemens S7 (`inputs.s7comm`); fallback: our own 4 to 20 mA sensors on a Modbus TCP I/O module and a cycle count from the mold-closed limit switch | per cycle plus 1 Hz | state what the PLC is and which of the two paths applies; the fallback sensor set is priced separately |
| Pheenix AD12 (Beckhoff PLC, remote monitoring supported) | same tag family as the Classic | OPC UA (Beckhoff TF6100) or ADS; `inputs.opcua` | per cycle plus 1 Hz | OPC UA license included, tag list delivered, remote monitoring routed through our jump host | | Pheenix AD12 (Beckhoff PLC, remote monitoring supported) | same tag family as the Classic | OPC UA (Beckhoff TF6100) or ADS; `inputs.opcua` | per cycle plus 1 Hz | OPC UA license included, tag list delivered, remote monitoring routed through our mesh, not the vendor's |
| Hydraulic aggregate | system pressure, oil temperature, tank level, filter differential pressure | via the press PLC if exposed, else own transmitters on the I/O module | 1 Hz | ask whether the aggregate's sensors are on the PLC | | Hydraulic aggregate | system pressure, oil temperature, tank level, filter differential pressure | via the press PLC if exposed, else own transmitters on the I/O module | 1 Hz | ask whether the aggregate's sensors are on the PLC |
| Label dry oven | chamber temperature, timer state | Modbus if the controller has it, else a thermocouple on the I/O module | 1/min | | | Label dry oven | chamber temperature, timer state | Modbus if the controller has it, else a thermocouple on the I/O module | 1/min | |
| PVC grinder and regrind scale | run state, batch weight | scale with RS232 to `st-press`, run contact to I/O | per batch | scale must have a serial output | | PVC grinder and regrind scale | run state, batch weight | scale with RS232 to `vinyl-st-press1`, run contact to I/O | per batch | scale must have a serial output |
### Utilities and energy ### Utilities and energy
@@ -305,10 +417,9 @@ is bought, not retrofitted. Rates are planning values.
Industrial RS485 Modbus RTU transmitters on wired loops to the nearest edge Industrial RS485 Modbus RTU transmitters on wired loops to the nearest edge
collector, one reading per minute, with a reference-grade PM instrument in collector, one reading per minute, with a reference-grade PM instrument in
the archive or cutting room to check the cheaper sensors against. Cheap the archive or cutting room to check the cheaper sensors against. Every
ESP32/MQTT sensor nodes are acceptable for extra density in non-critical zone alarm has two thresholds (warning, action) and a rate-of-change alarm
zones, never for the alarmed ones. Every zone alarm has two thresholds for the archive.
(warning, action) and a rate-of-change alarm for the archive.
### Galvanics (Expand phase) ### Galvanics (Expand phase)
@@ -324,9 +435,9 @@ results are entered at a station against the bath ID.
Access-control door events (badge ID, door, direction, time) from the Access-control door events (badge ID, door, direction, time) from the
controller's API or export; fire alarm panel zone and fault contacts, BESS controller's API or export; fire alarm panel zone and fault contacts, BESS
off-gas and early-smoke detection, UPS on-battery and low-battery (NUT on off-gas and early-smoke detection, UPS on-battery and low-battery (NUT on
`core-1` talking to the UPS) as dry contacts on `edge-utility`. CCTV stays `vinyl-core` talking to the UPS) as dry contacts on `vinyl-edge-util`.
on its own VLAN with its own retention; it is never joined to production CCTV stays on its own VLAN with its own retention; it is never joined to
data. production data.
### What produces no telemetry ### What produces no telemetry
@@ -340,21 +451,22 @@ logged by the sensors above.
| Instrument | How results reach the database | Key | Open integration item | | Instrument | How results reach the database | Key | Open integration item |
|---|---|---|---| |---|---|---|---|
| Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration) | vision controller sends one result string per record over TCP to a receiver on `edge-press`; images pushed by FTP/SFTP to object storage | press ID + cycle counter, passed from the PLC to the vision trigger | the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test | | Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration) | vision controller sends one result string per record over TCP to a receiver on `vinyl-edge-press`; images pushed by SFTP to the collector, synced to the core's image dataset | press ID + cycle counter, passed from the PLC to the vision trigger | the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test |
| Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer) | vendor Windows PCs export CSV and raw 3D to an SMB share on `st-metrology`; a watcher parses and posts | back-side Data Matrix serial, scanned at the bench before measurement | export format per instrument confirmed with Keyence Thailand | | Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer) | vendor Windows PCs export CSV and raw 3D to an SMB share on `vinyl-st-metrology`; a watcher parses and posts | back-side Data Matrix serial, scanned at the bench before measurement | export format per instrument confirmed with Keyence Thailand |
| PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR) | instruments with RS232/USB export are read at `st-lab`; the rest are typed in with the printout photographed; CoA PDF attached | compound lot ID (printed label scanned at intake) | which benchtop models have serial export | | PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR) | instruments with RS232/USB export are read at `vinyl-st-lab`; the rest are typed in with the printout photographed; CoA PDF attached | compound lot ID (printed label scanned at intake) | which benchtop models have serial export |
| Audio QC station | capture service on `st-audioqc` records the needle-drop (PipeWire, FLAC to object storage); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted | record key (stamper + press + cycle + side), typed or picked from the run list | first-of-kind pipeline, developed in Expand phase against launch-phase recordings | | Audio QC station | capture service on `vinyl-st-audioqc` records the needle-drop (PipeWire, FLAC to the core's capture dataset); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted | record key (stamper + press + cycle + side), typed or picked from the run list | first-of-kind pipeline, developed in Expand phase against launch-phase recordings |
| Packaging assembly check | camera with darkfield lighting reads the scribed dead-wax code (a handheld scanner will not read a scribed code on black vinyl reliably); handheld scanner reads the jacket barcode; barcode verifier grades it | order + record key | confirm dead-wax code readability with a sample before the packaging station is designed | | Packaging assembly check | camera with darkfield lighting reads the scribed dead-wax code (a handheld scanner will not read a scribed code on black vinyl reliably); handheld scanner reads the jacket barcode; barcode verifier grades it | order + record key | confirm dead-wax code readability with a sample before the packaging station is designed |
| Label print inspection | same vision family as the record label check; result per label lot | label lot ID | shared with the record inspection budget | | Label print inspection | same vision family as the record label check; result per label lot | label lot ID | shared with the record inspection budget |
## The factory application ## The factory application
Working name `cnx-factory`. Server-rendered web application in Rust (axum, Working name `cnx-factory`, its own repository. Server-rendered web
SQLx, minijinja templates, HTMX) on PostgreSQL, the stack B4L already uses application in Rust (axum, SQLx, minijinja templates, HTMX) on PostgreSQL,
for its other in-house tooling, with the same append-only ledger and the stack B4L already uses for its other in-house tooling, with the same
hash-chained audit log pattern. It subscribes to the MQTT event topics append-only ledger and hash-chained audit log pattern. It subscribes to
(cycle complete, machine alarm, instrument result) and owns every write to the MQTT event topics (cycle complete, machine alarm, instrument result)
the identity tables; Telegraf owns the writes to the time-series tables. and owns every write to the identity tables; Telegraf owns the writes to
VictoriaMetrics. Packaged as a flake so the clan repo pins it as an input.
Modules, in build order: Modules, in build order:
@@ -377,8 +489,9 @@ Modules, in build order:
lots with incoming QC. lots with incoming QC.
7. **Press runs and cycles**: a run binds order, stamper pair, press, 7. **Press runs and cycles**: a run binds order, stamper pair, press,
operator, shift, PVC lot in the hopper, regrind batch, label lot; each operator, shift, PVC lot in the hopper, regrind batch, label lot; each
cycle event from the PLC becomes a row keyed by press + counter; the cycle event from the PLC becomes a row keyed by press + counter with
record key is derived from it per side. the key press values and the zone environment snapshot; the record key
is derived from it per side.
8. **QC verdicts**: optical per record, audio per sampled record, lab per 8. **QC verdicts**: optical per record, audio per sampled record, lab per
lot, packaging per unit; reject reasons; test-pressing hold state. lot, packaging per unit; reject reasons; test-pressing hold state.
9. **Finishing, packaging, fulfillment**: assembly checks, consignment 9. **Finishing, packaging, fulfillment**: assembly checks, consignment
@@ -388,16 +501,15 @@ Modules, in build order:
one customer sees only its own orders; PDF export of the trace. one customer sees only its own orders; PDF export of the trace.
11. **Reports and signals**: yield per stamper, reject rate drift, audio 11. **Reports and signals**: yield per stamper, reject rate drift, audio
score trend versus play count (the automatic retirement signal), score trend versus play count (the automatic retirement signal),
energy per record, environment excursions overlaid on runs. energy per record, environment excursions overlaid on runs (the
curves fetched from VictoriaMetrics for the run's window).
Core entities: `customer`, `order`, `release`, `side`, `audio_asset`, Core entities: `customer`, `order`, `release`, `side`, `audio_asset`,
`gate_approval`, `session`, `lacquer`, `father`, `mother`, `stamper`, `gate_approval`, `session`, `lacquer`, `father`, `mother`, `stamper`,
`stamper_measurement`, `press`, `press_run`, `press_cycle`, `record`, `stamper_measurement`, `press`, `press_run`, `press_cycle`, `record`,
`optical_result`, `audio_result`, `pvc_lot`, `regrind_batch`, `label_lot`, `optical_result`, `audio_result`, `pvc_lot`, `regrind_batch`, `label_lot`,
`carton_lot`, `lab_result`, `packaging_check`, `shipment`, `operator`, `carton_lot`, `lab_result`, `packaging_check`, `shipment`, `operator`,
`shift`, `event` (the hash chain). Time-series hypertables: `telemetry` `shift`, `event` (the hash chain).
(machine, tag, value), `environment` (zone, metric, value), `energy`
(meter, kWh, kW), `water`.
Stations use the same application in kiosk mode: an operator taps a badge Stations use the same application in kiosk mode: an operator taps a badge
(the same card the access-control system issues, so one ID) and scans (the same card the access-control system issues, so one ID) and scans
@@ -405,55 +517,64 @@ whatever the step asks for. No forms longer than one screen.
## Observability and alerting ## Observability and alerting
Two tiers, one Grafana: Two tiers, split by where the data is:
- **Process alarms** from TimescaleDB: zone environment excursions and - **Process, on site** (`vinyl-core`: VictoriaMetrics, vmalert,
rate of change, boiler pressure and level, chiller return temperature, Grafana, Alertmanager): zone environment excursions and rate of change,
compressed-air pressure, hydraulic pressure, reject rate per run above boiler pressure and level, chiller return temperature, compressed-air
baseline, stamper approaching its play-count limit, PV or BESS faults, pressure, hydraulic pressure, reject rate per run above baseline,
fire and BESS detection contacts, UPS on battery. stamper approaching its play-count limit, PV or BESS faults, fire and
- **Infrastructure alarms** from the clan monitoring stack: host down, BESS detection contacts, UPS on battery. Works with the WAN down.
disk, service failed, backup missed, certificate expiry. - **Hosts, on the fleet** (`control`): node_exporter and journald from
every plant machine over the mesh, the existing `ScrapeTargetDown`,
`RootFilesystemFull`, `BackupJobFailed`, `BackupStale` rules apply to
them without changes.
- **Data-silence alarms** in both tiers: a collector that has published - **Data-silence alarms** in both tiers: a collector that has published
nothing for five minutes is an alarm, whatever the last value was. nothing for five minutes is an alarm, whatever the last value was.
Routing: Telegram or the LINE Messaging API for the shift group, email for Routing: Telegram or the LINE Messaging API for the shift group, email
the daily digest, SMS through a Thai gateway for the critical set (boiler, digest daily, SMS through a Thai gateway for the critical set (boiler,
fire, BESS, archive climate) with an on-call rota. The animated fire, BESS, archive climate) with an on-call rota. `control`'s vmalert can
drop `notifier.blackhole` and point at the same Alertmanager over the
mesh, which closes the fleet's own alert-delivery item. The animated
[plant operation page](plant-operation.html) is the design target for the [plant operation page](plant-operation.html) is the design target for the
live overview: once the telemetry tables exist it is fed from Grafana or live overview: once the series exist it is fed from VictoriaMetrics
directly from the database instead of its simulation. instead of its simulation.
## Backup, retention and recovery ## Backup, retention and recovery
- **Hourly ZFS snapshots** on `core-1` (sanoid), replicated to `backup-1` - **Hourly ZFS snapshots** on `vinyl-core` (sanoid), replicated to
(syncoid). `vinyl-backup` (syncoid) for the file datasets and the PostgreSQL data
- **Nightly Borg** from every `core` and `edge` machine to `backup-1` and directory.
to `vps-1` (clan borgbackup, two destinations). - **Nightly Borg** (clan `borgbackup`) from `vinyl-core` and the edge
collectors to `vinyl-backup`, and from `vinyl-core` to the off-site
destination as a second entry in `destinations`; repokey encryption as
the fleet already does.
- **PostgreSQL point-in-time recovery**: pgBackRest with WAL archiving to - **PostgreSQL point-in-time recovery**: pgBackRest with WAL archiving to
object storage, replicated to the off-site node; a streaming replica on `vinyl-backup`, its repository included in the off-site Borg set; a
`backup-1` for fast failover. streaming replica on `vinyl-backup` for fast failover.
- **Object storage** (Garage) with three nodes: `core-1`, `backup-1`, - **VictoriaMetrics**: `vmbackup` snapshots to `vinyl-backup` nightly; the
`vps-1`; audio captures, inspection images, metrology raw data and CoA fleet's own open item (backing up `control`'s TSDB) gets a destination
scans survive the loss of any one node. at the same time.
- **Configuration**: the clan repo in Gitea, mirrored off-site; any machine - **Configuration**: the clan repo and `cnx-factory` in Gitea; any machine
is rebuilt from it. is rebuilt from them.
- **Restore drill quarterly**: restore the database to a scratch VM and - **Restore drill quarterly**: restore the database to a scratch VM and
open a random order's trace in the portal. A backup that has never been open a random order's trace in the portal. A backup that has never been
restored does not count. restored does not count. The `BackupStale` rule already watches the
timers.
| Data | Retention | | Data | Retention |
|---|---| |---|---|
| genealogy, orders, gates, QC verdicts, event log | life of the customer relationship, never deleted while the metal parts exist | | genealogy, orders, gates, QC verdicts, event log | life of the customer relationship, never deleted while the metal parts exist |
| per-cycle press data | forever (small) | | per-cycle press rows | forever (small) |
| 1 Hz machine telemetry | raw 2 years, then 1-minute aggregates forever | | 1 Hz machine telemetry | 2 years in VictoriaMetrics (`retentionPeriod`); per-cycle rows carry the summary forever |
| zone environment | raw 5 years | | zone environment | 5 years (separate VM retention filter, or a second small instance) |
| energy and water | forever (small) | | energy and water | forever (small; also summarised per shift into PostgreSQL) |
| audio captures and inspection images | 2 years, longer for any record with a complaint | | audio captures and inspection images | 2 years, longer for any record with a complaint |
| client audio | per contract; default deleted 12 months after final delivery, or on request | | client audio | per contract; default deleted 12 months after final delivery, or on request |
| access-control events | 1 year | | access-control events | 1 year |
| CCTV | 30 days | | CCTV | 30 days |
| host logs and metrics | 90 days | | host logs and metrics | fleet defaults: 30 d logs, 180 d metrics on `control` |
PDPA applies to badge events, CCTV and customer contact data: a written PDPA applies to badge events, CCTV and customer contact data: a written
retention policy, role-based access, and export/deletion on request. The retention policy, role-based access, and export/deletion on request. The
@@ -461,18 +582,24 @@ customer portal shows operator roles, never operator names.
## Security ## Security
- OT VLANs have no internet and no inbound routes; edge collectors are the - OT segments have no gateway interface and no internet; edge collectors
only bridge and they only publish. are the only bridge and they only publish.
- Staff sign-on through Pocket ID (OIDC, passkeys) with roles: admin, - Stations and office reach the application only through the gateway's
engineer, operator, sales, customer. Stations authenticate by badge for Caddy, with the wildcard certificate the fleet already issues; the
attribution, not for privilege. servers VLAN accepts nothing from other VLANs.
- Secrets only in clan vars; nothing in plain text in the repo. - Staff sign-on through an OIDC provider on `vinyl-core` (Pocket ID,
passkeys; the fleet has no identity provider yet, so this is the first)
with roles: admin, engineer, operator, sales, customer. Stations
authenticate by badge for attribution, not for privilege.
- Secrets only in clan vars (YubiKey-gated, `admins` group); nothing in
plain text in either repo.
- Client audio: separate encrypted ZFS dataset, SFTP or portal upload with - Client audio: separate encrypted ZFS dataset, SFTP or portal upload with
checksums, every read logged, a written policy sales can show a label. checksums, every read logged, a written policy sales can show a label.
- UPS on `core-1`, `router-1`, the switches, the edge collectors and the - UPS on `vinyl-core`, `gw-lpn-1`, the switches, the edge collectors and
press PLCs (already required by facility security); NUT triggers a clean the press PLCs (already required by facility security); NUT triggers a
shutdown before the battery is exhausted. clean shutdown before the battery is exhausted.
- Signed commits on the clan repo; deployments only from a reviewed branch. - Admin access rides the mesh, as for every fleet machine; public SSH is
never opened.
## Procurement clauses ## Procurement clauses
@@ -481,7 +608,7 @@ To go into every RFQ that leaves from [roadmap](roadmap.md) items 4 to 8:
1. **Presses (Newbilt, Pheenix)**: OPC UA (or S7/ADS) read access licensed 1. **Presses (Newbilt, Pheenix)**: OPC UA (or S7/ADS) read access licensed
and enabled; the full tag list with units and the cycle counter as a and enabled; the full tag list with units and the cycle counter as a
tag; a defined hand-off of the cycle counter to the inspection trigger; tag; a defined hand-off of the cycle counter to the inspection trigger;
remote support through our jump host; no cloud agent as a warranty remote support through our mesh; no cloud agent as a warranty
condition; PLC clock accepting NTP. condition; PLC clock accepting NTP.
2. **Boiler, chiller, compressor, heat pump (Thai vendors)**: Modbus RTU or 2. **Boiler, chiller, compressor, heat pump (Thai vendors)**: Modbus RTU or
TCP with the register map delivered; alarm contacts wired to a terminal TCP with the register map delivered; alarm contacts wired to a terminal
@@ -491,13 +618,16 @@ To go into every RFQ that leaves from [roadmap](roadmap.md) items 4 to 8:
loops terminated in the edge cabinets. loops terminated in the edge cabinets.
4. **Electrical contractor**: Modbus energy meters on the feeders listed 4. **Electrical contractor**: Modbus energy meters on the feeders listed
above; RS485 to the inverters and BESS BMS without cloud dongles; above; RS485 to the inverters and BESS BMS without cloud dongles;
structured cabling and a VLAN-capable switch in every cabinet. structured cabling, a VLAN-capable industrial switch in every cabinet,
and Omada-managed switches and APs for the building so the gateway's
existing Omada controller covers the site.
5. **Galvanics (Digital Matrix)**: what the rectifiers and control rings 5. **Galvanics (Digital Matrix)**: what the rectifiers and control rings
expose, and how; Modbus preferred. expose, and how; Modbus preferred.
6. **Metrology (Keyence Thailand)**: confirmed CSV/3D export paths and 6. **Metrology (Keyence Thailand)**: confirmed CSV/3D export paths and
whether the instrument software can write to a network share unattended. whether the instrument software can write to a network share unattended.
7. **Vision integrator**: result string format, image push, cycle counter 7. **Vision integrator**: result string format, image push over SFTP,
correlation, and an acceptance test defined in records not in features. cycle counter correlation, and an acceptance test defined in records not
in features.
8. **Access control and fire**: API or export for door events; dry 8. **Access control and fire**: API or export for door events; dry
contacts for every alarm class; BESS detection outputs. contacts for every alarm class; BESS detection outputs.
@@ -509,23 +639,26 @@ Aligned with the [roadmap](roadmap.md) phases.
- Put the procurement clauses above into the Newbilt re-quote, the - Put the procurement clauses above into the Newbilt re-quote, the
Pheenix, Digital Matrix and Thai utility RFQs. Pheenix, Digital Matrix and Thai utility RFQs.
- Create the clan repo and a development clan of three small machines at - In the clan repo: reserve the site id and VLAN table, write
the office: a `core-1` stand-in, one edge collector with a Modbus and `modules/vinyl/{core,metrics,edge,station}.nix`, and stand up a dev
OPC UA simulator, one station. This proves the tooling before a single trio at the Chiang Mai site behind `gw-cnx-1` (a `vinyl-core` stand-in,
plant machine exists. one edge collector with a Modbus and OPC UA simulator, one station).
- Build the factory application v0 against a press simulator (the state This proves the modules, the mesh path and the proxy before a single
machine in the plant operation page is the seed: same states, same plant machine exists, and the machines move to the plant when it does.
cycle timing) so orders, gates, genealogy and press cycles can be - Land the three router-module extensions on `gw-cnx-1` first, where they
exercised end to end without hardware. are also useful.
- Decide the repository layout: this repository stays documents; the clan - Start `cnx-factory` v0 against a press simulator (the state machine in
flake and the application get their own repositories in the same Gitea the plant operation page is the seed: same states, same cycle timing) so
organisation, linked from here (see open questions). orders, gates, genealogy and press cycles can be exercised end to end
without hardware; package it as a flake from the first commit.
- Wire the Alertmanager and one real receiver; point `control`'s vmalert
at it.
**Build and commission (roadmap Phase 3).** **Build and commission (roadmap Phase 3).**
- Network, `router-1`, `core-1`, `backup-1` and `vps-1` installed before - `gw-lpn-1`, switches, `vinyl-core` and `vinyl-backup` installed before
the machinery arrives, so vendor commissioning engineers plug into a the machinery arrives, so vendor commissioning engineers plug into a
finished network. finished network; ns1 gets the gateway's ACME key at the same time.
- Zone sensor loops, meters and edge cabinets installed by the electrical - Zone sensor loops, meters and edge cabinets installed by the electrical
and HVAC contractors during fit-out; environment logging is live before and HVAC contractors during fit-out; environment logging is live before
pressing starts, which gives a baseline. pressing starts, which gives a baseline.
@@ -535,18 +668,19 @@ Aligned with the [roadmap](roadmap.md) phases.
- Application v1 live: intake, gates, bridge-stamper genealogy, press runs - Application v1 live: intake, gates, bridge-stamper genealogy, press runs
and cycles, PVC lot gate, packaging scan, shipments; stations at press, and cycles, PVC lot gate, packaging scan, shipments; stations at press,
lab, packaging and shipping. lab, packaging and shipping.
- Off-site backup destination live and a first restore drill done.
**Launch (roadmap Phase 4).** **Launch (roadmap Phase 4).**
- Optical inspection integrated and correlated; reject reasons flowing. - Optical inspection integrated and correlated; reject reasons flowing.
- Customer portal opened to the first labels. - Customer portal opened to the first labels through `web01`.
- Energy and environment dashboards used for the first PV and HVAC - Energy and environment dashboards used for the first PV and HVAC
tuning. tuning.
**Expand (roadmap Phase 5).** **Expand (roadmap Phase 5).**
- `edge-galvanics`, bath sensors, rectifier data, metrology ingestion at - `vinyl-edge-galv`, bath sensors, rectifier data, metrology ingestion at
`st-metrology`; genealogy switches from bridged to in-house. `vinyl-st-metrology`; genealogy switches from bridged to in-house.
- Audio QC station and pipeline; retirement signal live. - Audio QC station and pipeline; retirement signal live.
- Automated press line joins the same collector with the same tag family. - Automated press line joins the same collector with the same tag family.
- Biomass boiler on the same Modbus path as its predecessor. - Biomass boiler on the same Modbus path as its predecessor.
@@ -555,21 +689,23 @@ Aligned with the [roadmap](roadmap.md) phases.
Planning estimates, September 2026, excluding the QC instruments (those Planning estimates, September 2026, excluding the QC instruments (those
are in quality automation) and excluding the vendors' OPC UA license are in quality automation) and excluding the vendors' OPC UA license
options, which come back with the quotes. options, which come back with the quotes. Fleet pieces already paid for
(`control`, `ns1`, `ns2`, `web01`, the mesh) cost the plant nothing extra.
| Block | Estimate (USD) | | Block | Estimate (USD) |
|---|---| |---|---|
| `core-1` and `backup-1` servers, UPS, rack | 8,000 to 15,000 | | `gw-lpn-1` (Topton 1U, fleet standard) | 300 to 500 |
| `router-1`, managed switches (industrial in the cabinets), cabling allowance | 4,000 to 8,000 | | `vinyl-core` and `vinyl-backup` servers, UPS, rack | 8,000 to 15,000 |
| Omada switches and APs for the building, industrial switches in the cabinets, cabling allowance | 4,000 to 8,000 |
| 3 edge collectors with I/O modules and enclosures | 3,000 to 6,000 | | 3 edge collectors with I/O modules and enclosures | 3,000 to 6,000 |
| 8 station kiosks with readers, scanners, one barcode verifier, one darkfield camera | 8,000 to 15,000 | | 8 station kiosks with readers, scanners, one barcode verifier, one darkfield camera | 8,000 to 15,000 |
| zone environment sensors (12 to 16 points), room dP, reference PM instrument | 4,000 to 8,000 | | zone environment sensors (12 to 16 points), room dP, reference PM instrument | 4,000 to 8,000 |
| energy meters (10 feeders), water meters, level transmitters | 3,000 to 6,000 | | energy meters (10 feeders), water meters, level transmitters | 3,000 to 6,000 |
| audio QC hardware (turntable, arm, cartridge, phono stage, interface) | 2,000 to 5,000 | | audio QC hardware (turntable, arm, cartridge, phono stage, interface) | 2,000 to 5,000 |
| `vps-1` and off-site storage, per year | 500 to 1,500 | | off-site backup storage, per year | 100 to 500 |
| **Hardware total** | **~32,000 to 64,000** | | **Hardware total** | **~32,000 to 64,000** |
| Application development to v1 (orders through shipments, stations, portal) | 4 to 6 person-months | | `cnx-factory` to v1 (orders through shipments, stations, portal) | 4 to 6 person-months |
| Clan fleet, modules and collectors to commissioning | 1 to 2 person-months | | Clan modules, router extensions, collectors, dashboards and rules to commissioning | 1 to 2 person-months |
| Vision, metrology and audio pipeline integration (Launch and Expand) | 3 to 5 person-months, plus integrator fees in the vision quote | | Vision, metrology and audio pipeline integration (Launch and Expand) | 3 to 5 person-months, plus integrator fees in the vision quote |
Headcount note: [scale](scale.md) lists no IT or automation role in the Headcount note: [scale](scale.md) lists no IT or automation role in the
@@ -578,22 +714,22 @@ Build onward, part-time at launch; add it to the headcount.
## Open questions ## Open questions
- Site code and id: `lpn` / 3 assumed; confirm when the site is chosen and
the fleet site table is updated.
- Semi-automatic press PLC: does the HRP 12 expose data, or is the sensor - Semi-automatic press PLC: does the HRP 12 expose data, or is the sensor
fallback needed? (Ask in the Newbilt re-quote.) fallback needed? (Ask in the Newbilt re-quote.)
- Cycle counter hand-off to the vision system: PLC output, network - Cycle counter hand-off to the vision system: PLC output, network
message, or time correlation at the unloader? Decided with the vision message, or time correlation at the unloader? Decided with the vision
integrator, tested at acceptance. integrator, tested at acceptance.
- HVAC BMS protocol and who owns the zone sensors: HVAC contractor or us. - HVAC BMS protocol and who owns the zone sensors: HVAC contractor or us.
- Where `vps-1` lives: outside Thailand (latency to the mesh, data - Off-site backup destination: Hetzner Storage Box (cheapest) or a storage
residency for customer data) or a Bangkok provider. VPS that is also a mesh peer (uniform tooling). Data residency for
- Portal availability: served through the mesh from `core-1` (down when customer audio may argue for a Bangkok provider instead.
the plant's internet is down) or a read replica on `vps-1` (more moving - Customer portal domain: the plant's brand domain (new zone on ns1/ns2,
parts). Start with the former. second cert on `web01`) or a name under `cnx.network` for launch.
- Portal availability during a plant WAN outage: accept it (the portal is
a view of live data) or add a read replica on the fleet side later.
- Accounting and ERP: out of scope for the application; if an ERP is - Accounting and ERP: out of scope for the application; if an ERP is
adopted later it references the same order IDs. Decide after launch. adopted later it references the same order IDs. Decide after launch.
- Repository layout: documents here, `cnx-infra` (clan) and `cnx-factory`
(application) as separate repositories, or everything in one. The
README currently promises this repository will hold the software; a
link satisfies that.
- Dead-wax code readability on black and coloured vinyl with a camera: - Dead-wax code readability on black and coloured vinyl with a camera:
test with samples before the packaging station is designed. test with samples before the packaging station is designed.