cnx_vinyl/docs/digital-factory.md

Digital Factory: Systems Plan

The traceability spec says what must be logged and the quality automation document says which instruments produce the numbers. This document is the plan for the systems in between: which computers, networks, collectors, databases and applications have to exist so that environmental data, machine data, QC results and the batch genealogy all land in one place, keyed to one record identity, and can be shown to a customer. It is a planning position as of September 2026; every cost figure is an estimate until a quote replaces it.

Standing decision: every service runs on NixOS, and the plant is one more site in the existing B4L fleet, the cnx-network clan. That clan already runs the ZeroTier mesh, authoritative DNS, mail, fleet monitoring, backups, a public reverse proxy and the first site gateway, all declared in one repository and deployed with the clan CLI. The plant adds a gateway, a core server, a backup server, edge collectors and station kiosks to that inventory and reuses everything the fleet already provides. Nothing is built twice. Vendor instrument PCs (Keyence, vision controllers, PLC HMIs) are appliances, not servers: they export into the NixOS estate and never host a service of ours.

Standing decision (October 2026): the whole plant runs ERPNext, and it is the only system of record. Self-hosted on a NixOS machine in the same clan, it keeps every record of the business and of production: customers, prices, orders and the artist gates, invoices and payments, items, BOMs, stock and batches, work orders and job cards, quality inspections, metal parts and their genealogy, deliveries, assets and the accounts. What we build are tools and interfaces that read and write ERPNext (station kiosks, the customer portal, the artist platform, instrument adapters, reports); none keeps business records of its own. The one exception is machine data: time series, per-cycle and per-record rows, instrument raw files and images, audio captures. That stays in the machine-data store on the plant core, keyed to ERPNext's document names (ERPNext).

Open source throughout, and working with other plants

Decided (user, 17 October 2026): everything the plant runs and everything we build is open source, and our own code is published.

Layer	What runs	Licence
Operating system and deployment	NixOS on every server, gateway, collector and kiosk, declared in the cnx-network clan (clan-core)	nixpkgs: MIT
System of record	ERPNext on the Frappe framework, self-hosted (ERPNext)	ERPNext: GPLv3; Frappe: MIT
Machine data	PostgreSQL, VictoriaMetrics, Grafana, Mosquitto (MQTT), Telegraf	PostgreSQL licence, Apache-2.0, AGPLv3, EPL-2.0, MIT
Edge and backup	Caddy, BorgBackup, ZFS	Apache-2.0, BSD, CDDL
Our applications	the factory application (cnx-factory, Rust: axum, SQLx, minijinja, HTMX), the plant overview, the artist platform (name undecided), the plant and business plan calculator, instrument adapters, NixOS modules	ours, to be published
Design and documents	FreeCAD for the plant model, python-docx and LibreOffice for the business plan	LGPL, MIT, MPL-2.0

• Our licence is a decision point. AGPL-3.0 keeps hosted copies of the factory application open; Apache-2.0 or MIT is the easiest for another plant to adopt. Decide before the first public release.
• The exceptions are inside the machines. The presses' PLCs and HMIs (Newbilt's Siemens, Pheenix's Beckhoff), the instruments (Keyence, the vision controllers), inverter and battery firmware and the vendors' OPC UA licences are closed. They are appliances: they export data into the NixOS estate and host none of our services. The fleet's access-point controller in the router module is closed too.
• What it gives the plant: no licence fees (ERPNext has no per-seat fee), no vendor lock-in, every machine rebuilt from the repository, and a record a customer can audit (the hash-chained event log).

Later: a service for other pressing plants. Once the plant has run the stack in production (at least a year), it can be offered to other plants around the world:

1. An open vinyl plant stack: the NixOS modules, an ERPNext app for pressing (stampers and their genealogy, test pressings, the three gates), the press adapters for Newbilt and Pheenix over OPC UA, S7 or ADS, the kiosk and QC tools.
2. Services: factory automation and data acquisition for another plant (integration and commissioning), remote monitoring and support, hosting of ERPNext and the factory application, training.
3. Shared development: each plant that joins contributes the adapters for its own machines, so the stack covers more presses over time.

None of this is in the business plan's revenue: no price or volume is set, and it is an upside like stamper supply to other plants.

What the fleet already provides

Read the clan's own runbook (docs/ in the repo, served on control) for detail; this is what the plant plan leans on.

Fleet piece	Where	What the plant gets from it
Clan repo, clan-core 26.05, sops vars gated by YubiKey, admins group applied to every machine automatically	git.b4l.co.th/B4L/cnx-network-clan	one inventory, one deploy path (clan machines install / update), secrets minted with clan vars generate
ZeroTier mesh, controller on control, IPv6 ULA /88, modules/mesh-hosts.nix reads every peer's address from vars	control	admin SSH, deploys and scrapes for plant machines behind PPPoE NAT, exactly as gw-cnx-1 is reached today; no port forwards, no VPS needed
modules/router/: cnx.router.* on Topton 1U boxes: PPPoE WAN, VLAN-filtering bridge, nftables with no inter-VLAN traffic, Kea, Blocky, CrowdSec, Omada, iperf3, hourly speed test, Caddy internal proxy with a real wildcard *.<site><n>.cnx.network via DNS-01	site gateways	the plant router is a copy of machines/gw-cnx-1/ with its own VLANs; the internal proxy gives every station a browser-trusted https://factory.<site>1.cnx.network
Addressing convention 10.<siteId>.<vlanId>.0/24, VLAN ids 10 mgmt and 20 lan mandatory, 30 guest, 40 iot reserved	fleet-wide	the plant takes the next free siteId and adds its own VLANs above 40
VictoriaMetrics (180 d), VictoriaLogs (30 d), vmalert, Grafana, blackbox probes	control	node_exporter on every plant machine scraped over the mesh, journald shipped to VictoriaLogs, host alerts from the existing rules; process data stays on site (below)
Knot DNS (cnx.network, buildfor.life, cnx.email) with per-consumer TSIG keys for ACME	ns1, ns2	the plant's public names and the gateway's wildcard cert; a brand domain for the customer portal is one zone file away
web01: public Caddy with *.cnx.network, backends dialed over the mesh	Hetzner	the customer portal is one virtualHosts entry pointing at the plant core's mesh address, the same way grafana.cnx.network works now
clan borgbackup service, control as server, repokey encryption, backups over the mesh	control	the pattern; the plant needs its own server because plant data is far larger than the DNSSEC keystore
chrony everywhere, fixed-offset time zones (Etc/GMT-7 for Thailand)	fleet baseline	plant machines inherit; edge collectors additionally serve NTP to the PLCs
mdbook runbook in docs/src/, built and served on control	control	the plant's operational runbook goes there as a chapter; this document stays the planning source

Known gaps in the fleet that the plant needs closed (all on the clan's own TODO list or small extensions to existing modules):

• Alert delivery: vmalert currently uses notifier.blackhole. A boiler alarm needs a phone. The plant brings an Alertmanager with Telegram or LINE and SMS receivers; control's vmalert can point at it too.
• Off-site backup destination: control's disk is not sized for plant images and audio. Add a Hetzner Storage Box or a storage VPS as a second borgbackup destination (the clan service supports several per client).
• Router module: three small extensions, listed under Networks.

Design decisions

1. One clan, one repo, rebuildable from git. Plant machines are entries in clan.nix's fleet set with a machines/<name>/ directory each (configuration.nix, disko.nix, facter.json), like every machine already there. A failed box is replaced from a fresh disk with clan machines install; a change is clan machines update. Nothing is configured by hand on a running machine.
2. ERPNext for records, a machine-data store for machine data. Every record of the business and of production (orders, gates, genealogy, lots, press runs, inspections, deliveries, money) lives in ERPNext (decision 10). What machines produce at machine speed lives in the machine-data store on the plant core, in two parts. Rows with an identity that are too many for an ERP (press cycles keyed by press and counter, per-record optical results, audio sample scores, per-unit packaging scans, the hash-chained event log) go to PostgreSQL, each carrying the ERPNext job card, batch and serial numbers it belongs to. Time series (1 Hz machine tags, zone environment, energy, water) go to a plant-local VictoriaMetrics, the same engine and modules the fleet already runs on control, so dashboards, PromQL and vmalert rules carry over unchanged. Per-cycle rows carry a snapshot of the zone environment and the key press values, so the relational join "this record, that cycle, those conditions" never needs the time-series store; the curve behind the snapshot is one PromQL range query away. (TimescaleDB in PostgreSQL is the fallback if that split proves awkward; it is not the starting point.)
3. MQTT is the plant bus, Telegraf is the universal collector. Every machine, meter and sensor is read by a Telegraf instance on the edge collector nearest it (Modbus, OPC UA, Siemens S7 and MQTT inputs are stock plugins) and published to Mosquitto on the core. On the core, Telegraf writes the stream to VictoriaMetrics (InfluxDB line protocol, native to VM) and the factory application subscribes to the event topics (cycle complete, alarm, instrument result), writes rows to the machine-data store and posts the totals and verdicts to ERPNext. Custom code exists only where no plugin does (vision results, instrument CSV ingestion, the audio QC pipeline).
4. Read-only toward machines. The data platform never writes to a PLC or an instrument. Control, safety and interlocks (fume extraction, steam shutoff, boiler safety chain) stay in the vendor PLCs and hard wiring, as facility security already requires.
5. Join keys are counters, not clocks. Every press cycle carries the machine's own cycle counter; every instrument result carries the part serial it was read from. Wall-clock time is recorded (UTC, chrony from the fleet baseline) but is never what two systems are joined on.
6. Append-only, hash-chained. Machine events are written once and never updated; corrections are new events that reference the old. Each event carries the hash of the previous one so the log's integrity can be checked, and records the ERPNext document it led to. In ERPNext, gate approvals, quality inspections, job cards and stock entries are submitted documents: a correction is a cancel and amend that keeps both, with ERPNext's version history. Together that is what lets the customer portal be trusted.
7. The customer portal is a view, not a copy. It reads the customer's records in ERPNext and the machine data of the same orders through one portal tool scoped to the ERPNext customer; there is no exported report to keep in sync.
8. The plant runs without the internet. Everything that pressing, QC and traceability need is on site. A WAN outage stops only the customer portal, off-site backup, fleet monitoring and remote support, and all four catch up when it returns.
9. Operators see the data first. Every live value the platform collects is shown at the station where it can change the next cycle (the press-side kiosk with cycle values against the recipe band, the inspection reject reasons, the stamper's play count) before it is rolled up into reports. Sibert's advice after Haarlem was exactly this: live feedback loops that the staff use to improve production, rather than only for management reporting, decide whether a plant succeeds (exhibitor briefing).
10. ERPNext is the system of record; we build tools and interfaces. (October 2026.) Invoices, stock and every other record of the business and of production live in ERPNext: customer, sales order, artist gate, work order, job card, batch, quality inspection, serial number per metal part with its parents, stock entry, delivery note, invoice, asset. What we build reads and writes those records; the only data kept outside is machine data (decision 2), which ERPNext receives as totals and verdicts per job card, run, batch and shipment. A new need is met with a standard ERPNext document type first, then with custom fields or a custom document type in our cnx_erp app, and only then with a tool of ours that still keeps its records in ERPNext; never with a second database of business records. ERPNext document names are the keys everything shares (ERPNext).

The systems on one page

System	What it does	Runs on	Phase
Site gateway	PPPoE WAN, VLAN zoning, DHCP/DNS, NTP for the site, internal TLS proxy, CrowdSec	gw-<site>-1 (existing cnx.router module)	Build
Plant core	Machine-data store (PostgreSQL for cycle and record rows, VictoriaMetrics + vmalert + Grafana for time series), Mosquitto, Telegraf persister, Alertmanager, factory application, file datasets on ZFS	vinyl-core	Build
ERPNext	The system of record: accounting with Thai VAT and withholding tax, customers, prices, sales and purchase orders, artist gates, items, BOMs, work orders and job cards, stock and batches, quality inspections, metal parts and their genealogy, deliveries, assets and maintenance, payroll	vinyl-erp	from the first capex invoice; production records at commissioning
Factory application	The tools and interfaces we build on ERPNext: station kiosks working the job cards, customer portal, machine-data service (cycles and per-record results in, totals and verdicts to ERPNext), instrument adapters, reports; the ERPNext integration layer. No business records of its own	vinyl-core	Build (v1), Launch (v2)
Edge collectors	Telegraf per machine zone, disk-backed store-and-forward, NTP to the OT segment, vision result receiver	vinyl-edge-press, vinyl-edge-util, vinyl-edge-galv	Build, Expand
Environmental monitoring	Zone temperature, humidity, PM, differential pressure; alarms; correlation with production	RS485 loops into the edge collectors	Build
Machine telemetry	Presses, extruders, hydraulics, boiler, chiller, compressor, heat pump, HVAC, PV, BESS, energy and water meters	edge collectors	Build
QC instrument integration	Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans	stations plus vinyl-edge-press	Launch, Expand
Station kiosks	Badge check-in and scanning at each manual step against the station's ERPNext job card; kiosk browser on the factory application	vinyl-st-*	Build
Process alerting	Environment, utilities, reject rate, stamper life, PV/BESS, fire and BESS detection contacts, UPS, data silence; delivered to phones	vinyl-core (vmalert + Alertmanager)	Build
Fleet monitoring	Host metrics, journals, host alerts for every plant machine	control (existing)	Build
Backup and DR	ZFS snapshots and replication, Borg to the plant backup server and off site, PostgreSQL point-in-time recovery, restore drills	vinyl-backup, off-site destination	Build
Identity	Staff sign-on (OIDC), badge IDs for operators at stations, customer role	vinyl-core	Build
Client audio vault	Encrypted, isolated dataset for pre-release audio with checksums and access logging	vinyl-core	Build
Site systems integration	Access-control door events, fire panel and BESS detection contacts, UPS state	vinyl-edge-util	Build
Customer portal, public	TLS termination and forwarding over the mesh to the plant core	web01 (existing)	Launch
CCTV	Recording on its own VLAN	vendor NVR appliance, or vinyl-nvr (Frigate) if wanted	Build

Architecture

flowchart LR
    subgraph L0[Field, isolated OT segments]
        PLC[Press and extruder PLCs<br/>OPC UA / S7]
        UT[Boiler, chiller, compressor,<br/>heat pump, HVAC<br/>Modbus / BACnet]
        EN[Energy meters, Deye inverters,<br/>BESS, water meters<br/>Modbus / pulse]
        ENV[Zone T / RH / PM / dP sensors<br/>Modbus RTU]
        GV[Rectifiers, bath sensors<br/>Modbus]
        VIS[Vision controller<br/>TCP results, SFTP images]
        INST[Keyence, XRF, lab instruments<br/>CSV export]
    end
    subgraph L1[Edge, NixOS, dual-homed]
        EP[vinyl-edge-press<br/>Telegraf + vision receiver]
        EU[vinyl-edge-util<br/>Telegraf + dry-contact I/O]
        EG[vinyl-edge-galv<br/>Telegraf]
        ST[vinyl-st-* stations<br/>kiosk, scanners, ingestion]
    end
    subgraph L2[Plant core, NixOS, servers VLAN]
        MQ[Mosquitto]
        VM[(VictoriaMetrics<br/>vmalert, Grafana)]
        DB[(PostgreSQL<br/>machine data)]
        APP[Factory application<br/>tools on ERPNext]
        AM[Alertmanager]
        ERP[(ERPNext<br/>vinyl-erp)]
    end
    subgraph GW[gw-site-1]
        PX[Caddy internal proxy<br/>factory.site1.cnx.network]
    end
    subgraph L3[Fleet, Hetzner, over the mesh]
        CTL[control: node metrics,<br/>journals, host alerts]
        WEB[web01: public proxy]
        BK[off-site borg destination]
        CUST[Customer portal]
        PH[Phones: alerts]
    end
    PLC --> EP
    VIS --> EP
    UT --> EU
    EN --> EU
    ENV --> EU
    GV --> EG
    INST --> ST
    EP --> MQ
    EU --> MQ
    EG --> MQ
    MQ --> VM
    MQ --> APP
    APP --> DB
    APP -->|REST, outbox| ERP
    ERP -.->|webhooks| APP
    ST --> PX --> APP
    VM --> AM --> PH
    APP -.mesh.-> WEB --> CUST
    L2 -.mesh.-> CTL
    L2 -.mesh.-> BK

Three rules keep this simple: field devices sit on isolated OT segments whose only other member is their edge collector; edge collectors publish to the core and nothing else; stations and office clients reach the application only through the gateway's proxy. Everything that leaves the site rides the ZeroTier mesh, never an open port.

Machines

All NixOS, all entries in the clan inventory. Names follow the fleet convention (gw-<city>-<n> for gateways, short role names for servers) with a vinyl- prefix so the plant's machines are recognisable in a multi-site inventory. <site> is the city code chosen at site selection; Mueang Chi is in Lamphun province, so lpn is the working assumption and the gateway is gw-lpn-1, giving internal names under *.lpn1.cnx.network. Hardware classes are planning positions; models are chosen at purchase.

Machine	Role	Hardware class	Where
gw-lpn-1	Site gateway: cnx.router with the plant VLANs, internal proxy for the factory application and Grafana, NTP for the site	Topton 1U N300, 4x 2.5G (fleet standard); a 10G SFP+ variant if the core server link warrants it	IT room
vinyl-core	PostgreSQL, Mosquitto, Telegraf persister, VictoriaMetrics + vmalert + Grafana, Alertmanager, factory application, ZFS datasets for files (audio captures, inspection images, metrology raw data, CoA scans, client audio)	1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD mirror or raidz for files, dual PSU on UPS	IT room
vinyl-erp	ERPNext (Frappe web, workers, scheduler, socket.io), MariaDB, Redis, the Thai localisation app and our cnx_erp app	1U server, 32 GB RAM, mirrored NVMe, on UPS; or a VM on vinyl-core if its hardware has room	IT room
vinyl-backup	Borg server for the plant clients, ZFS receive target for vinyl-core's datasets, PostgreSQL streaming replica, pgBackRest repository	tower or 2U, large HDD pool	separate room from vinyl-core (archive side of the building)
vinyl-edge-press	Telegraf (OPC UA / S7 to presses and extruders, Modbus to hydraulics and label oven), vision result receiver, SFTP target for inspection images, chrony serving the OT segment, disk-backed MQTT buffer	fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485	press hall control cabinet
vinyl-edge-util	Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS	same class, more RS485	electrical room or utility yard cabinet
vinyl-edge-galv	Telegraf (rectifiers, bath sensors, extraction status)	same class, corrosion-protected enclosure outside the wet area	galvanics service corridor (Expand phase)
vinyl-st-cutting	Session log entry, lacquer registration, scribe ID	mini PC, touch screen, badge reader	cutting room
vinyl-st-press1, vinyl-st-press2	Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons	mini PC, touch screen, badge reader, 2D scanner	at each press
vinyl-st-lab	PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment	mini PC, RS232/USB to instruments	material lab
vinyl-st-metrology	SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan	mini PC	metrology bench (Expand phase)
vinyl-st-audioqc	Needle-drop capture and analysis pipeline	mini PC with audio interface, turntable, phono stage	listening/QC room (Expand phase)
vinyl-st-pack1	Runout code camera read, jacket barcode scan, BOM check, barcode verifier	mini PC, camera, scanner, verifier	packaging
vinyl-st-ship	Consignment assembly, carrier label and tracking capture	mini PC, label printer	shipping
vinyl-nvr	Frigate CCTV recording on the cctv VLAN (optional; a vendor NVR appliance is the alternative)	small server with GPU or Coral accelerator	IT room

Office audio intake and gate approvals happen from ordinary staff laptops on the lan VLAN through the same proxy; no dedicated machine. Stations run a kiosk browser (cage plus Firefox or Chromium) pointed at the application; badge readers and scanners are USB HID keyboard devices, so there is no driver work. Sixteen plant machines at launch, about twenty-one at maturity; every one is a directory and a line in the inventory.

Networks

The plant is site <siteId> (next free number in the fleet's site table; cnx is 1 and the public-WiFi site is 2, so 3 is the working assumption) and owns 10.<siteId>.0.0/16. VLANs follow the fleet convention; the ids above 40 are proposals.

VLAN	id	Members	allowWan	Rule
mgmt	10	switches, APs, Omada, IPMI, admin laptops on site	yes	fleet convention: trusted, reaches everything
lan	20	office laptops, printers	yes	proxy to factory app and Grafana, internet
guest	30	visitors, vendor laptops	yes	internet only
servers	100	vinyl-core, vinyl-backup, the servers-side NIC of every edge collector	yes (mesh, nix, journald, off-site backup)	no inbound from other VLANs; the gateway's own Caddy dials the app here
stations	110	vinyl-st-*, instrument PCs (Keyence, vision setup laptop), label printer	yes (mesh, deploys, journald)	proxy to the factory app only; dhcp.reservations for every station
cctv	120	cameras, NVR	no	reachable from mgmt; NVR UI via the proxy if wanted
ot-press	210	press and extruder PLCs and HMIs, vision controller, OT NIC of vinyl-edge-press	no	isolated segment, see below
ot-util	220	boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, OT NIC of vinyl-edge-util	no	isolated segment
ot-galv	230	rectifiers, bath transmitters, OT NIC of vinyl-edge-galv	no	isolated segment

OT segments are islands, not routes. Each edge collector is dual-homed: one NIC on servers (an access port on the cabinet switch), one NIC on its OT segment. The OT segment lives on the cabinet's industrial switch; if it must span cabinets it rides the building trunk as a VLAN with dhcp.enable = false and allowWan = false, and the router's default no-inter-VLAN policy keeps it closed. Either way the gateway forwards nothing into or out of an OT segment; the collector is the only bridge, and it only publishes. Zone sensors on RS485 loops terminate directly on the collectors and have no IP presence at all. Cheap ESP32/MQTT sensor nodes are acceptable for extra density in non-critical zones (on stations, publishing to the core), never for alarmed zones.

Stations never route to servers. They reach https://factory.lpn1.cnx.network, which Blocky resolves to the gateway and Caddy forwards to vinyl-core on the servers VLAN, with the real wildcard certificate the router module already issues. proxy.allowVlans gains stations. The same proxy serves grafana.lpn1.cnx.network to the office. This keeps the fleet's trust model intact: no VLAN talks to another VLAN; the router talks to its backends.

Router module extensions the plant needs (small, and useful for the other sites too):

1. NTP served by the gateway to every VLAN (chrony allow for the site's /16, UDP 123 in the per-VLAN firewall set) so stations and the NVR sync locally; today the gateway is only a chrony client.
2. Static-only VLANs: a dhcp.enable = false VLAN should still accept reservations semantics for documentation, or simply be documented as address-by-hand; either is fine, decide when writing it.
3. Second WAN: a 4G/5G modem as failover for the PPPoE session (wan.backupInterface with a metric and a health check). Not needed for launch because the plant runs without the internet; needed before the customer portal is promised as always-on.

Vendor remote support (Newbilt and Pheenix both offer remote monitoring) goes through an admin's SSH port-forward across the mesh into the relevant edge collector and from there to the one PLC HMI they need, time-boxed and logged. No machine ever gets a vendor's own cellular router or cloud agent on an OT segment.

What changes in the clan repo

Following the "Adding a site" procedure in the clan runbook, plus the plant's own modules. Everything below is one pull request per machine class, in this order.

clan.nix
  fleet: + gw-lpn-1, vinyl-core, vinyl-backup, vinyl-edge-press,
           vinyl-edge-util, vinyl-st-press1, vinyl-st-lab,
           vinyl-st-pack1, vinyl-st-ship  (then the rest as they come)
  borgbackup:
    roles.server.machines.vinyl-backup = { };
    roles.client.machines.vinyl-core.settings.destinations = {
      plant  = { repo = "borg@vinyl-backup:vinyl-core"; };
      offsite = { repo = "<storage-box or bk host>:vinyl-core"; };
    };
    roles.client.machines.vinyl-edge-press = { ... plant only ... };
    # side benefit: control can now back up its VictoriaMetrics TSDB to
    # vinyl-backup, closing the open item in the clan TODO
  (no `internet` entries: plant machines sit behind PPPoE like gw-cnx-1
   and are reached over the mesh)

modules/mesh-hosts.nix      + every plant machine in the name list
modules/monitoring/server.nix
                            + node_exporter targets for every plant machine
                            + a second Grafana datasource "vinyl-vm" pointing
                              at [vinyl-core mesh]:8428 so fleet dashboards
                              can show plant series
modules/monitoring/alerts.nix
                            unchanged; host rules already cover new targets
machines/ns1/configuration.nix
                            + dns-acme-gw-lpn-1-knot generator, keyFiles entry,
                              acl_acme_gw_lpn_1 scoped to _acme-challenge.lpn1
                              (copy the gw-cnx-1 blocks)
modules/dns/acme-gw-secret.nix
                            reused as-is with "gw-lpn-1"
modules/web-proxy.nix       + virtualHosts."portal.<domain>" -> [vinyl-core mesh]:8080
                              (second cert + scoped TSIG key if <domain> is
                              not cnx.network, following the acme_web01 pattern)

machines/gw-lpn-1/          copy of gw-cnx-1: site "lpn", siteId 3, the VLAN
                            table above, proxy.services.factory and .grafana,
                            proxy.allowVlans = [ "mgmt" "lan" "stations" ]
machines/vinyl-core/        imports modules/vinyl/{core,metrics,factory-app,
                            alertmanager}.nix + monitoring/exporters.nix
machines/vinyl-backup/      imports modules/vinyl/backup.nix + exporters.nix
machines/vinyl-edge-*/      imports modules/vinyl/edge.nix with the cabinet's
                            Telegraf inputs and the OT NIC config
machines/vinyl-st-*/        imports modules/vinyl/station.nix with the kiosk URL

modules/vinyl/core.nix      PostgreSQL (ZFS dataset, pg_hba for the app and
                            the replica), Mosquitto (per-client passwords from
                            vars generators), Telegraf persister (MQTT in,
                            VictoriaMetrics out), ZFS datasets and sanoid
modules/vinyl/metrics.nix   VictoriaMetrics + vmalert + Grafana for the plant,
                            a sibling of monitoring/server.nix with the
                            process rules and dashboards under
                            modules/vinyl/dashboards/
modules/vinyl/alertmanager.nix
                            receivers: Telegram or LINE for the shift group,
                            SMS gateway for the critical set, email digest;
                            tokens from vars generators
modules/vinyl/factory-app.nix
                            the cnx-factory package as a flake input (like
                            nixos-mailserver is today), systemd service,
                            migrations on start, DB password from vars,
                            OIDC client secret from vars
modules/vinyl/edge.nix      Telegraf with per-machine input lists, MQTT output
                            with disk buffer, chrony `allow` on the OT NIC,
                            networkd config for the two NICs, SFTP chroot for
                            vision images (press collector only)
modules/vinyl/station.nix   cage + browser kiosk, auto-login, HID readers,
                            SMB share and ingestion watcher where needed
modules/vinyl/backup.nix    borg server role settings, syncoid receive,
                            PostgreSQL replica, pgBackRest repo
modules/router/…            the three extensions above
docs/src/vinyl.md           the plant chapter of the runbook, + SUMMARY.md

Operational notes that already bit the fleet and apply here: git add a new machines/ directory before evaluating (untracked files are invisible to the flake); run clan vars generate <machine> (YubiKey) before the first install and again after adding a generator; check facter.json for the real NIC names after install; the clan wrapper in the dev shell updates machines sequentially so YubiKey prompts are not buried.

Repository layout, decided: infrastructure lives in cnx-network-clan; the factory application gets its own repository cnx-factory in the B4L organisation and is consumed as a flake input; this repository stays documents and links to both. That satisfies the README's promise that the project repository "hosts the software" without mixing Nix, Rust and planning prose in one history.

Data acquisition, source by source

Every row is also a purchasing clause: the "requirement" column is what goes into the RFQ, per the traceability principle that machine data access is bought, not retrofitted. Rates are planning values.

Presses and extruders

Source	Signals	Interface	Rate	Requirement to write into the RFQ
Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA)	cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes	OPC UA server on the PLC, read by Telegraf inputs.opcua	per cycle plus 1 Hz	OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty, PLC clock accepting NTP
Newbilt HRP 12 semi-automatic and VEX 50 extruder (PLC type unconfirmed)	as above where the PLC has them; on a relay-logic machine at least press closed/open, steam on/off, cycle count	OPC UA or Siemens S7 (inputs.s7comm); fallback: our own 4 to 20 mA sensors on a Modbus TCP I/O module and a cycle count from the mold-closed limit switch	per cycle plus 1 Hz	state what the PLC is and which of the two paths applies; the fallback sensor set is priced separately
Pheenix AD12 (Beckhoff PLC, remote monitoring supported)	same tag family as the Classic	OPC UA (Beckhoff TF6100) or ADS; inputs.opcua	per cycle plus 1 Hz	OPC UA license included, tag list delivered, remote monitoring routed through our mesh, not the vendor's
Hydraulic aggregate	system pressure, oil temperature, tank level, filter differential pressure	via the press PLC if exposed, else own transmitters on the I/O module	1 Hz	ask whether the aggregate's sensors are on the PLC
Label dry oven	chamber temperature, timer state	Modbus if the controller has it, else a thermocouple on the I/O module	1/min
PVC grinder and regrind scale	run state, batch weight	scale with RS232 to vinyl-st-press1, run contact to I/O	per batch	scale must have a serial output

Utilities and energy

Source	Signals	Interface	Rate	Requirement
Steam boiler (Thai-sourced)	pressure, steam temperature, feedwater temperature, drum level, burner or heater state, fuel or kW, blowdown events, water treatment conductivity and hardness alarm	Modbus RTU or TCP on the boiler controller	1 Hz	Modbus register map delivered with the boiler; Thai boiler-regulation inspection data logged too
Steam accumulator	pressure	pressure transmitter to I/O	1 Hz
Chiller and cooling tower	supply and return temperatures, flow, compressor power, tower fan state, makeup water	Modbus	1 Hz	register map
Feedwater heat pump	source and sink temperatures, power	Modbus	1/10 s	register map
Compressor	line pressure, run hours, dew point	Modbus	1/10 s	register map
HVAC / BMS	AHU states, setpoints, supply temperatures, filter differential pressure, positive-pressure room dP	BACnet/IP or Modbus from the BMS the HVAC contractor supplies	1/min	the RFQ names BACnet/IP or Modbus TCP read access as a deliverable, with point list
Electrical	main incomer, feeders for press hall, chiller, HVAC, boiler, galvanics, packaging, IT; PV inverters (Deye, Modbus RTU/TCP register map is public); BESS state of charge, power, cell temperatures (through the Deye battery registers or the Gobel BMS RS485)	Modbus energy meters on each feeder; inverters direct, no cloud dongle	1/10 s	meters specified in the electrical design, not added later
Water	mains or well meter, process tank level, boiler and cooling tower makeup meters, galvanics rinse recycle, fire tank level	pulse counters and level transmitters on I/O	1/min	meters in the plumbing design

Environment

Zone	Sensors	Alarm tier
cutting room	T, RH, PM2.5/PM10, room dP	tight, positive pressure
galvanics clean room	T, RH, PM, room dP	tight, positive pressure
master and stamper archive	T, RH, PM, water leak	tightest
press hall (two points at launch, four at maturity)	T, RH, PM	process spec 20 to 26 °C, 35 to 60 % RH
material lab and PVC store	T, RH	process
packaging and finished goods	T, RH, PM	process
listening/QC room	T, RH	comfort
IT room	T, RH, water leak	tight
office	T, RH, PM	comfort
outdoor reference	T, RH, PM2.5	reference for burning-season episodes

Industrial RS485 Modbus RTU transmitters on wired loops to the nearest edge collector, one reading per minute, with a reference-grade PM instrument in the archive or cutting room to check the cheaper sensors against. Every zone alarm has two thresholds (warning, action) and a rate-of-change alarm for the archive.

Galvanics (Expand phase)

Rectifier current, voltage and ampere-hours per cell (Digital Matrix: ask what the control rings and rectifiers expose; Pheenix Alpha: its plating units are PLC-controlled with a network connection, protocol unstated, see galvanics and stamper prep; Modbus is the requirement), bath temperature, pH and conductivity from transmitters, filtration and agitation pump states, fume extraction running status (the interlock itself stays hard-wired), rinse water conductivity and evaporator state. Titration results are entered at a station against the bath ID. On the Pheenix units, the machine's own alerts (filter cartridge change, nickel pellet top-up, anode basket cleaning) are logged as bath events.

Site systems

Access-control door events (badge ID, door, direction, time) from the controller's API or export; fire alarm panel zone and fault contacts, BESS off-gas and early-smoke detection, UPS on-battery and low-battery (NUT on vinyl-core talking to the UPS) as dry contacts on vinyl-edge-util. CCTV stays on its own VLAN with its own retention; it is never joined to production data.

What produces no telemetry

The Sibert stamper-prep machines (their current specifications, 2022 and 2023, list no data or network interface; Sibert's 2009 optical-disc punch offered barcode and LAN options, and Sibert offered to carry features of its fully automatic VSPa punch across to the VSP, so ask) and the shrink wrapper produce no machine data, and neither does the mastering chain on the Agnew route. They get station check-ins: who, when, which part, which parameters, with files attached (cutting parameters, DAW session export). The room environment around them is logged by the sensors above.

If the lathe is a Sillitoe SDMS instead, its Space / Time Control software (a vendor application on the cutting room's own PC, which stays an appliance) writes files worth collecting: the master cue sheet (catalogue number, side, matrix number, engineer, groove diameters per track), an Open Groove export of the groove itself, the project file, and a stylus-hours counter. They are dropped on an SMB share on the cutting station and attached to the cutting session against the lacquer ID, the same watcher pattern as the metrology exports below. See vendor landscape.

QC instrument integration

Instrument	How results reach the database	Key	Open integration item
Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration)	vision controller sends one result string per record over TCP to a receiver on vinyl-edge-press; images pushed by SFTP to the collector, synced to the core's image dataset	press ID + cycle counter, passed from the PLC to the vision trigger	the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test
Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer)	vendor Windows PCs export CSV and raw 3D to an SMB share on vinyl-st-metrology; a watcher parses and posts	back-side Data Matrix serial, scanned at the bench before measurement	export format per instrument confirmed with Keyence Thailand
PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR)	instruments with RS232/USB export are read at vinyl-st-lab; the rest are typed in with the printout photographed; CoA PDF attached	compound lot ID (printed label scanned at intake)	which benchtop models have serial export
Audio QC station	capture service on vinyl-st-audioqc records the needle-drop (PipeWire, FLAC to the core's capture dataset); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted	record key (stamper + press + cycle + side), typed or picked from the run list	first-of-kind pipeline, developed in Expand phase against launch-phase recordings
Packaging assembly check	camera with darkfield lighting reads the scribed dead-wax code (a handheld scanner will not read a scribed code on black vinyl reliably); handheld scanner reads the jacket barcode; barcode verifier grades it	order + record key	confirm dead-wax code readability with a sample before the packaging station is designed
Label print inspection	same vision family as the record label check; result per label lot	label lot ID	shared with the record inspection budget

The factory application

Working name cnx-factory, its own repository: the tools and interfaces we build on ERPNext, not a second system of record. Server-rendered web application in Rust (axum, SQLx, minijinja templates, HTMX), the stack B4L already uses for its other in-house tooling. Every business and production record it shows or changes is read from and written to ERPNext through the integration layer (ERPNext). The only data it owns is the machine-data store: it subscribes to the MQTT event topics (cycle complete, machine alarm, instrument result), writes cycle and record rows to PostgreSQL with the append-only, hash-chained event log, and posts totals and verdicts to ERPNext; Telegraf owns the writes to VictoriaMetrics. Packaged as a flake so the clan repo pins it as an input.

Demo. Factory console and order status (test site /demos/factory/) shows what the application would look like on a simulated plant day: the press floor with live counters per press, work orders per station in the kiosk form described below (badge in, scan what the step asks for, confirm), the optical inspection stream, audio samples, the material lab gate with a lot on hold, stamper metrology and genealogy, the hash-chained event log, the ERPNext link (every posting the floor makes: sales orders in by webhook, job cards, batches and quality inspections, stock entries, downtime entries, delivery notes and invoices, through the outbox, with the ERPNext documents on each order and work order), and the customer portal of the same orders (progress through the three gates, copies pressed so far, the lab certificate of the compound lot, listening checks). All orders, values and IDs are simulated; limits are illustrative until the acceptance specifications are written.

Modules, in build order. Each keeps its records in ERPNext; the last column is the machine data it keeps outside.

#	Module	Records in ERPNext	What we build	Machine data, outside ERPNext
1	Orders and releases	Customer, Sales Order, an Item and BOM per release and format (record, inner sleeve, jacket, inserts, shrink), custom Release and Side documents, due dates	order intake on the artist platform, order views	none
2	Audio intake and the client vault	custom Audio Asset document per file version: checksum, handler, technical QA form	upload via portal or SFTP with automatic checks	the audio files, in an encrypted, access-logged dataset separate from everything else
3	The three artist gates	custom Artist Gate document per gate (master plan, master, test pressing), submitted on approval: who, when, from which portal session or on whose verbal instruction; the next job card cannot start without it	approve and report-a-problem in the portal	none
4	Mastering and cutting sessions	Job Card with custom fields: engineer, chain settings, lacquer serial numbers, dead-wax scribe text and fiducial, microscope check	mastering and cutting-room kiosks	lathe logs, microscope images
5	Metal genealogy	Serial No per lacquer, father, mother and stamper with its parent (or the bridged supplier and batch), its warehouse (archive with box and shelf position in the mobile shelving, facility security; rack; press), play count and retirement reason, stamper sets grown per mother; a Quality Inspection per measured part	plating, stamper-prep and archive scanning	Keyence and XRF raw files
6	Materials	Batch per PVC lot, regrind batch, label lot, carton and sleeve lot; Purchase Receipt; Quality Inspection for the lab gate (CoA attached), label print inspection and incoming QC; Stock Entry into the hopper	lab station, goods-in scanning	instrument raw files
7	Press runs	Job Card per run: press (workstation), stamper serial numbers, operator, shift, batches consumed, good and rejected quantities; Downtime Entry per stop; Stock Entry (manufacture) at the end	press-side kiosk, the machine-data service	every cycle as a row keyed by press + counter with the key press values and the zone environment snapshot; the record key is derived from it per side
8	QC verdicts	Quality Inspection per run, batch and test pressing with reject reasons; flagged listening checks with the engineer's verdict; the test-pressing hold	QC views, listening station	optical result and image per record, audio captures and scores per sampled record, packaging scan per unit
9	Finishing, packaging, fulfillment	packing Job Card, Delivery Note, Shipment with carrier and tracking per consignment or dropshipped end customer, Sales Invoice	packing and dispatch kiosks	per-unit assembly scans (dead-wax code against jacket barcode)
10	Customer portal	reads the customer's records above, scoped to the ERPNext customer; PDF export of the trace	the portal	reads the machine data of the customer's own orders
11	Reports and signals	ERPNext reports for money, stock and work orders	yield per stamper, reject-rate drift, audio score versus play count (the automatic retirement signal), energy per record, environment excursions over runs	the curves from VictoriaMetrics for the run's window

The public front of order intake (price calculator, uploads with automatic checks, fan-funded campaigns) is planned in artist platform, including why its public side should not depend on the plant's WAN link.

The machine-data store has the only tables we own: press_cycle, record, optical_result, audio_result, packaging_scan, instrument_file, event (the hash chain), each carrying the ERPNext names it belongs to (job card, batch, serial number). In ERPNext the cnx_erp app adds the custom document types Release, Side, Audio Asset and Artist Gate, and custom fields on Job Card, Serial No, Batch and Quality Inspection; everything else is a standard ERPNext document type.

Stations use the same application in kiosk mode: an operator taps a badge (the same card the access-control system issues, so one ID) and scans whatever the step asks for; each confirmation is a time log or a completed quantity on the station's ERPNext job card. No forms longer than one screen. The office works in ERPNext's own desk (purchasing, accounts, stock counts, planning); we build an interface only where the desk does not fit the work.

ERPNext: the plant's ERP

Decided October 2026: the whole plant runs ERPNext, the open-source ERP on the Frappe framework (GPLv3, no licence fee; version 15 is the current stable line). It closes the item this plan had left for after launch, and it sets the rule for everything we build: ERPNext is the system of record, and what we build are tools and interfaces that interact with it, through the integration described here. Invoices, stock and every other record of the business and of production are kept in ERPNext; only machine data (telemetry and the other machine-speed data of decision 2) is kept outside it.

Who owns what

Object	In ERPNext	Outside ERPNext (machine data)	Our tools
Customers, contacts, addresses (fans who buy included)	Customer, Contact, Address	none	the portal is scoped to the ERPNext customer; the artist platform links each account to one
Price list, item prices, pricing rules	Price List, Item Price, Pricing Rule	none	the artist-platform calculator reads them; the demos' price list becomes an ERPNext price list
Quotations, sales orders, invoices, payments (fan pledges in the ring-fenced account included)	Quotation, Sales Order, Sales Invoice, Payment Entry	none	the artist platform creates them
Releases, audio versions, the three artist gates	custom Release, Side, Audio Asset and Artist Gate documents	the audio files, in the client vault	portal upload and approval
Items and BOMs: one item per release and format (record, sleeves, jacket, labels, compound per copy), raw materials, packaging, spares	Item, BOM	none	none
Work orders, job cards per operation (mastering, cutting, plating, stamper prep, test pressings, pressing, packing), workstations	Work Order, Job Card with time logs and quantities, Operation, Workstation	none	the station kiosks work the job cards
Stock: compound lots, regrind, label and carton lots, finished goods, spares, warehouses including the archive	Batch, Purchase Receipt, Stock Entry, Warehouse	which lot sat in which hopper for each cycle	lab and goods-in stations, the machine-data service posts consumption
Lab gates and inspections	Quality Inspection per batch, run, stamper and test pressing, with templates carrying the parameters and limits of quality automation; CoA attached	instrument raw files and images	instrument adapters create the inspections
Metal parts: lacquers, fathers, mothers, stampers	Serial No with its parent, location (archive box and shelf, rack, press), play count, retirement, sets per mother	metrology raw files	plating, prep and archive kiosks
Press runs and stops	Job Card per run (good, rejected, scrap to regrind), Downtime Entry per stop (workstation, reason, from, to)	per-cycle rows, PLC telemetry	the machine-data service posts the totals
Per-record inspection, audio samples, packaging scans	totals and verdicts per run, the engineer's verdict on each flagged sample	every record's optical result and image, audio captures and scores, per-unit scans	QC views
Environment, energy, utilities	energy per record in reports, from totals	VictoriaMetrics time series	Grafana, alerting
Deliveries	Delivery Note, Shipment (carrier, tracking)	none	dispatch kiosk
Assets: presses, lathe, galvanics, Sibert line, utilities, PV and BESS, IT	Asset, depreciation, Asset Maintenance, maintenance schedules	run hours, alarms	none
Purchasing, suppliers, accounting, Thai VAT and withholding tax, payroll	ERPNext, with the HR app for payroll	none	none

How every tool interacts with it

1. One integration layer. A typed client for the Frappe REST API (/api/resource/<DocType> and whitelisted methods) lives in cnx-factory and is the only path in or out. Each tool authenticates as its own integration user (API key and secret in clan vars) whose role grants only the document types it writes. ERPNext pushes changes (submitted sales orders and work orders, purchase receipts, items and prices) to the factory application with Frappe's webhooks.
2. Outbox, not live calls. Every posting is written to an outbox table in the same transaction as the floor event and sent by a worker with retries; each carries the event's ID in a custom field, so a retry never creates a second document. ERPNext down for an upgrade stops nothing on the floor; the outbox catches up (decision 8 extended).
3. Business-sized postings. Per job card, per run, per lot, per stop, per shift. Never per record or per cycle.
4. ERPNext names are the shared keys (sales order, work order, job card, batch, serial number). The machine-data store has keys of its own only for cycles and records (press + counter, and the record key on the sleeve, which never changes) and stores the ERPNext names beside them.
5. Customisation as code. Custom fields, document types, print formats and webhooks ship as one Frappe app in our Gitea (working name cnx_erp), installed and upgraded by the deployment, never clicked into production by hand.
6. Nightly reconciliation of totals per work order and per batch in both directions; a mismatch is an alarm, not a silent fix.
7. The public side never reaches ERPNext directly. The artist platform's public process (see artist platform) posts through the plant's API over the mesh; payment-provider webhooks land on the public process and become payment entries through the same outbox. No ERPNext endpoint is published.
8. New tools follow the same rule. A dashboard, a script or a kiosk app built later uses the same client and an integration user of its own, and keeps its records in ERPNext. A tool that cannot read from or write to ERPNext this way is not finished; a tool with its own database of business records is not allowed. Machine data is the only exception.

Hosting on NixOS

• ERPNext has no package or module in nixpkgs (the request, NixOS/nixpkgs#241921, has been open since 2023). Community Nix packaging exists: Avunu's frappe-nix (a NixOS module services.frappe with per-site systemd units, a uv2nix Python environment and a database snapshot before every migration) and pub.solar's erpnext-nix. Plan: start from frappe-nix, pinned as a flake input in the clan repo. Fallback: the official frappe_docker images declared as virtualisation.oci-containers on the same NixOS machine, still declared in the clan, never a hand-built Docker host. Both checked October 2026.
• MariaDB, ERPNext's standard database (10.11 on the version 15 stack), with Redis. PostgreSQL support in Frappe is growing but is not ERPNext's main path, so the plant runs two database engines. Backups: Frappe's site backup plus mariabackup to vinyl-backup and the off-site Borg set, and the quarterly restore drill includes ERPNext.
• On site (decision 8): machine vinyl-erp, reached by the office through the gateway proxy at erp.lpn1.cnx.network; staff sign on with the plant's OIDC provider (Frappe supports custom OAuth providers for login); the external accountant over the mesh.
• Upgrades run on a restored copy first; the clan pins the ERPNext version, so a major upgrade is a reviewed change, not a surprise.

Thai localisation and a partner

• erpnext_thailand (Ecosoft, Bangkok; version 15; Frappe Cloud marketplace, checked October 2026): Thai VAT including service VAT at payment, withholding tax and its certificates, PND and purchase and sales tax reports for the Revenue Department, billing notes, deposit invoices and Bank of Thailand exchange rates.
• Ecosoft (ecosoft.co.th) is a Frappe certified partner and the maintainer of that app; ask it for an implementation and support quote alongside doing it in-house. Its site shows an ERPNext rollout at a firm named Lamphun Plastpack.
• Payroll: Thai social security and PND 1 are not in that app's list; check what Frappe HR covers with configuration and what Ecosoft uses.
• BOI: machinery and raw materials imported under promotion have to be accounted for (BOI); the asset register and the stock ledger are where that record lives. Check the raw-material reporting the BOI asks for against what ERPNext produces before go-live.

Phasing and cost

• Now: a dev ERPNext next to the dev trio behind gw-cnx-1, the cnx_erp app skeleton, and the integration client in cnx-factory v0 against it.
• From the first capex invoice: accounting, purchasing and the asset register, so the BOI asset list and depreciation start there. The build's cost entries in proplan (project CNX-01) are the source for capitalising the plant at handover.
• Commissioning and launch: items, BOMs, workstations, inspection templates, releases and artist gates, work orders and job cards, serial numbers for every metal part (bought stampers as well as our own); stations work the job cards.
• Expand: payroll if it did not start earlier.

Estimates: no licence; vinyl-erp 2,000 to 4,000 USD of hardware (none if it runs as a VM); ERPNext configuration (chart of accounts, Thai tax, items, BOMs, workstations, inspection templates, print formats) 1 to 2 person-months or a partner quote; the integration and the cnx_erp app (custom documents for releases, sides, audio assets and gates; custom fields on job cards, serial numbers, batches and inspections) 2 to 3 person-months; Nix packaging and the clan module 0.5 to 1 person-month. The factory application's own estimate shrinks by what ERPNext now holds (orders, stock, genealogy, inspections, shipping); re-estimate both once the dev ERPNext runs.

Observability and alerting

Two tiers, split by where the data is:

• Process, on site (vinyl-core: VictoriaMetrics, vmalert, Grafana, Alertmanager): zone environment excursions and rate of change, boiler pressure and level, chiller return temperature, compressed-air pressure, hydraulic pressure, reject rate per run above baseline, stamper approaching its play-count limit, PV or BESS faults, fire and BESS detection contacts, UPS on battery. Works with the WAN down.
• Hosts, on the fleet (control): node_exporter and journald from every plant machine over the mesh, the existing ScrapeTargetDown, RootFilesystemFull, BackupJobFailed, BackupStale rules apply to them without changes.
• Data-silence alarms in both tiers: a collector that has published nothing for five minutes is an alarm, whatever the last value was.

Routing: Telegram or the LINE Messaging API for the shift group, email digest daily, SMS through a Thai gateway for the critical set (boiler, fire, BESS, archive climate) with an on-call rota. control's vmalert can drop notifier.blackhole and point at the same Alertmanager over the mesh, which closes the fleet's own alert-delivery item. The animated plant operation page is the design target for the live overview: once the series exist it is fed from VictoriaMetrics instead of its simulation.

Backup, retention and recovery

• Hourly ZFS snapshots on vinyl-core (sanoid), replicated to vinyl-backup (syncoid) for the file datasets and the PostgreSQL data directory.
• Nightly Borg (clan borgbackup) from vinyl-core and the edge collectors to vinyl-backup, and from vinyl-core to the off-site destination as a second entry in destinations; repokey encryption as the fleet already does.
• PostgreSQL point-in-time recovery: pgBackRest with WAL archiving to vinyl-backup, its repository included in the off-site Borg set; a streaming replica on vinyl-backup for fast failover.
• VictoriaMetrics: vmbackup snapshots to vinyl-backup nightly; the fleet's own open item (backing up control's TSDB) gets a destination at the same time.
• ERPNext: Frappe's site backup and mariabackup to vinyl-backup nightly, in the off-site Borg set, restored in the same drill.
• Configuration: the clan repo, cnx-factory and the cnx_erp app in Gitea; any machine is rebuilt from them.
• Restore drill quarterly: restore the database to a scratch VM and open a random order's trace in the portal. A backup that has never been restored does not count. The BackupStale rule already watches the timers.

Data	Retention
genealogy, orders, gates, QC verdicts, event log	life of the customer relationship, never deleted while the metal parts exist
per-cycle press rows	forever (small)
1 Hz machine telemetry	2 years in VictoriaMetrics (retentionPeriod); per-cycle rows carry the summary forever
zone environment	5 years (separate VM retention filter, or a second small instance)
energy and water	forever (small; also summarised per shift into PostgreSQL)
audio captures and inspection images	2 years, longer for any record with a complaint
client audio	per contract; default deleted 12 months after final delivery, or on request
access-control events	1 year
CCTV	30 days
host logs and metrics	fleet defaults: 30 d logs, 180 d metrics on control

PDPA applies to badge events, CCTV and customer contact data: a written retention policy, role-based access, and export/deletion on request. The customer portal shows operator roles, never operator names.

Security

• OT segments have no gateway interface and no internet; edge collectors are the only bridge and they only publish.
• Stations and office reach the application only through the gateway's Caddy, with the wildcard certificate the fleet already issues; the servers VLAN accepts nothing from other VLANs.
• Staff sign-on through an OIDC provider on vinyl-core (Pocket ID, passkeys; the fleet has no identity provider yet, so this is the first) with roles: admin, engineer, operator, sales, customer. Stations authenticate by badge for attribution, not for privilege.
• Secrets only in clan vars (YubiKey-gated, admins group); nothing in plain text in either repo.
• Client audio: separate encrypted ZFS dataset, SFTP or portal upload with checksums, every read logged, a written policy sales can show a label.
• UPS on vinyl-core, gw-lpn-1, the switches, the edge collectors and the press PLCs (already required by facility security); NUT triggers a clean shutdown before the battery is exhausted.
• Admin access rides the mesh, as for every fleet machine; public SSH is never opened.

Procurement clauses

To go into every RFQ that leaves from roadmap items 4 to 8:

1. Presses (Newbilt, Pheenix): OPC UA (or S7/ADS) read access licensed and enabled; the full tag list with units and the cycle counter as a tag; a defined hand-off of the cycle counter to the inspection trigger; remote support through our mesh; no cloud agent as a warranty condition; PLC clock accepting NTP.
2. Boiler, chiller, compressor, heat pump (Thai vendors): Modbus RTU or TCP with the register map delivered; alarm contacts wired to a terminal strip for us.
3. HVAC contractor: a BMS with BACnet/IP or Modbus TCP read access and a point list; zone sensor wiring and room dP sensors in scope; sensor loops terminated in the edge cabinets.
4. Electrical contractor: Modbus energy meters on the feeders listed above; RS485 to the inverters and BESS BMS without cloud dongles; structured cabling, a VLAN-capable industrial switch in every cabinet, and Omada-managed switches and APs for the building so the gateway's existing Omada controller covers the site.
5. Galvanics (Digital Matrix, Pheenix Alpha): what the rectifiers, control rings and PLCs expose, and how; Modbus preferred; remote support through our mesh, no vendor cloud. Sibert: whether the VS machines can take a stamper serial in and write the centring result out, and which features of the fully automatic VSPa can move to the VSP.
6. Metrology (Keyence Thailand): confirmed CSV/3D export paths and whether the instrument software can write to a network share unattended.
7. Vision integrator: result string format, image push over SFTP, cycle counter correlation, and an acceptance test defined in records not in features.
8. Access control and fire: API or export for door events; dry contacts for every alarm class; BESS detection outputs.

Build order

Aligned with the roadmap phases.

Now (roadmap Phase 0 and 1): design and dev fleet.

• Put the procurement clauses above into the Newbilt re-quote, the Pheenix, Digital Matrix and Thai utility RFQs.
• In the clan repo: reserve the site id and VLAN table, write modules/vinyl/{core,metrics,edge,station}.nix, and stand up a dev trio at the Chiang Mai site behind gw-cnx-1 (a vinyl-core stand-in, one edge collector with a Modbus and OPC UA simulator, one station). This proves the modules, the mesh path and the proxy before a single plant machine exists, and the machines move to the plant when it does.
• Land the three router-module extensions on gw-cnx-1 first, where they are also useful.
• Stand up a dev ERPNext beside the dev trio (frappe-nix module, or the container fallback) and build the integration client against it from cnx-factory v0 onward.
• Start cnx-factory v0 against a press simulator (the state machine in the plant operation page is the seed: same states, same cycle timing) so orders, gates, genealogy and press cycles can be exercised end to end without hardware; package it as a flake from the first commit.
• Wire the Alertmanager and one real receiver; point control's vmalert at it.

Build and commission (roadmap Phase 3).

• gw-lpn-1, switches, vinyl-core and vinyl-backup installed before the machinery arrives, so vendor commissioning engineers plug into a finished network; ns1 gets the gateway's ACME key at the same time.
• Zone sensor loops, meters and edge cabinets installed by the electrical and HVAC contractors during fit-out; environment logging is live before pressing starts, which gives a baseline.
• Machinery commissioning includes data acceptance: every tag on the list readable, the cycle counter increments in the database with each cycle, data-silence alarms proven by pulling a cable.
• Application v1 live: intake, gates, bridge-stamper genealogy, press runs and cycles, PVC lot gate, packaging scan, shipments; stations at press, lab, packaging and shipping.
• Off-site backup destination live and a first restore drill done.

Launch (roadmap Phase 4).

• Optical inspection integrated and correlated; reject reasons flowing.
• Customer portal opened to the first labels through web01.
• Energy and environment dashboards used for the first PV and HVAC tuning.

Expand (roadmap Phase 5).

• vinyl-edge-galv, bath sensors, rectifier data, metrology ingestion at vinyl-st-metrology; genealogy switches from bridged to in-house.
• Audio QC station and pipeline; retirement signal live.
• Automated press line joins the same collector with the same tag family.
• Biomass boiler on the same Modbus path as its predecessor.

Rough cost

Planning estimates, September 2026, excluding the QC instruments (those are in quality automation) and excluding the vendors' OPC UA license options, which come back with the quotes. Fleet pieces already paid for (control, ns1, ns2, web01, the mesh) cost the plant nothing extra.

Block	Estimate (USD)
gw-lpn-1 (Topton 1U, fleet standard)	300 to 500
vinyl-core and vinyl-backup servers, UPS, rack	8,000 to 15,000
vinyl-erp server (none if it runs as a VM)	2,000 to 4,000
Omada switches and APs for the building, industrial switches in the cabinets, cabling allowance	4,000 to 8,000
3 edge collectors with I/O modules and enclosures	3,000 to 6,000
8 station kiosks with readers, scanners, one barcode verifier, one darkfield camera	8,000 to 15,000
zone environment sensors (12 to 16 points), room dP, reference PM instrument	4,000 to 8,000
energy meters (10 feeders), water meters, level transmitters	3,000 to 6,000
audio QC hardware (turntable, arm, cartridge, phono stage, interface)	2,000 to 5,000
off-site backup storage, per year	100 to 500
Hardware total	~34,000 to 68,000
cnx-factory to v1 (orders through shipments, stations, portal)	4 to 6 person-months
ERPNext: configuration with Thai tax, the integration and the cnx_erp app, Nix packaging	3.5 to 6 person-months, or a partner quote for the configuration
Clan modules, router extensions, collectors, dashboards and rules to commissioning	1 to 2 person-months
Vision, metrology and audio pipeline integration (Launch and Expand)	3 to 5 person-months, plus integrator fees in the vision quote

Headcount note: scale lists no IT or automation role in the 15 to 25 people. Running this needs one automation and IT engineer from Build onward, part-time at launch; add it to the headcount.

Open questions

• Site code and id: lpn / 3 assumed; confirm when the site is chosen and the fleet site table is updated.
• Semi-automatic press PLC: does the HRP 12 expose data, or is the sensor fallback needed? (Ask in the Newbilt re-quote.)
• Cycle counter hand-off to the vision system: PLC output, network message, or time correlation at the unloader? Decided with the vision integrator, tested at acceptance.
• HVAC BMS protocol and who owns the zone sensors: HVAC contractor or us.
• Off-site backup destination: Hetzner Storage Box (cheapest) or a storage VPS that is also a mesh peer (uniform tooling). Data residency for customer audio may argue for a Bangkok provider instead.
• Customer portal domain: the plant's brand domain (new zone on ns1/ns2, second cert on web01) or a name under cnx.network for launch. Internal testing has its own domain, cnxvinyl.lol (registered 2026-10-02, DNS on Cloudflare): a static demo site built from site/ and served by the buildfor.life web server, not a plant service. The brand domain decision stays open.
• Portal availability during a plant WAN outage: accept it (the portal is a view of live data) or add a read replica on the fleet side later.
• ERPNext (decided October 2026, see its section): own machine or a VM on vinyl-core; frappe-nix or the container fallback; Ecosoft or in-house for the configuration; payroll localisation; the BOI raw-material reporting; when the company's books move into it.
• Dead-wax code readability on black and coloured vinyl with a camera: test with samples before the packaging station is designed.