cnx_vinyl/docs/digital-factory.md

Digital Factory: Systems Plan

The traceability spec says what must be logged and the quality automation document says which instruments produce the numbers. This document is the plan for the systems in between: which computers, networks, collectors, databases and applications have to exist so that environmental data, machine data, QC results and the batch genealogy all land in one place, keyed to one record identity, and can be shown to a customer. It is a planning position as of September 2026; every cost figure is an estimate until a quote replaces it.

Standing decision: every service runs on NixOS, and the whole fleet is declared and deployed as one Clan. Servers, edge collectors, station kiosks, the router and the off-site VPS are all NixOS machines in a single inventory. Vendor instrument PCs (Keyence, vision controllers, PLC HMIs) are appliances, not servers: they export into the NixOS estate and never host a service of ours.

Design decisions

1. One clan, one repo, rebuildable from git. Machine configuration, service placement, mesh networking, backups, host monitoring and secrets (clan vars, sops-encrypted) live in one flake. A failed machine is replaced with clan machines install <name> from a fresh disk; a configuration change is clan machines update. Nothing is configured by hand on a running box.
2. One relational database for everything that has identity. PostgreSQL with TimescaleDB holds the genealogy, orders, gates, QC results, per-cycle press data, environment history and energy history. Joining "this record" to "the mold temperature on that cycle" and to "the humidity in the press hall that hour" is a query, not an export. Host and infrastructure metrics (CPU, disk, systemd, journals) go to the separate Prometheus/Loki pair that the clan monitoring service provides; they are not process data.
3. MQTT is the plant bus, Telegraf is the universal collector. Every machine, meter and sensor is read by a Telegraf instance on an edge collector near it (Modbus, OPC UA, Siemens S7 and MQTT inputs are stock plugins) and published to the broker on the core server. The core side persists everything to TimescaleDB with the same tool. Custom code exists only where no plugin does (vision results, instrument CSV ingestion, the audio QC pipeline).
4. Read-only toward machines. The data platform never writes to a PLC or an instrument. Control, safety and interlocks (fume extraction, steam shutoff, boiler safety chain) stay in the vendor PLCs and hard wiring, as facility security already requires.
5. Join keys are counters, not clocks. Every press cycle carries the machine's own cycle counter; every instrument result carries the part serial it was read from. Wall-clock time is recorded (UTC, NTP from the router) but is never the thing two systems are joined on.
6. Append-only, hash-chained. Process events, gate approvals and QC verdicts are written once and never updated; corrections are new events that reference the old. Each event carries the hash of the previous one so the log's integrity can be checked, which is what lets the customer portal be trusted.
7. The customer portal is a view, not a copy. It reads the same database through the same application with a customer role; there is no exported report to keep in sync.

The systems on one page

System	What it does	Runs on	Phase
Network and firewall	VLAN zoning (OT, sensors, servers, stations, office, CCTV, guest), dual WAN, NTP, DHCP/DNS	router-1 (NixOS, nftables) plus managed switches	Build
Clan fleet	Inventory of every NixOS machine, ZeroTier mesh, SSH and user management, secrets, backups, host monitoring	git repo, applied to all machines	Now (dev), Build (plant)
Plant data platform	PostgreSQL + TimescaleDB, MQTT broker, Telegraf persister, Grafana, S3-compatible object storage	core-1	Build
Factory application	Orders, gates, genealogy, materials, press runs, QC verdicts, packaging, fulfillment, customer portal, audit log	core-1	Build (v1), Launch (v2)
Edge collectors	Telegraf per machine zone, local store-and-forward, vision result receiver	edge-press, edge-utility, edge-galvanics	Build, Expand
Environmental monitoring	Zone temperature, humidity, PM, differential pressure; alarms; correlation with production	sensors on RS485 loops into edge-utility and edge-press	Build
Machine telemetry	Presses, extruders, hydraulics, boiler, chiller, compressor, heat pump, HVAC, PV, BESS, energy and water meters	edge collectors	Build
QC instrument integration	Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans	stations plus edge-press	Launch, Expand
Station kiosks	Badge check-in and scanning at each manual step; kiosk browser on the factory application	st-* machines	Build
Observability and alerting	Process alarms (Grafana on TimescaleDB), host alarms (clan monitoring), routing to phones, data-silence alarms	core-1	Build
Backup and DR	ZFS snapshots, Borg to backup-1 and off-site, PostgreSQL point-in-time recovery, restore drills	backup-1, vps-1	Build
Identity	OIDC single sign-on for staff, badge IDs for operators at stations, role model for customers	core-1 (Pocket ID)	Build
Client audio vault	Encrypted, isolated storage for pre-release audio with checksums and access logging	core-1 (separate dataset)	Build
Site systems integration	Access-control door events, fire panel and BESS detection contacts, UPS state, CCTV	edge-utility, optional nvr-1	Build
Off-site node	ZeroTier moon with a public IP, off-site backup target, public reverse proxy for the portal	vps-1	Build

Architecture

flowchart LR
    subgraph L0[Field]
        PLC[Press and extruder PLCs<br/>OPC UA / S7]
        UT[Boiler, chiller, compressor,<br/>heat pump, HVAC<br/>Modbus / BACnet]
        EN[Energy meters, Deye inverters,<br/>BESS, water meters<br/>Modbus / pulse]
        ENV[Zone T / RH / PM / dP sensors<br/>Modbus RTU]
        GV[Rectifiers, bath sensors<br/>Modbus]
        VIS[Vision controller<br/>TCP results, FTP images]
        INST[Keyence, XRF, lab instruments<br/>CSV export]
    end
    subgraph L1[Edge, NixOS]
        EP[edge-press<br/>Telegraf + vision receiver]
        EU[edge-utility<br/>Telegraf]
        EG[edge-galvanics<br/>Telegraf]
        ST[st-* stations<br/>kiosk, scanners, ingestion]
    end
    subgraph L2[Core, NixOS]
        MQ[MQTT broker]
        DB[(PostgreSQL + TimescaleDB)]
        OBJ[(Object storage)]
        APP[Factory application]
        GF[Grafana + alerting]
    end
    subgraph L3[Outside]
        VPS[vps-1: moon, backup, proxy]
        CUST[Customer portal]
        PH[Phones: alerts]
    end
    PLC --> EP
    VIS --> EP
    UT --> EU
    EN --> EU
    ENV --> EU
    GV --> EG
    INST --> ST
    EP --> MQ
    EU --> MQ
    EG --> MQ
    ST --> APP
    MQ --> DB
    MQ --> APP
    APP --> DB
    APP --> OBJ
    EP --> OBJ
    DB --> GF
    GF --> PH
    APP --> VPS --> CUST

Three rules keep this simple: field devices talk only to their edge collector (they are on VLANs with no route anywhere else); edge collectors talk only to the broker and object storage; the application and Grafana talk only to the database, the broker and object storage. The VPS reaches the core over the ZeroTier mesh, never over an open port.

Machines

All NixOS, all in the clan inventory. Hardware classes are planning positions; models are chosen at purchase.

Machine	Role	Hardware class	Where	Clan tags
router-1	Firewall, VLANs, DHCP/DNS, NTP (chrony), dual WAN failover	fanless x86, 4+ NICs	IT room	core
core-1	PostgreSQL + TimescaleDB, Mosquitto, Telegraf persister, factory app, Grafana, Garage object storage, Pocket ID, Caddy	1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD pool for objects, dual PSU on UPS	IT room	core, db
backup-1	Borg server, PostgreSQL streaming replica, second Garage node, ZFS receive target	tower or 2U, large HDD pool	separate room from core-1 (archive side of the building)	core, backup
vps-1	ZeroTier moon (public IP), off-site Borg destination, third Garage node, reverse proxy for the customer portal	small cloud VPS with block storage, outside Thailand or in Bangkok, decision open	cloud	offsite
edge-press	Telegraf (OPC UA / S7 to the presses and extruders, Modbus to hydraulics and label oven), vision result receiver, disk-backed buffer	fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485	press hall control cabinet	edge
edge-utility	Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS, water; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS	same class, more RS485	electrical room or utility yard cabinet	edge
edge-galvanics	Telegraf (rectifiers, bath sensors, extraction status)	same class, corrosion-protected enclosure outside the wet area	galvanics service corridor	edge (Expand phase)
st-intake	Office workstation for audio intake and gates (plain browser; not a kiosk)	any NixOS desktop	office	station
st-cutting	Session log entry, lacquer registration, scribe ID	mini PC, touch screen, badge reader	cutting room	station
st-press-1, st-press-2	Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons	mini PC, touch screen, badge reader, 2D scanner	at each press	station
st-lab	PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment	mini PC, RS232/USB to instruments	material lab	station
st-metrology	SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan	mini PC	metrology bench (Expand phase)	station
st-audioqc	Needle-drop capture and analysis pipeline	mini PC with audio interface, turntable, phono stage	listening/QC room (Expand phase)	station
st-pack-1	Runout code camera read, jacket barcode scan, BOM check, barcode verifier	mini PC, camera, scanner, verifier	packaging	station
st-ship	Consignment assembly, carrier label and tracking capture	mini PC, label printer	shipping	station
nvr-1	Frigate CCTV recording on its own VLAN (optional; a vendor NVR appliance is the alternative)	small server with GPU or Coral accelerator	IT room	cctv

Stations run a kiosk browser (cage plus Firefox or Chromium) pointed at the factory application; badge readers and scanners are USB HID keyboard devices, so there is no driver work. Fifteen machines at launch, about twenty at maturity; every one is a few lines in the inventory.

Networks

VLAN	Members	Rule
ot-press	press and extruder PLCs and HMIs, vision controller, edge-press	no internet, no route to other VLANs; edge-press is the only gateway
ot-utility	boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, edge-utility	same
ot-galvanics	rectifiers, bath transmitters, edge-galvanics	same
sensors	RS485 loops terminate on the edge collectors, so zone sensors have no IP presence; any IP sensor that is added lives here	no internet
servers	core-1, backup-1, router-1 management	reachable from stations, office, edge collectors and the mesh
stations	st-* kiosks, instrument PCs (Keyence, vision setup laptop)	reach core-1 only
office	staff laptops, printers	internet, core-1
cctv	cameras, nvr-1	no internet; NVR reachable from office by role
guest	visitors, vendor laptops	internet only
mgmt	switch and IPMI management interfaces	router-1 and core-1 admins only

Vendor remote support (Newbilt and Pheenix both offer remote monitoring) goes through a jump account on router-1 over the ZeroTier mesh, time-boxed and logged, with a port forward to the one PLC HMI they need. No machine ever gets a vendor's own cellular router or cloud agent on the OT VLAN.

Dual WAN: fibre plus a 4G/5G modem on router-1, failover only. The plant does not depend on the internet to run: everything in the tables above is on site except vps-1, and the only things that stop when the internet drops are the customer portal and off-site backup.

Clan inventory (sketch)

The plant clan in outline. Option names follow the clan release current at writing (26.05); check them against the pinned release before use, and expect the monitoring service to change, since it is still marked experimental upstream.

{
  meta.name = "cnx-vinyl";
  meta.domain = "cnx";              # mesh-only TLD; hosts resolve as <name>.cnx

  inventory.machines = {
    router-1       = { tags = [ "core" ]; };
    core-1         = { tags = [ "core" "db" "monitoring-server" ]; };
    backup-1       = { tags = [ "core" "backup" ]; };
    vps-1          = { tags = [ "offsite" ]; };
    edge-press     = { tags = [ "edge" ]; };
    edge-utility   = { tags = [ "edge" ]; };
    edge-galvanics = { tags = [ "edge" ]; };
    st-cutting     = { tags = [ "station" ]; };
    st-press-1     = { tags = [ "station" ]; };
    st-lab         = { tags = [ "station" ]; };
    st-pack-1      = { tags = [ "station" ]; };
    st-ship        = { tags = [ "station" ]; };
  };

  inventory.instances = {
    sshd.roles.server.tags.all = { };
    admin.roles.default.tags.all = { };          # admin users and keys

    zerotier = {
      roles.controller.machines.core-1 = { };
      roles.moon.machines.vps-1.settings.stableEndpoints = [ "<public-ip>/9993" ];
      roles.peer.tags.all = { };
    };

    borgbackup = {
      roles.server.machines.backup-1 = { };
      roles.server.machines.vps-1 = { };
      roles.client.tags.core = { };
      roles.client.tags.edge = { };
    };

    monitoring = {
      roles.server.machines.core-1.settings.grafana.enable = true;
      roles.client.tags.all = { };
    };
  };

  # Our own NixOS modules, applied by tag or by machine:
  #   modules/edge-collector.nix   Telegraf with the tag lists for that cabinet,
  #                                Mosquitto bridge with disk buffer
  #   modules/core-data.nix        PostgreSQL + TimescaleDB, Mosquitto, Telegraf
  #                                persister, Garage, Grafana, pgBackRest
  #   modules/factory-app.nix      the application service and its migrations
  #   modules/station-kiosk.nix    cage + browser, HID readers, auto-login
  #   modules/router.nix           nftables zoning, VLAN interfaces, chrony,
  #                                dual-WAN failover
}

What clan provides out of the box and we do not write ourselves: SSH and admin user management, the ZeroTier mesh (controller on core-1, the VPS as a publicly reachable moon so the plant and remote consultants find each other behind NAT), Borg backups with generated keys, host metrics and journal shipping with Grafana on the server role, and sops-encrypted secrets per machine. What we write: the five modules listed in the sketch and the factory application itself.

Data acquisition, source by source

Every row is also a purchasing clause: the "requirement" column is what goes into the RFQ, per the traceability principle that machine data access is bought, not retrofitted. Rates are planning values.

Presses and extruders

Source	Signals	Interface	Rate	Requirement to write into the RFQ
Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA)	cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes	OPC UA server on the PLC, read by Telegraf inputs.opcua	per cycle plus 1 Hz	OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty
Newbilt HRP 12 semi-automatic and VEX 50 extruder (PLC type unconfirmed)	as above where the PLC has them; on a relay-logic machine at least press closed/open, steam on/off, cycle count	OPC UA or Siemens S7 (inputs.s7comm); fallback: our own 4 to 20 mA sensors on a Modbus TCP I/O module and a cycle count from the mold-closed limit switch	per cycle plus 1 Hz	state what the PLC is and which of the two paths applies; the fallback sensor set is priced separately
Pheenix AD12 (Beckhoff PLC, remote monitoring supported)	same tag family as the Classic	OPC UA (Beckhoff TF6100) or ADS; inputs.opcua	per cycle plus 1 Hz	OPC UA license included, tag list delivered, remote monitoring routed through our jump host
Hydraulic aggregate	system pressure, oil temperature, tank level, filter differential pressure	via the press PLC if exposed, else own transmitters on the I/O module	1 Hz	ask whether the aggregate's sensors are on the PLC
Label dry oven	chamber temperature, timer state	Modbus if the controller has it, else a thermocouple on the I/O module	1/min
PVC grinder and regrind scale	run state, batch weight	scale with RS232 to st-press, run contact to I/O	per batch	scale must have a serial output

Utilities and energy

Source	Signals	Interface	Rate	Requirement
Steam boiler (Thai-sourced)	pressure, steam temperature, feedwater temperature, drum level, burner or heater state, fuel or kW, blowdown events, water treatment conductivity and hardness alarm	Modbus RTU or TCP on the boiler controller	1 Hz	Modbus register map delivered with the boiler; Thai boiler-regulation inspection data logged too
Steam accumulator	pressure	pressure transmitter to I/O	1 Hz
Chiller and cooling tower	supply and return temperatures, flow, compressor power, tower fan state, makeup water	Modbus	1 Hz	register map
Feedwater heat pump	source and sink temperatures, power	Modbus	1/10 s	register map
Compressor	line pressure, run hours, dew point	Modbus	1/10 s	register map
HVAC / BMS	AHU states, setpoints, supply temperatures, filter differential pressure, positive-pressure room dP	BACnet/IP or Modbus from the BMS the HVAC contractor supplies	1/min	the RFQ names BACnet/IP or Modbus TCP read access as a deliverable, with point list
Electrical	main incomer, feeders for press hall, chiller, HVAC, boiler, galvanics, packaging, IT; PV inverters (Deye, Modbus RTU/TCP register map is public); BESS state of charge, power, cell temperatures (through the Deye battery registers or the Gobel BMS RS485)	Modbus energy meters on each feeder; inverters direct, no cloud dongle	1/10 s	meters specified in the electrical design, not added later
Water	mains or well meter, process tank level, boiler and cooling tower makeup meters, galvanics rinse recycle, fire tank level	pulse counters and level transmitters on I/O	1/min	meters in the plumbing design

Environment

Zone	Sensors	Alarm tier
cutting room	T, RH, PM2.5/PM10, room dP	tight, positive pressure
galvanics clean room	T, RH, PM, room dP	tight, positive pressure
master and stamper archive	T, RH, PM, water leak	tightest
press hall (two points at launch, four at maturity)	T, RH, PM	process spec 20 to 26 °C, 35 to 60 % RH
material lab and PVC store	T, RH	process
packaging and finished goods	T, RH, PM	process
listening/QC room	T, RH	comfort
IT room	T, RH, water leak	tight
office	T, RH, PM	comfort
outdoor reference	T, RH, PM2.5	reference for burning-season episodes

Industrial RS485 Modbus RTU transmitters on wired loops to the nearest edge collector, one reading per minute, with a reference-grade PM instrument in the archive or cutting room to check the cheaper sensors against. Cheap ESP32/MQTT sensor nodes are acceptable for extra density in non-critical zones, never for the alarmed ones. Every zone alarm has two thresholds (warning, action) and a rate-of-change alarm for the archive.

Galvanics (Expand phase)

Rectifier current, voltage and ampere-hours per cell (Digital Matrix: ask what the control rings and rectifiers expose; Modbus is the requirement), bath temperature, pH and conductivity from transmitters, filtration and agitation pump states, fume extraction running status (the interlock itself stays hard-wired), rinse water conductivity and evaporator state. Titration results are entered at a station against the bath ID.

Site systems

Access-control door events (badge ID, door, direction, time) from the controller's API or export; fire alarm panel zone and fault contacts, BESS off-gas and early-smoke detection, UPS on-battery and low-battery (NUT on core-1 talking to the UPS) as dry contacts on edge-utility. CCTV stays on its own VLAN with its own retention; it is never joined to production data.

What produces no telemetry

The Agnew lathe and mastering chain, the Sibert stamper-prep machines and the shrink wrapper produce no machine data. They get station check-ins: who, when, which part, which parameters, with files attached (cutting parameters, DAW session export). The room environment around them is logged by the sensors above.

QC instrument integration

Instrument	How results reach the database	Key	Open integration item
Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration)	vision controller sends one result string per record over TCP to a receiver on edge-press; images pushed by FTP/SFTP to object storage	press ID + cycle counter, passed from the PLC to the vision trigger	the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test
Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer)	vendor Windows PCs export CSV and raw 3D to an SMB share on st-metrology; a watcher parses and posts	back-side Data Matrix serial, scanned at the bench before measurement	export format per instrument confirmed with Keyence Thailand
PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR)	instruments with RS232/USB export are read at st-lab; the rest are typed in with the printout photographed; CoA PDF attached	compound lot ID (printed label scanned at intake)	which benchtop models have serial export
Audio QC station	capture service on st-audioqc records the needle-drop (PipeWire, FLAC to object storage); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted	record key (stamper + press + cycle + side), typed or picked from the run list	first-of-kind pipeline, developed in Expand phase against launch-phase recordings
Packaging assembly check	camera with darkfield lighting reads the scribed dead-wax code (a handheld scanner will not read a scribed code on black vinyl reliably); handheld scanner reads the jacket barcode; barcode verifier grades it	order + record key	confirm dead-wax code readability with a sample before the packaging station is designed
Label print inspection	same vision family as the record label check; result per label lot	label lot ID	shared with the record inspection budget

The factory application

Working name cnx-factory. Server-rendered web application in Rust (axum, SQLx, minijinja templates, HTMX) on PostgreSQL, the stack B4L already uses for its other in-house tooling, with the same append-only ledger and hash-chained audit log pattern. It subscribes to the MQTT event topics (cycle complete, machine alarm, instrument result) and owns every write to the identity tables; Telegraf owns the writes to the time-series tables.

Modules, in build order:

1. Orders and releases: customer, order, release, sides, formats, run quantity, bill of materials (record, inner sleeve, jacket, inserts, shrink), due dates.
2. Audio intake and the client vault: upload via portal or SFTP with checksums, versions, handler, technical QA form; files in an encrypted, access-logged dataset separate from everything else.
3. The three artist gates: master plan, master, test pressing; each is an event with who approved, when, from which portal session or on whose verbal instruction, and nothing downstream is schedulable without it.
4. Mastering and cutting sessions: engineer, chain settings, lacquer IDs, dead-wax scribe text and fiducial applied, microscope check result.
5. Metal genealogy: lacquer to father to mother to stamper, or bridged supplier plus batch; per-part metrology history; stamper lifecycle (mounted on which press, play count, retired why), archive location.
6. Materials: PVC lots with the lab gate and CoA, regrind batches, label lots with print inspection and oven pre-dry, carton and sleeve lots with incoming QC.
7. Press runs and cycles: a run binds order, stamper pair, press, operator, shift, PVC lot in the hopper, regrind batch, label lot; each cycle event from the PLC becomes a row keyed by press + counter; the record key is derived from it per side.
8. QC verdicts: optical per record, audio per sampled record, lab per lot, packaging per unit; reject reasons; test-pressing hold state.
9. Finishing, packaging, fulfillment: assembly checks, consignment against the BOM, carrier and tracking per shipment or dropship end customer.
10. Customer portal: per-order timeline of everything above, scoped so one customer sees only its own orders; PDF export of the trace.
11. Reports and signals: yield per stamper, reject rate drift, audio score trend versus play count (the automatic retirement signal), energy per record, environment excursions overlaid on runs.

Core entities: customer, order, release, side, audio_asset, gate_approval, session, lacquer, father, mother, stamper, stamper_measurement, press, press_run, press_cycle, record, optical_result, audio_result, pvc_lot, regrind_batch, label_lot, carton_lot, lab_result, packaging_check, shipment, operator, shift, event (the hash chain). Time-series hypertables: telemetry (machine, tag, value), environment (zone, metric, value), energy (meter, kWh, kW), water.

Stations use the same application in kiosk mode: an operator taps a badge (the same card the access-control system issues, so one ID) and scans whatever the step asks for. No forms longer than one screen.

Observability and alerting

Two tiers, one Grafana:

• Process alarms from TimescaleDB: zone environment excursions and rate of change, boiler pressure and level, chiller return temperature, compressed-air pressure, hydraulic pressure, reject rate per run above baseline, stamper approaching its play-count limit, PV or BESS faults, fire and BESS detection contacts, UPS on battery.
• Infrastructure alarms from the clan monitoring stack: host down, disk, service failed, backup missed, certificate expiry.
• Data-silence alarms in both tiers: a collector that has published nothing for five minutes is an alarm, whatever the last value was.

Routing: Telegram or the LINE Messaging API for the shift group, email for the daily digest, SMS through a Thai gateway for the critical set (boiler, fire, BESS, archive climate) with an on-call rota. The animated plant operation page is the design target for the live overview: once the telemetry tables exist it is fed from Grafana or directly from the database instead of its simulation.

Backup, retention and recovery

• Hourly ZFS snapshots on core-1 (sanoid), replicated to backup-1 (syncoid).
• Nightly Borg from every core and edge machine to backup-1 and to vps-1 (clan borgbackup, two destinations).
• PostgreSQL point-in-time recovery: pgBackRest with WAL archiving to object storage, replicated to the off-site node; a streaming replica on backup-1 for fast failover.
• Object storage (Garage) with three nodes: core-1, backup-1, vps-1; audio captures, inspection images, metrology raw data and CoA scans survive the loss of any one node.
• Configuration: the clan repo in Gitea, mirrored off-site; any machine is rebuilt from it.
• Restore drill quarterly: restore the database to a scratch VM and open a random order's trace in the portal. A backup that has never been restored does not count.

Data	Retention
genealogy, orders, gates, QC verdicts, event log	life of the customer relationship, never deleted while the metal parts exist
per-cycle press data	forever (small)
1 Hz machine telemetry	raw 2 years, then 1-minute aggregates forever
zone environment	raw 5 years
energy and water	forever (small)
audio captures and inspection images	2 years, longer for any record with a complaint
client audio	per contract; default deleted 12 months after final delivery, or on request
access-control events	1 year
CCTV	30 days
host logs and metrics	90 days

PDPA applies to badge events, CCTV and customer contact data: a written retention policy, role-based access, and export/deletion on request. The customer portal shows operator roles, never operator names.

Security

• OT VLANs have no internet and no inbound routes; edge collectors are the only bridge and they only publish.
• Staff sign-on through Pocket ID (OIDC, passkeys) with roles: admin, engineer, operator, sales, customer. Stations authenticate by badge for attribution, not for privilege.
• Secrets only in clan vars; nothing in plain text in the repo.
• Client audio: separate encrypted ZFS dataset, SFTP or portal upload with checksums, every read logged, a written policy sales can show a label.
• UPS on core-1, router-1, the switches, the edge collectors and the press PLCs (already required by facility security); NUT triggers a clean shutdown before the battery is exhausted.
• Signed commits on the clan repo; deployments only from a reviewed branch.

Procurement clauses

To go into every RFQ that leaves from roadmap items 4 to 8:

1. Presses (Newbilt, Pheenix): OPC UA (or S7/ADS) read access licensed and enabled; the full tag list with units and the cycle counter as a tag; a defined hand-off of the cycle counter to the inspection trigger; remote support through our jump host; no cloud agent as a warranty condition; PLC clock accepting NTP.
2. Boiler, chiller, compressor, heat pump (Thai vendors): Modbus RTU or TCP with the register map delivered; alarm contacts wired to a terminal strip for us.
3. HVAC contractor: a BMS with BACnet/IP or Modbus TCP read access and a point list; zone sensor wiring and room dP sensors in scope; sensor loops terminated in the edge cabinets.
4. Electrical contractor: Modbus energy meters on the feeders listed above; RS485 to the inverters and BESS BMS without cloud dongles; structured cabling and a VLAN-capable switch in every cabinet.
5. Galvanics (Digital Matrix): what the rectifiers and control rings expose, and how; Modbus preferred.
6. Metrology (Keyence Thailand): confirmed CSV/3D export paths and whether the instrument software can write to a network share unattended.
7. Vision integrator: result string format, image push, cycle counter correlation, and an acceptance test defined in records not in features.
8. Access control and fire: API or export for door events; dry contacts for every alarm class; BESS detection outputs.

Build order

Aligned with the roadmap phases.

Now (roadmap Phase 0 and 1): design and dev fleet.

• Put the procurement clauses above into the Newbilt re-quote, the Pheenix, Digital Matrix and Thai utility RFQs.
• Create the clan repo and a development clan of three small machines at the office: a core-1 stand-in, one edge collector with a Modbus and OPC UA simulator, one station. This proves the tooling before a single plant machine exists.
• Build the factory application v0 against a press simulator (the state machine in the plant operation page is the seed: same states, same cycle timing) so orders, gates, genealogy and press cycles can be exercised end to end without hardware.
• Decide the repository layout: this repository stays documents; the clan flake and the application get their own repositories in the same Gitea organisation, linked from here (see open questions).

Build and commission (roadmap Phase 3).

• Network, router-1, core-1, backup-1 and vps-1 installed before the machinery arrives, so vendor commissioning engineers plug into a finished network.
• Zone sensor loops, meters and edge cabinets installed by the electrical and HVAC contractors during fit-out; environment logging is live before pressing starts, which gives a baseline.
• Machinery commissioning includes data acceptance: every tag on the list readable, the cycle counter increments in the database with each cycle, data-silence alarms proven by pulling a cable.
• Application v1 live: intake, gates, bridge-stamper genealogy, press runs and cycles, PVC lot gate, packaging scan, shipments; stations at press, lab, packaging and shipping.

Launch (roadmap Phase 4).

• Optical inspection integrated and correlated; reject reasons flowing.
• Customer portal opened to the first labels.
• Energy and environment dashboards used for the first PV and HVAC tuning.

Expand (roadmap Phase 5).

• edge-galvanics, bath sensors, rectifier data, metrology ingestion at st-metrology; genealogy switches from bridged to in-house.
• Audio QC station and pipeline; retirement signal live.
• Automated press line joins the same collector with the same tag family.
• Biomass boiler on the same Modbus path as its predecessor.

Rough cost

Planning estimates, September 2026, excluding the QC instruments (those are in quality automation) and excluding the vendors' OPC UA license options, which come back with the quotes.

Block	Estimate (USD)
core-1 and backup-1 servers, UPS, rack	8,000 to 15,000
router-1, managed switches (industrial in the cabinets), cabling allowance	4,000 to 8,000
3 edge collectors with I/O modules and enclosures	3,000 to 6,000
8 station kiosks with readers, scanners, one barcode verifier, one darkfield camera	8,000 to 15,000
zone environment sensors (12 to 16 points), room dP, reference PM instrument	4,000 to 8,000
energy meters (10 feeders), water meters, level transmitters	3,000 to 6,000
audio QC hardware (turntable, arm, cartridge, phono stage, interface)	2,000 to 5,000
vps-1 and off-site storage, per year	500 to 1,500
Hardware total	~32,000 to 64,000
Application development to v1 (orders through shipments, stations, portal)	4 to 6 person-months
Clan fleet, modules and collectors to commissioning	1 to 2 person-months
Vision, metrology and audio pipeline integration (Launch and Expand)	3 to 5 person-months, plus integrator fees in the vision quote

Headcount note: scale lists no IT or automation role in the 15 to 25 people. Running this needs one automation and IT engineer from Build onward, part-time at launch; add it to the headcount.

Open questions

• Semi-automatic press PLC: does the HRP 12 expose data, or is the sensor fallback needed? (Ask in the Newbilt re-quote.)
• Cycle counter hand-off to the vision system: PLC output, network message, or time correlation at the unloader? Decided with the vision integrator, tested at acceptance.
• HVAC BMS protocol and who owns the zone sensors: HVAC contractor or us.
• Where vps-1 lives: outside Thailand (latency to the mesh, data residency for customer data) or a Bangkok provider.
• Portal availability: served through the mesh from core-1 (down when the plant's internet is down) or a read replica on vps-1 (more moving parts). Start with the former.
• Accounting and ERP: out of scope for the application; if an ERP is adopted later it references the same order IDs. Decide after launch.
• Repository layout: documents here, cnx-infra (clan) and cnx-factory (application) as separate repositories, or everything in one. The README currently promises this repository will hold the software; a link satisfies that.
• Dead-wax code readability on black and coloured vinyl with a camera: test with samples before the packaging station is designed.