# Digital Factory: Systems Plan

The [traceability spec](traceability.md) says what must be logged and the
[quality automation](quality-automation.md) document says which instruments
produce the numbers. This document is the plan for the systems in between:
which computers, networks, collectors, databases and applications have to
exist so that environmental data, machine data, QC results and the batch
genealogy all land in one place, keyed to one record identity, and can be
shown to a customer. It is a planning position as of September 2026; every
cost figure is an estimate until a quote replaces it.

**Standing decision: every service runs on NixOS, and the whole fleet is
declared and deployed as one [Clan](https://clan.lol).** Servers, edge
collectors, station kiosks, the router and the off-site VPS are all NixOS
machines in a single inventory. Vendor instrument PCs (Keyence, vision
controllers, PLC HMIs) are appliances, not servers: they export into the
NixOS estate and never host a service of ours.

## Design decisions

1. **One clan, one repo, rebuildable from git.** Machine configuration,
   service placement, mesh networking, backups, host monitoring and
   secrets (clan vars, sops-encrypted) live in one flake. A failed machine
   is replaced with `clan machines install <name>` from a fresh disk; a
   configuration change is `clan machines update`. Nothing is configured by
   hand on a running box.
2. **One relational database for everything that has identity.**
   PostgreSQL with TimescaleDB holds the genealogy, orders, gates, QC
   results, per-cycle press data, environment history and energy history.
   Joining "this record" to "the mold temperature on that cycle" and to
   "the humidity in the press hall that hour" is a query, not an export.
   Host and infrastructure metrics (CPU, disk, systemd, journals) go to
   the separate Prometheus/Loki pair that the clan monitoring service
   provides; they are not process data.
3. **MQTT is the plant bus, Telegraf is the universal collector.** Every
   machine, meter and sensor is read by a Telegraf instance on an edge
   collector near it (Modbus, OPC UA, Siemens S7 and MQTT inputs are
   stock plugins) and published to the broker on the core server. The
   core side persists everything to TimescaleDB with the same tool. Custom
   code exists only where no plugin does (vision results, instrument CSV
   ingestion, the audio QC pipeline).
4. **Read-only toward machines.** The data platform never writes to a PLC
   or an instrument. Control, safety and interlocks (fume extraction, steam
   shutoff, boiler safety chain) stay in the vendor PLCs and hard wiring,
   as [facility security](facility-security.md) already requires.
5. **Join keys are counters, not clocks.** Every press cycle carries the
   machine's own cycle counter; every instrument result carries the part
   serial it was read from. Wall-clock time is recorded (UTC, NTP from the
   router) but is never the thing two systems are joined on.
6. **Append-only, hash-chained.** Process events, gate approvals and QC
   verdicts are written once and never updated; corrections are new events
   that reference the old. Each event carries the hash of the previous one
   so the log's integrity can be checked, which is what lets the customer
   portal be trusted.
7. **The customer portal is a view, not a copy.** It reads the same
   database through the same application with a customer role; there is
   no exported report to keep in sync.

## The systems on one page

| System | What it does | Runs on | Phase |
|---|---|---|---|
| Network and firewall | VLAN zoning (OT, sensors, servers, stations, office, CCTV, guest), dual WAN, NTP, DHCP/DNS | `router-1` (NixOS, nftables) plus managed switches | Build |
| Clan fleet | Inventory of every NixOS machine, ZeroTier mesh, SSH and user management, secrets, backups, host monitoring | git repo, applied to all machines | Now (dev), Build (plant) |
| Plant data platform | PostgreSQL + TimescaleDB, MQTT broker, Telegraf persister, Grafana, S3-compatible object storage | `core-1` | Build |
| Factory application | Orders, gates, genealogy, materials, press runs, QC verdicts, packaging, fulfillment, customer portal, audit log | `core-1` | Build (v1), Launch (v2) |
| Edge collectors | Telegraf per machine zone, local store-and-forward, vision result receiver | `edge-press`, `edge-utility`, `edge-galvanics` | Build, Expand |
| Environmental monitoring | Zone temperature, humidity, PM, differential pressure; alarms; correlation with production | sensors on RS485 loops into `edge-utility` and `edge-press` | Build |
| Machine telemetry | Presses, extruders, hydraulics, boiler, chiller, compressor, heat pump, HVAC, PV, BESS, energy and water meters | edge collectors | Build |
| QC instrument integration | Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans | stations plus `edge-press` | Launch, Expand |
| Station kiosks | Badge check-in and scanning at each manual step; kiosk browser on the factory application | `st-*` machines | Build |
| Observability and alerting | Process alarms (Grafana on TimescaleDB), host alarms (clan monitoring), routing to phones, data-silence alarms | `core-1` | Build |
| Backup and DR | ZFS snapshots, Borg to `backup-1` and off-site, PostgreSQL point-in-time recovery, restore drills | `backup-1`, `vps-1` | Build |
| Identity | OIDC single sign-on for staff, badge IDs for operators at stations, role model for customers | `core-1` (Pocket ID) | Build |
| Client audio vault | Encrypted, isolated storage for pre-release audio with checksums and access logging | `core-1` (separate dataset) | Build |
| Site systems integration | Access-control door events, fire panel and BESS detection contacts, UPS state, CCTV | `edge-utility`, optional `nvr-1` | Build |
| Off-site node | ZeroTier moon with a public IP, off-site backup target, public reverse proxy for the portal | `vps-1` | Build |

## Architecture

```mermaid
flowchart LR
    subgraph L0[Field]
        PLC[Press and extruder PLCs<br/>OPC UA / S7]
        UT[Boiler, chiller, compressor,<br/>heat pump, HVAC<br/>Modbus / BACnet]
        EN[Energy meters, Deye inverters,<br/>BESS, water meters<br/>Modbus / pulse]
        ENV[Zone T / RH / PM / dP sensors<br/>Modbus RTU]
        GV[Rectifiers, bath sensors<br/>Modbus]
        VIS[Vision controller<br/>TCP results, FTP images]
        INST[Keyence, XRF, lab instruments<br/>CSV export]
    end
    subgraph L1[Edge, NixOS]
        EP[edge-press<br/>Telegraf + vision receiver]
        EU[edge-utility<br/>Telegraf]
        EG[edge-galvanics<br/>Telegraf]
        ST[st-* stations<br/>kiosk, scanners, ingestion]
    end
    subgraph L2[Core, NixOS]
        MQ[MQTT broker]
        DB[(PostgreSQL + TimescaleDB)]
        OBJ[(Object storage)]
        APP[Factory application]
        GF[Grafana + alerting]
    end
    subgraph L3[Outside]
        VPS[vps-1: moon, backup, proxy]
        CUST[Customer portal]
        PH[Phones: alerts]
    end
    PLC --> EP
    VIS --> EP
    UT --> EU
    EN --> EU
    ENV --> EU
    GV --> EG
    INST --> ST
    EP --> MQ
    EU --> MQ
    EG --> MQ
    ST --> APP
    MQ --> DB
    MQ --> APP
    APP --> DB
    APP --> OBJ
    EP --> OBJ
    DB --> GF
    GF --> PH
    APP --> VPS --> CUST
```

Three rules keep this simple: field devices talk only to their edge
collector (they are on VLANs with no route anywhere else); edge collectors
talk only to the broker and object storage; the application and Grafana
talk only to the database, the broker and object storage. The VPS reaches
the core over the ZeroTier mesh, never over an open port.

## Machines

All NixOS, all in the clan inventory. Hardware classes are planning
positions; models are chosen at purchase.

| Machine | Role | Hardware class | Where | Clan tags |
|---|---|---|---|---|
| `router-1` | Firewall, VLANs, DHCP/DNS, NTP (chrony), dual WAN failover | fanless x86, 4+ NICs | IT room | `core` |
| `core-1` | PostgreSQL + TimescaleDB, Mosquitto, Telegraf persister, factory app, Grafana, Garage object storage, Pocket ID, Caddy | 1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD pool for objects, dual PSU on UPS | IT room | `core`, `db` |
| `backup-1` | Borg server, PostgreSQL streaming replica, second Garage node, ZFS receive target | tower or 2U, large HDD pool | separate room from `core-1` (archive side of the building) | `core`, `backup` |
| `vps-1` | ZeroTier moon (public IP), off-site Borg destination, third Garage node, reverse proxy for the customer portal | small cloud VPS with block storage, outside Thailand or in Bangkok, decision open | cloud | `offsite` |
| `edge-press` | Telegraf (OPC UA / S7 to the presses and extruders, Modbus to hydraulics and label oven), vision result receiver, disk-backed buffer | fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485 | press hall control cabinet | `edge` |
| `edge-utility` | Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS, water; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS | same class, more RS485 | electrical room or utility yard cabinet | `edge` |
| `edge-galvanics` | Telegraf (rectifiers, bath sensors, extraction status) | same class, corrosion-protected enclosure outside the wet area | galvanics service corridor | `edge` (Expand phase) |
| `st-intake` | Office workstation for audio intake and gates (plain browser; not a kiosk) | any NixOS desktop | office | `station` |
| `st-cutting` | Session log entry, lacquer registration, scribe ID | mini PC, touch screen, badge reader | cutting room | `station` |
| `st-press-1`, `st-press-2` | Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons | mini PC, touch screen, badge reader, 2D scanner | at each press | `station` |
| `st-lab` | PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment | mini PC, RS232/USB to instruments | material lab | `station` |
| `st-metrology` | SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan | mini PC | metrology bench (Expand phase) | `station` |
| `st-audioqc` | Needle-drop capture and analysis pipeline | mini PC with audio interface, turntable, phono stage | listening/QC room (Expand phase) | `station` |
| `st-pack-1` | Runout code camera read, jacket barcode scan, BOM check, barcode verifier | mini PC, camera, scanner, verifier | packaging | `station` |
| `st-ship` | Consignment assembly, carrier label and tracking capture | mini PC, label printer | shipping | `station` |
| `nvr-1` | Frigate CCTV recording on its own VLAN (optional; a vendor NVR appliance is the alternative) | small server with GPU or Coral accelerator | IT room | `cctv` |

Stations run a kiosk browser (`cage` plus Firefox or Chromium) pointed at
the factory application; badge readers and scanners are USB HID keyboard
devices, so there is no driver work. Fifteen machines at launch, about
twenty at maturity; every one is a few lines in the inventory.

## Networks

| VLAN | Members | Rule |
|---|---|---|
| `ot-press` | press and extruder PLCs and HMIs, vision controller, `edge-press` | no internet, no route to other VLANs; `edge-press` is the only gateway |
| `ot-utility` | boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, `edge-utility` | same |
| `ot-galvanics` | rectifiers, bath transmitters, `edge-galvanics` | same |
| `sensors` | RS485 loops terminate on the edge collectors, so zone sensors have no IP presence; any IP sensor that is added lives here | no internet |
| `servers` | `core-1`, `backup-1`, `router-1` management | reachable from `stations`, `office`, edge collectors and the mesh |
| `stations` | `st-*` kiosks, instrument PCs (Keyence, vision setup laptop) | reach `core-1` only |
| `office` | staff laptops, printers | internet, `core-1` |
| `cctv` | cameras, `nvr-1` | no internet; NVR reachable from `office` by role |
| `guest` | visitors, vendor laptops | internet only |
| `mgmt` | switch and IPMI management interfaces | `router-1` and `core-1` admins only |

Vendor remote support (Newbilt and Pheenix both offer remote monitoring) goes
through a jump account on `router-1` over the ZeroTier mesh, time-boxed and
logged, with a port forward to the one PLC HMI they need. No machine ever
gets a vendor's own cellular router or cloud agent on the OT VLAN.

Dual WAN: fibre plus a 4G/5G modem on `router-1`, failover only. The plant
does not depend on the internet to run: everything in the tables above is
on site except `vps-1`, and the only things that stop when the internet
drops are the customer portal and off-site backup.

## Clan inventory (sketch)

The plant clan in outline. Option names follow the clan release current at
writing (26.05); check them against the pinned release before use, and
expect the monitoring service to change, since it is still marked
experimental upstream.

```nix
{
  meta.name = "cnx-vinyl";
  meta.domain = "cnx";              # mesh-only TLD; hosts resolve as <name>.cnx

  inventory.machines = {
    router-1       = { tags = [ "core" ]; };
    core-1         = { tags = [ "core" "db" "monitoring-server" ]; };
    backup-1       = { tags = [ "core" "backup" ]; };
    vps-1          = { tags = [ "offsite" ]; };
    edge-press     = { tags = [ "edge" ]; };
    edge-utility   = { tags = [ "edge" ]; };
    edge-galvanics = { tags = [ "edge" ]; };
    st-cutting     = { tags = [ "station" ]; };
    st-press-1     = { tags = [ "station" ]; };
    st-lab         = { tags = [ "station" ]; };
    st-pack-1      = { tags = [ "station" ]; };
    st-ship        = { tags = [ "station" ]; };
  };

  inventory.instances = {
    sshd.roles.server.tags.all = { };
    admin.roles.default.tags.all = { };          # admin users and keys

    zerotier = {
      roles.controller.machines.core-1 = { };
      roles.moon.machines.vps-1.settings.stableEndpoints = [ "<public-ip>/9993" ];
      roles.peer.tags.all = { };
    };

    borgbackup = {
      roles.server.machines.backup-1 = { };
      roles.server.machines.vps-1 = { };
      roles.client.tags.core = { };
      roles.client.tags.edge = { };
    };

    monitoring = {
      roles.server.machines.core-1.settings.grafana.enable = true;
      roles.client.tags.all = { };
    };
  };

  # Our own NixOS modules, applied by tag or by machine:
  #   modules/edge-collector.nix   Telegraf with the tag lists for that cabinet,
  #                                Mosquitto bridge with disk buffer
  #   modules/core-data.nix        PostgreSQL + TimescaleDB, Mosquitto, Telegraf
  #                                persister, Garage, Grafana, pgBackRest
  #   modules/factory-app.nix      the application service and its migrations
  #   modules/station-kiosk.nix    cage + browser, HID readers, auto-login
  #   modules/router.nix           nftables zoning, VLAN interfaces, chrony,
  #                                dual-WAN failover
}
```

What clan provides out of the box and we do not write ourselves: SSH and
admin user management, the ZeroTier mesh (controller on `core-1`, the VPS
as a publicly reachable moon so the plant and remote consultants find each
other behind NAT), Borg backups with generated keys, host metrics and
journal shipping with Grafana on the server role, and sops-encrypted
secrets per machine. What we write: the five modules listed in the sketch
and the factory application itself.

## Data acquisition, source by source

Every row is also a purchasing clause: the "requirement" column is what
goes into the RFQ, per the traceability principle that machine data access
is bought, not retrofitted. Rates are planning values.

### Presses and extruders

| Source | Signals | Interface | Rate | Requirement to write into the RFQ |
|---|---|---|---|---|
| Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA) | cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes | OPC UA server on the PLC, read by Telegraf `inputs.opcua` | per cycle plus 1 Hz | OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty |
| Newbilt HRP 12 semi-automatic and VEX 50 extruder (PLC type unconfirmed) | as above where the PLC has them; on a relay-logic machine at least press closed/open, steam on/off, cycle count | OPC UA or Siemens S7 (`inputs.s7comm`); fallback: our own 4 to 20 mA sensors on a Modbus TCP I/O module and a cycle count from the mold-closed limit switch | per cycle plus 1 Hz | state what the PLC is and which of the two paths applies; the fallback sensor set is priced separately |
| Pheenix AD12 (Beckhoff PLC, remote monitoring supported) | same tag family as the Classic | OPC UA (Beckhoff TF6100) or ADS; `inputs.opcua` | per cycle plus 1 Hz | OPC UA license included, tag list delivered, remote monitoring routed through our jump host |
| Hydraulic aggregate | system pressure, oil temperature, tank level, filter differential pressure | via the press PLC if exposed, else own transmitters on the I/O module | 1 Hz | ask whether the aggregate's sensors are on the PLC |
| Label dry oven | chamber temperature, timer state | Modbus if the controller has it, else a thermocouple on the I/O module | 1/min | |
| PVC grinder and regrind scale | run state, batch weight | scale with RS232 to `st-press`, run contact to I/O | per batch | scale must have a serial output |

### Utilities and energy

| Source | Signals | Interface | Rate | Requirement |
|---|---|---|---|---|
| Steam boiler (Thai-sourced) | pressure, steam temperature, feedwater temperature, drum level, burner or heater state, fuel or kW, blowdown events, water treatment conductivity and hardness alarm | Modbus RTU or TCP on the boiler controller | 1 Hz | Modbus register map delivered with the boiler; Thai boiler-regulation inspection data logged too |
| Steam accumulator | pressure | pressure transmitter to I/O | 1 Hz | |
| Chiller and cooling tower | supply and return temperatures, flow, compressor power, tower fan state, makeup water | Modbus | 1 Hz | register map |
| Feedwater heat pump | source and sink temperatures, power | Modbus | 1/10 s | register map |
| Compressor | line pressure, run hours, dew point | Modbus | 1/10 s | register map |
| HVAC / BMS | AHU states, setpoints, supply temperatures, filter differential pressure, positive-pressure room dP | BACnet/IP or Modbus from the BMS the HVAC contractor supplies | 1/min | the RFQ names BACnet/IP or Modbus TCP read access as a deliverable, with point list |
| Electrical | main incomer, feeders for press hall, chiller, HVAC, boiler, galvanics, packaging, IT; PV inverters (Deye, Modbus RTU/TCP register map is public); BESS state of charge, power, cell temperatures (through the Deye battery registers or the Gobel BMS RS485) | Modbus energy meters on each feeder; inverters direct, no cloud dongle | 1/10 s | meters specified in the electrical design, not added later |
| Water | mains or well meter, process tank level, boiler and cooling tower makeup meters, galvanics rinse recycle, fire tank level | pulse counters and level transmitters on I/O | 1/min | meters in the plumbing design |

### Environment

| Zone | Sensors | Alarm tier |
|---|---|---|
| cutting room | T, RH, PM2.5/PM10, room dP | tight, positive pressure |
| galvanics clean room | T, RH, PM, room dP | tight, positive pressure |
| master and stamper archive | T, RH, PM, water leak | tightest |
| press hall (two points at launch, four at maturity) | T, RH, PM | process spec 20 to 26 °C, 35 to 60 % RH |
| material lab and PVC store | T, RH | process |
| packaging and finished goods | T, RH, PM | process |
| listening/QC room | T, RH | comfort |
| IT room | T, RH, water leak | tight |
| office | T, RH, PM | comfort |
| outdoor reference | T, RH, PM2.5 | reference for burning-season episodes |

Industrial RS485 Modbus RTU transmitters on wired loops to the nearest edge
collector, one reading per minute, with a reference-grade PM instrument in
the archive or cutting room to check the cheaper sensors against. Cheap
ESP32/MQTT sensor nodes are acceptable for extra density in non-critical
zones, never for the alarmed ones. Every zone alarm has two thresholds
(warning, action) and a rate-of-change alarm for the archive.

### Galvanics (Expand phase)

Rectifier current, voltage and ampere-hours per cell (Digital Matrix: ask
what the control rings and rectifiers expose; Modbus is the requirement),
bath temperature, pH and conductivity from transmitters, filtration and
agitation pump states, fume extraction running status (the interlock itself
stays hard-wired), rinse water conductivity and evaporator state. Titration
results are entered at a station against the bath ID.

### Site systems

Access-control door events (badge ID, door, direction, time) from the
controller's API or export; fire alarm panel zone and fault contacts, BESS
off-gas and early-smoke detection, UPS on-battery and low-battery (NUT on
`core-1` talking to the UPS) as dry contacts on `edge-utility`. CCTV stays
on its own VLAN with its own retention; it is never joined to production
data.

### What produces no telemetry

The Agnew lathe and mastering chain, the Sibert stamper-prep machines and
the shrink wrapper produce no machine data. They get station check-ins:
who, when, which part, which parameters, with files attached (cutting
parameters, DAW session export). The room environment around them is
logged by the sensors above.

## QC instrument integration

| Instrument | How results reach the database | Key | Open integration item |
|---|---|---|---|
| Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration) | vision controller sends one result string per record over TCP to a receiver on `edge-press`; images pushed by FTP/SFTP to object storage | press ID + cycle counter, passed from the PLC to the vision trigger | the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test |
| Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer) | vendor Windows PCs export CSV and raw 3D to an SMB share on `st-metrology`; a watcher parses and posts | back-side Data Matrix serial, scanned at the bench before measurement | export format per instrument confirmed with Keyence Thailand |
| PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR) | instruments with RS232/USB export are read at `st-lab`; the rest are typed in with the printout photographed; CoA PDF attached | compound lot ID (printed label scanned at intake) | which benchtop models have serial export |
| Audio QC station | capture service on `st-audioqc` records the needle-drop (PipeWire, FLAC to object storage); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted | record key (stamper + press + cycle + side), typed or picked from the run list | first-of-kind pipeline, developed in Expand phase against launch-phase recordings |
| Packaging assembly check | camera with darkfield lighting reads the scribed dead-wax code (a handheld scanner will not read a scribed code on black vinyl reliably); handheld scanner reads the jacket barcode; barcode verifier grades it | order + record key | confirm dead-wax code readability with a sample before the packaging station is designed |
| Label print inspection | same vision family as the record label check; result per label lot | label lot ID | shared with the record inspection budget |

## The factory application

Working name `cnx-factory`. Server-rendered web application in Rust (axum,
SQLx, minijinja templates, HTMX) on PostgreSQL, the stack B4L already uses
for its other in-house tooling, with the same append-only ledger and
hash-chained audit log pattern. It subscribes to the MQTT event topics
(cycle complete, machine alarm, instrument result) and owns every write to
the identity tables; Telegraf owns the writes to the time-series tables.

Modules, in build order:

1. **Orders and releases**: customer, order, release, sides, formats, run
   quantity, bill of materials (record, inner sleeve, jacket, inserts,
   shrink), due dates.
2. **Audio intake and the client vault**: upload via portal or SFTP with
   checksums, versions, handler, technical QA form; files in an encrypted,
   access-logged dataset separate from everything else.
3. **The three artist gates**: master plan, master, test pressing; each is
   an event with who approved, when, from which portal session or on whose
   verbal instruction, and nothing downstream is schedulable without it.
4. **Mastering and cutting sessions**: engineer, chain settings, lacquer
   IDs, dead-wax scribe text and fiducial applied, microscope check result.
5. **Metal genealogy**: lacquer to father to mother to stamper, or bridged
   supplier plus batch; per-part metrology history; stamper lifecycle
   (mounted on which press, play count, retired why), archive location.
6. **Materials**: PVC lots with the lab gate and CoA, regrind batches,
   label lots with print inspection and oven pre-dry, carton and sleeve
   lots with incoming QC.
7. **Press runs and cycles**: a run binds order, stamper pair, press,
   operator, shift, PVC lot in the hopper, regrind batch, label lot; each
   cycle event from the PLC becomes a row keyed by press + counter; the
   record key is derived from it per side.
8. **QC verdicts**: optical per record, audio per sampled record, lab per
   lot, packaging per unit; reject reasons; test-pressing hold state.
9. **Finishing, packaging, fulfillment**: assembly checks, consignment
   against the BOM, carrier and tracking per shipment or dropship end
   customer.
10. **Customer portal**: per-order timeline of everything above, scoped so
    one customer sees only its own orders; PDF export of the trace.
11. **Reports and signals**: yield per stamper, reject rate drift, audio
    score trend versus play count (the automatic retirement signal),
    energy per record, environment excursions overlaid on runs.

Core entities: `customer`, `order`, `release`, `side`, `audio_asset`,
`gate_approval`, `session`, `lacquer`, `father`, `mother`, `stamper`,
`stamper_measurement`, `press`, `press_run`, `press_cycle`, `record`,
`optical_result`, `audio_result`, `pvc_lot`, `regrind_batch`, `label_lot`,
`carton_lot`, `lab_result`, `packaging_check`, `shipment`, `operator`,
`shift`, `event` (the hash chain). Time-series hypertables: `telemetry`
(machine, tag, value), `environment` (zone, metric, value), `energy`
(meter, kWh, kW), `water`.

Stations use the same application in kiosk mode: an operator taps a badge
(the same card the access-control system issues, so one ID) and scans
whatever the step asks for. No forms longer than one screen.

## Observability and alerting

Two tiers, one Grafana:

- **Process alarms** from TimescaleDB: zone environment excursions and
  rate of change, boiler pressure and level, chiller return temperature,
  compressed-air pressure, hydraulic pressure, reject rate per run above
  baseline, stamper approaching its play-count limit, PV or BESS faults,
  fire and BESS detection contacts, UPS on battery.
- **Infrastructure alarms** from the clan monitoring stack: host down,
  disk, service failed, backup missed, certificate expiry.
- **Data-silence alarms** in both tiers: a collector that has published
  nothing for five minutes is an alarm, whatever the last value was.

Routing: Telegram or the LINE Messaging API for the shift group, email for
the daily digest, SMS through a Thai gateway for the critical set (boiler,
fire, BESS, archive climate) with an on-call rota. The animated
[plant operation page](plant-operation.html) is the design target for the
live overview: once the telemetry tables exist it is fed from Grafana or
directly from the database instead of its simulation.

## Backup, retention and recovery

- **Hourly ZFS snapshots** on `core-1` (sanoid), replicated to `backup-1`
  (syncoid).
- **Nightly Borg** from every `core` and `edge` machine to `backup-1` and
  to `vps-1` (clan borgbackup, two destinations).
- **PostgreSQL point-in-time recovery**: pgBackRest with WAL archiving to
  object storage, replicated to the off-site node; a streaming replica on
  `backup-1` for fast failover.
- **Object storage** (Garage) with three nodes: `core-1`, `backup-1`,
  `vps-1`; audio captures, inspection images, metrology raw data and CoA
  scans survive the loss of any one node.
- **Configuration**: the clan repo in Gitea, mirrored off-site; any machine
  is rebuilt from it.
- **Restore drill quarterly**: restore the database to a scratch VM and
  open a random order's trace in the portal. A backup that has never been
  restored does not count.

| Data | Retention |
|---|---|
| genealogy, orders, gates, QC verdicts, event log | life of the customer relationship, never deleted while the metal parts exist |
| per-cycle press data | forever (small) |
| 1 Hz machine telemetry | raw 2 years, then 1-minute aggregates forever |
| zone environment | raw 5 years |
| energy and water | forever (small) |
| audio captures and inspection images | 2 years, longer for any record with a complaint |
| client audio | per contract; default deleted 12 months after final delivery, or on request |
| access-control events | 1 year |
| CCTV | 30 days |
| host logs and metrics | 90 days |

PDPA applies to badge events, CCTV and customer contact data: a written
retention policy, role-based access, and export/deletion on request. The
customer portal shows operator roles, never operator names.

## Security

- OT VLANs have no internet and no inbound routes; edge collectors are the
  only bridge and they only publish.
- Staff sign-on through Pocket ID (OIDC, passkeys) with roles: admin,
  engineer, operator, sales, customer. Stations authenticate by badge for
  attribution, not for privilege.
- Secrets only in clan vars; nothing in plain text in the repo.
- Client audio: separate encrypted ZFS dataset, SFTP or portal upload with
  checksums, every read logged, a written policy sales can show a label.
- UPS on `core-1`, `router-1`, the switches, the edge collectors and the
  press PLCs (already required by facility security); NUT triggers a clean
  shutdown before the battery is exhausted.
- Signed commits on the clan repo; deployments only from a reviewed branch.

## Procurement clauses

To go into every RFQ that leaves from [roadmap](roadmap.md) items 4 to 8:

1. **Presses (Newbilt, Pheenix)**: OPC UA (or S7/ADS) read access licensed
   and enabled; the full tag list with units and the cycle counter as a
   tag; a defined hand-off of the cycle counter to the inspection trigger;
   remote support through our jump host; no cloud agent as a warranty
   condition; PLC clock accepting NTP.
2. **Boiler, chiller, compressor, heat pump (Thai vendors)**: Modbus RTU or
   TCP with the register map delivered; alarm contacts wired to a terminal
   strip for us.
3. **HVAC contractor**: a BMS with BACnet/IP or Modbus TCP read access and
   a point list; zone sensor wiring and room dP sensors in scope; sensor
   loops terminated in the edge cabinets.
4. **Electrical contractor**: Modbus energy meters on the feeders listed
   above; RS485 to the inverters and BESS BMS without cloud dongles;
   structured cabling and a VLAN-capable switch in every cabinet.
5. **Galvanics (Digital Matrix)**: what the rectifiers and control rings
   expose, and how; Modbus preferred.
6. **Metrology (Keyence Thailand)**: confirmed CSV/3D export paths and
   whether the instrument software can write to a network share unattended.
7. **Vision integrator**: result string format, image push, cycle counter
   correlation, and an acceptance test defined in records not in features.
8. **Access control and fire**: API or export for door events; dry
   contacts for every alarm class; BESS detection outputs.

## Build order

Aligned with the [roadmap](roadmap.md) phases.

**Now (roadmap Phase 0 and 1): design and dev fleet.**

- Put the procurement clauses above into the Newbilt re-quote, the
  Pheenix, Digital Matrix and Thai utility RFQs.
- Create the clan repo and a development clan of three small machines at
  the office: a `core-1` stand-in, one edge collector with a Modbus and
  OPC UA simulator, one station. This proves the tooling before a single
  plant machine exists.
- Build the factory application v0 against a press simulator (the state
  machine in the plant operation page is the seed: same states, same
  cycle timing) so orders, gates, genealogy and press cycles can be
  exercised end to end without hardware.
- Decide the repository layout: this repository stays documents; the clan
  flake and the application get their own repositories in the same Gitea
  organisation, linked from here (see open questions).

**Build and commission (roadmap Phase 3).**

- Network, `router-1`, `core-1`, `backup-1` and `vps-1` installed before
  the machinery arrives, so vendor commissioning engineers plug into a
  finished network.
- Zone sensor loops, meters and edge cabinets installed by the electrical
  and HVAC contractors during fit-out; environment logging is live before
  pressing starts, which gives a baseline.
- Machinery commissioning includes data acceptance: every tag on the list
  readable, the cycle counter increments in the database with each cycle,
  data-silence alarms proven by pulling a cable.
- Application v1 live: intake, gates, bridge-stamper genealogy, press runs
  and cycles, PVC lot gate, packaging scan, shipments; stations at press,
  lab, packaging and shipping.

**Launch (roadmap Phase 4).**

- Optical inspection integrated and correlated; reject reasons flowing.
- Customer portal opened to the first labels.
- Energy and environment dashboards used for the first PV and HVAC
  tuning.

**Expand (roadmap Phase 5).**

- `edge-galvanics`, bath sensors, rectifier data, metrology ingestion at
  `st-metrology`; genealogy switches from bridged to in-house.
- Audio QC station and pipeline; retirement signal live.
- Automated press line joins the same collector with the same tag family.
- Biomass boiler on the same Modbus path as its predecessor.

## Rough cost

Planning estimates, September 2026, excluding the QC instruments (those
are in quality automation) and excluding the vendors' OPC UA license
options, which come back with the quotes.

| Block | Estimate (USD) |
|---|---|
| `core-1` and `backup-1` servers, UPS, rack | 8,000 to 15,000 |
| `router-1`, managed switches (industrial in the cabinets), cabling allowance | 4,000 to 8,000 |
| 3 edge collectors with I/O modules and enclosures | 3,000 to 6,000 |
| 8 station kiosks with readers, scanners, one barcode verifier, one darkfield camera | 8,000 to 15,000 |
| zone environment sensors (12 to 16 points), room dP, reference PM instrument | 4,000 to 8,000 |
| energy meters (10 feeders), water meters, level transmitters | 3,000 to 6,000 |
| audio QC hardware (turntable, arm, cartridge, phono stage, interface) | 2,000 to 5,000 |
| `vps-1` and off-site storage, per year | 500 to 1,500 |
| **Hardware total** | **~32,000 to 64,000** |
| Application development to v1 (orders through shipments, stations, portal) | 4 to 6 person-months |
| Clan fleet, modules and collectors to commissioning | 1 to 2 person-months |
| Vision, metrology and audio pipeline integration (Launch and Expand) | 3 to 5 person-months, plus integrator fees in the vision quote |

Headcount note: [scale](scale.md) lists no IT or automation role in the
15 to 25 people. Running this needs one automation and IT engineer from
Build onward, part-time at launch; add it to the headcount.

## Open questions

- Semi-automatic press PLC: does the HRP 12 expose data, or is the sensor
  fallback needed? (Ask in the Newbilt re-quote.)
- Cycle counter hand-off to the vision system: PLC output, network
  message, or time correlation at the unloader? Decided with the vision
  integrator, tested at acceptance.
- HVAC BMS protocol and who owns the zone sensors: HVAC contractor or us.
- Where `vps-1` lives: outside Thailand (latency to the mesh, data
  residency for customer data) or a Bangkok provider.
- Portal availability: served through the mesh from `core-1` (down when
  the plant's internet is down) or a read replica on `vps-1` (more moving
  parts). Start with the former.
- Accounting and ERP: out of scope for the application; if an ERP is
  adopted later it references the same order IDs. Decide after launch.
- Repository layout: documents here, `cnx-infra` (clan) and `cnx-factory`
  (application) as separate repositories, or everything in one. The
  README currently promises this repository will hold the software; a
  link satisfies that.
- Dead-wax code readability on black and coloured vinyl with a camera:
  test with samples before the packaging station is designed.