diff --git a/README.md b/README.md index 3b67ae7..dff1a5b 100644 --- a/README.md +++ b/README.md @@ -86,6 +86,13 @@ The detail lives in dedicated documents: - [Process logging and traceability](docs/traceability.md): end-to-end per-order traceability for customers, batch genealogy from audio file to shipment, and the requirements this sets for the factory software +- [Digital factory systems plan](docs/digital-factory.md): every system + that has to exist for environment, machine, QC and genealogy data to land + in one database keyed to one record identity: the NixOS machine fleet + managed as one Clan, networks and VLAN zoning, Telegraf/MQTT acquisition + source by source, QC instrument integration, the factory application, + alerting, backup, security, the data clauses for every RFQ, build order + and rough cost - [Quality automation](docs/quality-automation.md): stamper metrology (Keyence-class confocal, profiler, XRF), layered pressed-record QC (100% optical inspection, per-cycle press telemetry, sampled automated audio @@ -141,6 +148,7 @@ docs/ ├── suppliers/ Supplier contacts, vendor landscape, supplier map ├── biomass.md Biomass fuel strategy and the 60 rai coppice idea ├── boi.md BOI promotion strategy and requirements +├── digital-factory.md Data platform, NixOS/Clan fleet, acquisition, app ├── facility-security.md Security, HVAC, fire and life safety, compliance ├── process-flow.md Audio file to delivered record, with all checks ├── production-timeline.md Order turnaround, stage by stage diff --git a/docs/digital-factory.md b/docs/digital-factory.md new file mode 100644 index 0000000..b31c892 --- /dev/null +++ b/docs/digital-factory.md @@ -0,0 +1,599 @@ +# Digital Factory: Systems Plan + +The [traceability spec](traceability.md) says what must be logged and the +[quality automation](quality-automation.md) document says which instruments +produce the numbers. This document is the plan for the systems in between: +which computers, networks, collectors, databases and applications have to +exist so that environmental data, machine data, QC results and the batch +genealogy all land in one place, keyed to one record identity, and can be +shown to a customer. It is a planning position as of September 2026; every +cost figure is an estimate until a quote replaces it. + +**Standing decision: every service runs on NixOS, and the whole fleet is +declared and deployed as one [Clan](https://clan.lol).** Servers, edge +collectors, station kiosks, the router and the off-site VPS are all NixOS +machines in a single inventory. Vendor instrument PCs (Keyence, vision +controllers, PLC HMIs) are appliances, not servers: they export into the +NixOS estate and never host a service of ours. + +## Design decisions + +1. **One clan, one repo, rebuildable from git.** Machine configuration, + service placement, mesh networking, backups, host monitoring and + secrets (clan vars, sops-encrypted) live in one flake. A failed machine + is replaced with `clan machines install ` from a fresh disk; a + configuration change is `clan machines update`. Nothing is configured by + hand on a running box. +2. **One relational database for everything that has identity.** + PostgreSQL with TimescaleDB holds the genealogy, orders, gates, QC + results, per-cycle press data, environment history and energy history. + Joining "this record" to "the mold temperature on that cycle" and to + "the humidity in the press hall that hour" is a query, not an export. + Host and infrastructure metrics (CPU, disk, systemd, journals) go to + the separate Prometheus/Loki pair that the clan monitoring service + provides; they are not process data. +3. **MQTT is the plant bus, Telegraf is the universal collector.** Every + machine, meter and sensor is read by a Telegraf instance on an edge + collector near it (Modbus, OPC UA, Siemens S7 and MQTT inputs are + stock plugins) and published to the broker on the core server. The + core side persists everything to TimescaleDB with the same tool. Custom + code exists only where no plugin does (vision results, instrument CSV + ingestion, the audio QC pipeline). +4. **Read-only toward machines.** The data platform never writes to a PLC + or an instrument. Control, safety and interlocks (fume extraction, steam + shutoff, boiler safety chain) stay in the vendor PLCs and hard wiring, + as [facility security](facility-security.md) already requires. +5. **Join keys are counters, not clocks.** Every press cycle carries the + machine's own cycle counter; every instrument result carries the part + serial it was read from. Wall-clock time is recorded (UTC, NTP from the + router) but is never the thing two systems are joined on. +6. **Append-only, hash-chained.** Process events, gate approvals and QC + verdicts are written once and never updated; corrections are new events + that reference the old. Each event carries the hash of the previous one + so the log's integrity can be checked, which is what lets the customer + portal be trusted. +7. **The customer portal is a view, not a copy.** It reads the same + database through the same application with a customer role; there is + no exported report to keep in sync. + +## The systems on one page + +| System | What it does | Runs on | Phase | +|---|---|---|---| +| Network and firewall | VLAN zoning (OT, sensors, servers, stations, office, CCTV, guest), dual WAN, NTP, DHCP/DNS | `router-1` (NixOS, nftables) plus managed switches | Build | +| Clan fleet | Inventory of every NixOS machine, ZeroTier mesh, SSH and user management, secrets, backups, host monitoring | git repo, applied to all machines | Now (dev), Build (plant) | +| Plant data platform | PostgreSQL + TimescaleDB, MQTT broker, Telegraf persister, Grafana, S3-compatible object storage | `core-1` | Build | +| Factory application | Orders, gates, genealogy, materials, press runs, QC verdicts, packaging, fulfillment, customer portal, audit log | `core-1` | Build (v1), Launch (v2) | +| Edge collectors | Telegraf per machine zone, local store-and-forward, vision result receiver | `edge-press`, `edge-utility`, `edge-galvanics` | Build, Expand | +| Environmental monitoring | Zone temperature, humidity, PM, differential pressure; alarms; correlation with production | sensors on RS485 loops into `edge-utility` and `edge-press` | Build | +| Machine telemetry | Presses, extruders, hydraulics, boiler, chiller, compressor, heat pump, HVAC, PV, BESS, energy and water meters | edge collectors | Build | +| QC instrument integration | Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans | stations plus `edge-press` | Launch, Expand | +| Station kiosks | Badge check-in and scanning at each manual step; kiosk browser on the factory application | `st-*` machines | Build | +| Observability and alerting | Process alarms (Grafana on TimescaleDB), host alarms (clan monitoring), routing to phones, data-silence alarms | `core-1` | Build | +| Backup and DR | ZFS snapshots, Borg to `backup-1` and off-site, PostgreSQL point-in-time recovery, restore drills | `backup-1`, `vps-1` | Build | +| Identity | OIDC single sign-on for staff, badge IDs for operators at stations, role model for customers | `core-1` (Pocket ID) | Build | +| Client audio vault | Encrypted, isolated storage for pre-release audio with checksums and access logging | `core-1` (separate dataset) | Build | +| Site systems integration | Access-control door events, fire panel and BESS detection contacts, UPS state, CCTV | `edge-utility`, optional `nvr-1` | Build | +| Off-site node | ZeroTier moon with a public IP, off-site backup target, public reverse proxy for the portal | `vps-1` | Build | + +## Architecture + +```mermaid +flowchart LR + subgraph L0[Field] + PLC[Press and extruder PLCs
OPC UA / S7] + UT[Boiler, chiller, compressor,
heat pump, HVAC
Modbus / BACnet] + EN[Energy meters, Deye inverters,
BESS, water meters
Modbus / pulse] + ENV[Zone T / RH / PM / dP sensors
Modbus RTU] + GV[Rectifiers, bath sensors
Modbus] + VIS[Vision controller
TCP results, FTP images] + INST[Keyence, XRF, lab instruments
CSV export] + end + subgraph L1[Edge, NixOS] + EP[edge-press
Telegraf + vision receiver] + EU[edge-utility
Telegraf] + EG[edge-galvanics
Telegraf] + ST[st-* stations
kiosk, scanners, ingestion] + end + subgraph L2[Core, NixOS] + MQ[MQTT broker] + DB[(PostgreSQL + TimescaleDB)] + OBJ[(Object storage)] + APP[Factory application] + GF[Grafana + alerting] + end + subgraph L3[Outside] + VPS[vps-1: moon, backup, proxy] + CUST[Customer portal] + PH[Phones: alerts] + end + PLC --> EP + VIS --> EP + UT --> EU + EN --> EU + ENV --> EU + GV --> EG + INST --> ST + EP --> MQ + EU --> MQ + EG --> MQ + ST --> APP + MQ --> DB + MQ --> APP + APP --> DB + APP --> OBJ + EP --> OBJ + DB --> GF + GF --> PH + APP --> VPS --> CUST +``` + +Three rules keep this simple: field devices talk only to their edge +collector (they are on VLANs with no route anywhere else); edge collectors +talk only to the broker and object storage; the application and Grafana +talk only to the database, the broker and object storage. The VPS reaches +the core over the ZeroTier mesh, never over an open port. + +## Machines + +All NixOS, all in the clan inventory. Hardware classes are planning +positions; models are chosen at purchase. + +| Machine | Role | Hardware class | Where | Clan tags | +|---|---|---|---|---| +| `router-1` | Firewall, VLANs, DHCP/DNS, NTP (chrony), dual WAN failover | fanless x86, 4+ NICs | IT room | `core` | +| `core-1` | PostgreSQL + TimescaleDB, Mosquitto, Telegraf persister, factory app, Grafana, Garage object storage, Pocket ID, Caddy | 1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD pool for objects, dual PSU on UPS | IT room | `core`, `db` | +| `backup-1` | Borg server, PostgreSQL streaming replica, second Garage node, ZFS receive target | tower or 2U, large HDD pool | separate room from `core-1` (archive side of the building) | `core`, `backup` | +| `vps-1` | ZeroTier moon (public IP), off-site Borg destination, third Garage node, reverse proxy for the customer portal | small cloud VPS with block storage, outside Thailand or in Bangkok, decision open | cloud | `offsite` | +| `edge-press` | Telegraf (OPC UA / S7 to the presses and extruders, Modbus to hydraulics and label oven), vision result receiver, disk-backed buffer | fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485 | press hall control cabinet | `edge` | +| `edge-utility` | Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS, water; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS | same class, more RS485 | electrical room or utility yard cabinet | `edge` | +| `edge-galvanics` | Telegraf (rectifiers, bath sensors, extraction status) | same class, corrosion-protected enclosure outside the wet area | galvanics service corridor | `edge` (Expand phase) | +| `st-intake` | Office workstation for audio intake and gates (plain browser; not a kiosk) | any NixOS desktop | office | `station` | +| `st-cutting` | Session log entry, lacquer registration, scribe ID | mini PC, touch screen, badge reader | cutting room | `station` | +| `st-press-1`, `st-press-2` | Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons | mini PC, touch screen, badge reader, 2D scanner | at each press | `station` | +| `st-lab` | PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment | mini PC, RS232/USB to instruments | material lab | `station` | +| `st-metrology` | SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan | mini PC | metrology bench (Expand phase) | `station` | +| `st-audioqc` | Needle-drop capture and analysis pipeline | mini PC with audio interface, turntable, phono stage | listening/QC room (Expand phase) | `station` | +| `st-pack-1` | Runout code camera read, jacket barcode scan, BOM check, barcode verifier | mini PC, camera, scanner, verifier | packaging | `station` | +| `st-ship` | Consignment assembly, carrier label and tracking capture | mini PC, label printer | shipping | `station` | +| `nvr-1` | Frigate CCTV recording on its own VLAN (optional; a vendor NVR appliance is the alternative) | small server with GPU or Coral accelerator | IT room | `cctv` | + +Stations run a kiosk browser (`cage` plus Firefox or Chromium) pointed at +the factory application; badge readers and scanners are USB HID keyboard +devices, so there is no driver work. Fifteen machines at launch, about +twenty at maturity; every one is a few lines in the inventory. + +## Networks + +| VLAN | Members | Rule | +|---|---|---| +| `ot-press` | press and extruder PLCs and HMIs, vision controller, `edge-press` | no internet, no route to other VLANs; `edge-press` is the only gateway | +| `ot-utility` | boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, `edge-utility` | same | +| `ot-galvanics` | rectifiers, bath transmitters, `edge-galvanics` | same | +| `sensors` | RS485 loops terminate on the edge collectors, so zone sensors have no IP presence; any IP sensor that is added lives here | no internet | +| `servers` | `core-1`, `backup-1`, `router-1` management | reachable from `stations`, `office`, edge collectors and the mesh | +| `stations` | `st-*` kiosks, instrument PCs (Keyence, vision setup laptop) | reach `core-1` only | +| `office` | staff laptops, printers | internet, `core-1` | +| `cctv` | cameras, `nvr-1` | no internet; NVR reachable from `office` by role | +| `guest` | visitors, vendor laptops | internet only | +| `mgmt` | switch and IPMI management interfaces | `router-1` and `core-1` admins only | + +Vendor remote support (Newbilt and Pheenix both offer remote monitoring) goes +through a jump account on `router-1` over the ZeroTier mesh, time-boxed and +logged, with a port forward to the one PLC HMI they need. No machine ever +gets a vendor's own cellular router or cloud agent on the OT VLAN. + +Dual WAN: fibre plus a 4G/5G modem on `router-1`, failover only. The plant +does not depend on the internet to run: everything in the tables above is +on site except `vps-1`, and the only things that stop when the internet +drops are the customer portal and off-site backup. + +## Clan inventory (sketch) + +The plant clan in outline. Option names follow the clan release current at +writing (26.05); check them against the pinned release before use, and +expect the monitoring service to change, since it is still marked +experimental upstream. + +```nix +{ + meta.name = "cnx-vinyl"; + meta.domain = "cnx"; # mesh-only TLD; hosts resolve as .cnx + + inventory.machines = { + router-1 = { tags = [ "core" ]; }; + core-1 = { tags = [ "core" "db" "monitoring-server" ]; }; + backup-1 = { tags = [ "core" "backup" ]; }; + vps-1 = { tags = [ "offsite" ]; }; + edge-press = { tags = [ "edge" ]; }; + edge-utility = { tags = [ "edge" ]; }; + edge-galvanics = { tags = [ "edge" ]; }; + st-cutting = { tags = [ "station" ]; }; + st-press-1 = { tags = [ "station" ]; }; + st-lab = { tags = [ "station" ]; }; + st-pack-1 = { tags = [ "station" ]; }; + st-ship = { tags = [ "station" ]; }; + }; + + inventory.instances = { + sshd.roles.server.tags.all = { }; + admin.roles.default.tags.all = { }; # admin users and keys + + zerotier = { + roles.controller.machines.core-1 = { }; + roles.moon.machines.vps-1.settings.stableEndpoints = [ "/9993" ]; + roles.peer.tags.all = { }; + }; + + borgbackup = { + roles.server.machines.backup-1 = { }; + roles.server.machines.vps-1 = { }; + roles.client.tags.core = { }; + roles.client.tags.edge = { }; + }; + + monitoring = { + roles.server.machines.core-1.settings.grafana.enable = true; + roles.client.tags.all = { }; + }; + }; + + # Our own NixOS modules, applied by tag or by machine: + # modules/edge-collector.nix Telegraf with the tag lists for that cabinet, + # Mosquitto bridge with disk buffer + # modules/core-data.nix PostgreSQL + TimescaleDB, Mosquitto, Telegraf + # persister, Garage, Grafana, pgBackRest + # modules/factory-app.nix the application service and its migrations + # modules/station-kiosk.nix cage + browser, HID readers, auto-login + # modules/router.nix nftables zoning, VLAN interfaces, chrony, + # dual-WAN failover +} +``` + +What clan provides out of the box and we do not write ourselves: SSH and +admin user management, the ZeroTier mesh (controller on `core-1`, the VPS +as a publicly reachable moon so the plant and remote consultants find each +other behind NAT), Borg backups with generated keys, host metrics and +journal shipping with Grafana on the server role, and sops-encrypted +secrets per machine. What we write: the five modules listed in the sketch +and the factory application itself. + +## Data acquisition, source by source + +Every row is also a purchasing clause: the "requirement" column is what +goes into the RFQ, per the traceability principle that machine data access +is bought, not retrofitted. Rates are planning values. + +### Presses and extruders + +| Source | Signals | Interface | Rate | Requirement to write into the RFQ | +|---|---|---|---|---| +| Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA) | cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes | OPC UA server on the PLC, read by Telegraf `inputs.opcua` | per cycle plus 1 Hz | OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty | +| Newbilt HRP 12 semi-automatic and VEX 50 extruder (PLC type unconfirmed) | as above where the PLC has them; on a relay-logic machine at least press closed/open, steam on/off, cycle count | OPC UA or Siemens S7 (`inputs.s7comm`); fallback: our own 4 to 20 mA sensors on a Modbus TCP I/O module and a cycle count from the mold-closed limit switch | per cycle plus 1 Hz | state what the PLC is and which of the two paths applies; the fallback sensor set is priced separately | +| Pheenix AD12 (Beckhoff PLC, remote monitoring supported) | same tag family as the Classic | OPC UA (Beckhoff TF6100) or ADS; `inputs.opcua` | per cycle plus 1 Hz | OPC UA license included, tag list delivered, remote monitoring routed through our jump host | +| Hydraulic aggregate | system pressure, oil temperature, tank level, filter differential pressure | via the press PLC if exposed, else own transmitters on the I/O module | 1 Hz | ask whether the aggregate's sensors are on the PLC | +| Label dry oven | chamber temperature, timer state | Modbus if the controller has it, else a thermocouple on the I/O module | 1/min | | +| PVC grinder and regrind scale | run state, batch weight | scale with RS232 to `st-press`, run contact to I/O | per batch | scale must have a serial output | + +### Utilities and energy + +| Source | Signals | Interface | Rate | Requirement | +|---|---|---|---|---| +| Steam boiler (Thai-sourced) | pressure, steam temperature, feedwater temperature, drum level, burner or heater state, fuel or kW, blowdown events, water treatment conductivity and hardness alarm | Modbus RTU or TCP on the boiler controller | 1 Hz | Modbus register map delivered with the boiler; Thai boiler-regulation inspection data logged too | +| Steam accumulator | pressure | pressure transmitter to I/O | 1 Hz | | +| Chiller and cooling tower | supply and return temperatures, flow, compressor power, tower fan state, makeup water | Modbus | 1 Hz | register map | +| Feedwater heat pump | source and sink temperatures, power | Modbus | 1/10 s | register map | +| Compressor | line pressure, run hours, dew point | Modbus | 1/10 s | register map | +| HVAC / BMS | AHU states, setpoints, supply temperatures, filter differential pressure, positive-pressure room dP | BACnet/IP or Modbus from the BMS the HVAC contractor supplies | 1/min | the RFQ names BACnet/IP or Modbus TCP read access as a deliverable, with point list | +| Electrical | main incomer, feeders for press hall, chiller, HVAC, boiler, galvanics, packaging, IT; PV inverters (Deye, Modbus RTU/TCP register map is public); BESS state of charge, power, cell temperatures (through the Deye battery registers or the Gobel BMS RS485) | Modbus energy meters on each feeder; inverters direct, no cloud dongle | 1/10 s | meters specified in the electrical design, not added later | +| Water | mains or well meter, process tank level, boiler and cooling tower makeup meters, galvanics rinse recycle, fire tank level | pulse counters and level transmitters on I/O | 1/min | meters in the plumbing design | + +### Environment + +| Zone | Sensors | Alarm tier | +|---|---|---| +| cutting room | T, RH, PM2.5/PM10, room dP | tight, positive pressure | +| galvanics clean room | T, RH, PM, room dP | tight, positive pressure | +| master and stamper archive | T, RH, PM, water leak | tightest | +| press hall (two points at launch, four at maturity) | T, RH, PM | process spec 20 to 26 °C, 35 to 60 % RH | +| material lab and PVC store | T, RH | process | +| packaging and finished goods | T, RH, PM | process | +| listening/QC room | T, RH | comfort | +| IT room | T, RH, water leak | tight | +| office | T, RH, PM | comfort | +| outdoor reference | T, RH, PM2.5 | reference for burning-season episodes | + +Industrial RS485 Modbus RTU transmitters on wired loops to the nearest edge +collector, one reading per minute, with a reference-grade PM instrument in +the archive or cutting room to check the cheaper sensors against. Cheap +ESP32/MQTT sensor nodes are acceptable for extra density in non-critical +zones, never for the alarmed ones. Every zone alarm has two thresholds +(warning, action) and a rate-of-change alarm for the archive. + +### Galvanics (Expand phase) + +Rectifier current, voltage and ampere-hours per cell (Digital Matrix: ask +what the control rings and rectifiers expose; Modbus is the requirement), +bath temperature, pH and conductivity from transmitters, filtration and +agitation pump states, fume extraction running status (the interlock itself +stays hard-wired), rinse water conductivity and evaporator state. Titration +results are entered at a station against the bath ID. + +### Site systems + +Access-control door events (badge ID, door, direction, time) from the +controller's API or export; fire alarm panel zone and fault contacts, BESS +off-gas and early-smoke detection, UPS on-battery and low-battery (NUT on +`core-1` talking to the UPS) as dry contacts on `edge-utility`. CCTV stays +on its own VLAN with its own retention; it is never joined to production +data. + +### What produces no telemetry + +The Agnew lathe and mastering chain, the Sibert stamper-prep machines and +the shrink wrapper produce no machine data. They get station check-ins: +who, when, which part, which parameters, with files attached (cutting +parameters, DAW session export). The room environment around them is +logged by the sensors above. + +## QC instrument integration + +| Instrument | How results reach the database | Key | Open integration item | +|---|---|---|---| +| Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration) | vision controller sends one result string per record over TCP to a receiver on `edge-press`; images pushed by FTP/SFTP to object storage | press ID + cycle counter, passed from the PLC to the vision trigger | the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test | +| Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer) | vendor Windows PCs export CSV and raw 3D to an SMB share on `st-metrology`; a watcher parses and posts | back-side Data Matrix serial, scanned at the bench before measurement | export format per instrument confirmed with Keyence Thailand | +| PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR) | instruments with RS232/USB export are read at `st-lab`; the rest are typed in with the printout photographed; CoA PDF attached | compound lot ID (printed label scanned at intake) | which benchtop models have serial export | +| Audio QC station | capture service on `st-audioqc` records the needle-drop (PipeWire, FLAC to object storage); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted | record key (stamper + press + cycle + side), typed or picked from the run list | first-of-kind pipeline, developed in Expand phase against launch-phase recordings | +| Packaging assembly check | camera with darkfield lighting reads the scribed dead-wax code (a handheld scanner will not read a scribed code on black vinyl reliably); handheld scanner reads the jacket barcode; barcode verifier grades it | order + record key | confirm dead-wax code readability with a sample before the packaging station is designed | +| Label print inspection | same vision family as the record label check; result per label lot | label lot ID | shared with the record inspection budget | + +## The factory application + +Working name `cnx-factory`. Server-rendered web application in Rust (axum, +SQLx, minijinja templates, HTMX) on PostgreSQL, the stack B4L already uses +for its other in-house tooling, with the same append-only ledger and +hash-chained audit log pattern. It subscribes to the MQTT event topics +(cycle complete, machine alarm, instrument result) and owns every write to +the identity tables; Telegraf owns the writes to the time-series tables. + +Modules, in build order: + +1. **Orders and releases**: customer, order, release, sides, formats, run + quantity, bill of materials (record, inner sleeve, jacket, inserts, + shrink), due dates. +2. **Audio intake and the client vault**: upload via portal or SFTP with + checksums, versions, handler, technical QA form; files in an encrypted, + access-logged dataset separate from everything else. +3. **The three artist gates**: master plan, master, test pressing; each is + an event with who approved, when, from which portal session or on whose + verbal instruction, and nothing downstream is schedulable without it. +4. **Mastering and cutting sessions**: engineer, chain settings, lacquer + IDs, dead-wax scribe text and fiducial applied, microscope check result. +5. **Metal genealogy**: lacquer to father to mother to stamper, or bridged + supplier plus batch; per-part metrology history; stamper lifecycle + (mounted on which press, play count, retired why), archive location. +6. **Materials**: PVC lots with the lab gate and CoA, regrind batches, + label lots with print inspection and oven pre-dry, carton and sleeve + lots with incoming QC. +7. **Press runs and cycles**: a run binds order, stamper pair, press, + operator, shift, PVC lot in the hopper, regrind batch, label lot; each + cycle event from the PLC becomes a row keyed by press + counter; the + record key is derived from it per side. +8. **QC verdicts**: optical per record, audio per sampled record, lab per + lot, packaging per unit; reject reasons; test-pressing hold state. +9. **Finishing, packaging, fulfillment**: assembly checks, consignment + against the BOM, carrier and tracking per shipment or dropship end + customer. +10. **Customer portal**: per-order timeline of everything above, scoped so + one customer sees only its own orders; PDF export of the trace. +11. **Reports and signals**: yield per stamper, reject rate drift, audio + score trend versus play count (the automatic retirement signal), + energy per record, environment excursions overlaid on runs. + +Core entities: `customer`, `order`, `release`, `side`, `audio_asset`, +`gate_approval`, `session`, `lacquer`, `father`, `mother`, `stamper`, +`stamper_measurement`, `press`, `press_run`, `press_cycle`, `record`, +`optical_result`, `audio_result`, `pvc_lot`, `regrind_batch`, `label_lot`, +`carton_lot`, `lab_result`, `packaging_check`, `shipment`, `operator`, +`shift`, `event` (the hash chain). Time-series hypertables: `telemetry` +(machine, tag, value), `environment` (zone, metric, value), `energy` +(meter, kWh, kW), `water`. + +Stations use the same application in kiosk mode: an operator taps a badge +(the same card the access-control system issues, so one ID) and scans +whatever the step asks for. No forms longer than one screen. + +## Observability and alerting + +Two tiers, one Grafana: + +- **Process alarms** from TimescaleDB: zone environment excursions and + rate of change, boiler pressure and level, chiller return temperature, + compressed-air pressure, hydraulic pressure, reject rate per run above + baseline, stamper approaching its play-count limit, PV or BESS faults, + fire and BESS detection contacts, UPS on battery. +- **Infrastructure alarms** from the clan monitoring stack: host down, + disk, service failed, backup missed, certificate expiry. +- **Data-silence alarms** in both tiers: a collector that has published + nothing for five minutes is an alarm, whatever the last value was. + +Routing: Telegram or the LINE Messaging API for the shift group, email for +the daily digest, SMS through a Thai gateway for the critical set (boiler, +fire, BESS, archive climate) with an on-call rota. The animated +[plant operation page](plant-operation.html) is the design target for the +live overview: once the telemetry tables exist it is fed from Grafana or +directly from the database instead of its simulation. + +## Backup, retention and recovery + +- **Hourly ZFS snapshots** on `core-1` (sanoid), replicated to `backup-1` + (syncoid). +- **Nightly Borg** from every `core` and `edge` machine to `backup-1` and + to `vps-1` (clan borgbackup, two destinations). +- **PostgreSQL point-in-time recovery**: pgBackRest with WAL archiving to + object storage, replicated to the off-site node; a streaming replica on + `backup-1` for fast failover. +- **Object storage** (Garage) with three nodes: `core-1`, `backup-1`, + `vps-1`; audio captures, inspection images, metrology raw data and CoA + scans survive the loss of any one node. +- **Configuration**: the clan repo in Gitea, mirrored off-site; any machine + is rebuilt from it. +- **Restore drill quarterly**: restore the database to a scratch VM and + open a random order's trace in the portal. A backup that has never been + restored does not count. + +| Data | Retention | +|---|---| +| genealogy, orders, gates, QC verdicts, event log | life of the customer relationship, never deleted while the metal parts exist | +| per-cycle press data | forever (small) | +| 1 Hz machine telemetry | raw 2 years, then 1-minute aggregates forever | +| zone environment | raw 5 years | +| energy and water | forever (small) | +| audio captures and inspection images | 2 years, longer for any record with a complaint | +| client audio | per contract; default deleted 12 months after final delivery, or on request | +| access-control events | 1 year | +| CCTV | 30 days | +| host logs and metrics | 90 days | + +PDPA applies to badge events, CCTV and customer contact data: a written +retention policy, role-based access, and export/deletion on request. The +customer portal shows operator roles, never operator names. + +## Security + +- OT VLANs have no internet and no inbound routes; edge collectors are the + only bridge and they only publish. +- Staff sign-on through Pocket ID (OIDC, passkeys) with roles: admin, + engineer, operator, sales, customer. Stations authenticate by badge for + attribution, not for privilege. +- Secrets only in clan vars; nothing in plain text in the repo. +- Client audio: separate encrypted ZFS dataset, SFTP or portal upload with + checksums, every read logged, a written policy sales can show a label. +- UPS on `core-1`, `router-1`, the switches, the edge collectors and the + press PLCs (already required by facility security); NUT triggers a clean + shutdown before the battery is exhausted. +- Signed commits on the clan repo; deployments only from a reviewed branch. + +## Procurement clauses + +To go into every RFQ that leaves from [roadmap](roadmap.md) items 4 to 8: + +1. **Presses (Newbilt, Pheenix)**: OPC UA (or S7/ADS) read access licensed + and enabled; the full tag list with units and the cycle counter as a + tag; a defined hand-off of the cycle counter to the inspection trigger; + remote support through our jump host; no cloud agent as a warranty + condition; PLC clock accepting NTP. +2. **Boiler, chiller, compressor, heat pump (Thai vendors)**: Modbus RTU or + TCP with the register map delivered; alarm contacts wired to a terminal + strip for us. +3. **HVAC contractor**: a BMS with BACnet/IP or Modbus TCP read access and + a point list; zone sensor wiring and room dP sensors in scope; sensor + loops terminated in the edge cabinets. +4. **Electrical contractor**: Modbus energy meters on the feeders listed + above; RS485 to the inverters and BESS BMS without cloud dongles; + structured cabling and a VLAN-capable switch in every cabinet. +5. **Galvanics (Digital Matrix)**: what the rectifiers and control rings + expose, and how; Modbus preferred. +6. **Metrology (Keyence Thailand)**: confirmed CSV/3D export paths and + whether the instrument software can write to a network share unattended. +7. **Vision integrator**: result string format, image push, cycle counter + correlation, and an acceptance test defined in records not in features. +8. **Access control and fire**: API or export for door events; dry + contacts for every alarm class; BESS detection outputs. + +## Build order + +Aligned with the [roadmap](roadmap.md) phases. + +**Now (roadmap Phase 0 and 1): design and dev fleet.** + +- Put the procurement clauses above into the Newbilt re-quote, the + Pheenix, Digital Matrix and Thai utility RFQs. +- Create the clan repo and a development clan of three small machines at + the office: a `core-1` stand-in, one edge collector with a Modbus and + OPC UA simulator, one station. This proves the tooling before a single + plant machine exists. +- Build the factory application v0 against a press simulator (the state + machine in the plant operation page is the seed: same states, same + cycle timing) so orders, gates, genealogy and press cycles can be + exercised end to end without hardware. +- Decide the repository layout: this repository stays documents; the clan + flake and the application get their own repositories in the same Gitea + organisation, linked from here (see open questions). + +**Build and commission (roadmap Phase 3).** + +- Network, `router-1`, `core-1`, `backup-1` and `vps-1` installed before + the machinery arrives, so vendor commissioning engineers plug into a + finished network. +- Zone sensor loops, meters and edge cabinets installed by the electrical + and HVAC contractors during fit-out; environment logging is live before + pressing starts, which gives a baseline. +- Machinery commissioning includes data acceptance: every tag on the list + readable, the cycle counter increments in the database with each cycle, + data-silence alarms proven by pulling a cable. +- Application v1 live: intake, gates, bridge-stamper genealogy, press runs + and cycles, PVC lot gate, packaging scan, shipments; stations at press, + lab, packaging and shipping. + +**Launch (roadmap Phase 4).** + +- Optical inspection integrated and correlated; reject reasons flowing. +- Customer portal opened to the first labels. +- Energy and environment dashboards used for the first PV and HVAC + tuning. + +**Expand (roadmap Phase 5).** + +- `edge-galvanics`, bath sensors, rectifier data, metrology ingestion at + `st-metrology`; genealogy switches from bridged to in-house. +- Audio QC station and pipeline; retirement signal live. +- Automated press line joins the same collector with the same tag family. +- Biomass boiler on the same Modbus path as its predecessor. + +## Rough cost + +Planning estimates, September 2026, excluding the QC instruments (those +are in quality automation) and excluding the vendors' OPC UA license +options, which come back with the quotes. + +| Block | Estimate (USD) | +|---|---| +| `core-1` and `backup-1` servers, UPS, rack | 8,000 to 15,000 | +| `router-1`, managed switches (industrial in the cabinets), cabling allowance | 4,000 to 8,000 | +| 3 edge collectors with I/O modules and enclosures | 3,000 to 6,000 | +| 8 station kiosks with readers, scanners, one barcode verifier, one darkfield camera | 8,000 to 15,000 | +| zone environment sensors (12 to 16 points), room dP, reference PM instrument | 4,000 to 8,000 | +| energy meters (10 feeders), water meters, level transmitters | 3,000 to 6,000 | +| audio QC hardware (turntable, arm, cartridge, phono stage, interface) | 2,000 to 5,000 | +| `vps-1` and off-site storage, per year | 500 to 1,500 | +| **Hardware total** | **~32,000 to 64,000** | +| Application development to v1 (orders through shipments, stations, portal) | 4 to 6 person-months | +| Clan fleet, modules and collectors to commissioning | 1 to 2 person-months | +| Vision, metrology and audio pipeline integration (Launch and Expand) | 3 to 5 person-months, plus integrator fees in the vision quote | + +Headcount note: [scale](scale.md) lists no IT or automation role in the +15 to 25 people. Running this needs one automation and IT engineer from +Build onward, part-time at launch; add it to the headcount. + +## Open questions + +- Semi-automatic press PLC: does the HRP 12 expose data, or is the sensor + fallback needed? (Ask in the Newbilt re-quote.) +- Cycle counter hand-off to the vision system: PLC output, network + message, or time correlation at the unloader? Decided with the vision + integrator, tested at acceptance. +- HVAC BMS protocol and who owns the zone sensors: HVAC contractor or us. +- Where `vps-1` lives: outside Thailand (latency to the mesh, data + residency for customer data) or a Bangkok provider. +- Portal availability: served through the mesh from `core-1` (down when + the plant's internet is down) or a read replica on `vps-1` (more moving + parts). Start with the former. +- Accounting and ERP: out of scope for the application; if an ERP is + adopted later it references the same order IDs. Decide after launch. +- Repository layout: documents here, `cnx-infra` (clan) and `cnx-factory` + (application) as separate repositories, or everything in one. The + README currently promises this repository will hold the software; a + link satisfies that. +- Dead-wax code readability on black and coloured vinyl with a camera: + test with samples before the packaging station is designed. diff --git a/docs/roadmap.md b/docs/roadmap.md index 319ea1a..99128ca 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -75,7 +75,9 @@ These three run in parallel and gate everything downstream: commissioning is mandatory for warranty; budget per the service rate card) 17. Traceability software v1: intake, batch genealogy, press telemetry - capture (spec in [traceability.md](traceability.md)) + capture (spec in [traceability.md](traceability.md), systems and build + order in [digital-factory.md](digital-factory.md); the network, servers + and edge collectors go in before the machinery arrives) 18. Hire and train the launch team; safety program stands up with the equipment, not after it diff --git a/docs/traceability.md b/docs/traceability.md index 898c040..ef0f233 100644 --- a/docs/traceability.md +++ b/docs/traceability.md @@ -58,4 +58,6 @@ For any order, the customer can follow the full chain: one This system should be specified alongside equipment procurement: machine -data access is a purchasing requirement, not an afterthought. +data access is a purchasing requirement, not an afterthought. The systems +that implement this specification (machines, networks, collectors, +database, application) are planned in [digital-factory.md](digital-factory.md).