grabowski
a101c9f7db
Deploy test site / deploy (push) Successful in 45s
Follows the decision that the whole plant runs ERPNext and every tool
we build interacts with it (e12ad2e).
- New page #erp (nav: ERPNext): every posting the floor makes, derived
from the same event log: sales orders in by webhook, work orders,
job cards per station, serial numbers for lacquers, fathers and
mothers (warehouse Archive), purchase receipts, quality inspections
and stock entries per compound batch, manufacture entries per run,
downtime entries per stop, delivery notes and invoices. Outbox status
per row (sent within a minute, held during the nightly backup
01:00 to 01:15), KPIs, who-owns-what table, counts by document type.
Rows use fixed ch grid columns; on phones a two-line row.
- Order page: ERPNext card (customer, sales order, work order with
produced quantity, job cards completed, batches, delivery note and
invoice). Work order page: its job card, quality inspection or asset
maintenance log.
- Names follow ERPNext's default naming series and are simulated;
nothing uses the simulation's random stream, so the seed is unchanged.
- Phones: the nav strip scrolls the current tab into view. Work order
IDs on the order page no longer wrap (wider fixed column).
digital-factory.md demo paragraph. Checked: node --check, Edge dark and
light at 1440/1024/390 (no overflow, no page errors, rows aligned), and
the pw_fc, pw_wo and pw_feed regressions.
955 lines
63 KiB
Markdown
955 lines
63 KiB
Markdown
# Digital Factory: Systems Plan
|
|
|
|
The [traceability spec](traceability.md) says what must be logged and the
|
|
[quality automation](quality-automation.md) document says which instruments
|
|
produce the numbers. This document is the plan for the systems in between:
|
|
which computers, networks, collectors, databases and applications have to
|
|
exist so that environmental data, machine data, QC results and the batch
|
|
genealogy all land in one place, keyed to one record identity, and can be
|
|
shown to a customer. It is a planning position as of September 2026; every
|
|
cost figure is an estimate until a quote replaces it.
|
|
|
|
**Standing decision: every service runs on NixOS, and the plant is one more
|
|
site in the existing B4L fleet, the
|
|
[cnx-network clan](https://git.b4l.co.th/B4L/cnx-network-clan).** That
|
|
clan already runs the ZeroTier mesh, authoritative DNS, mail, fleet
|
|
monitoring, backups, a public reverse proxy and the first site gateway, all
|
|
declared in one repository and deployed with the clan CLI. The plant adds
|
|
a gateway, a core server, a backup server, edge collectors and station
|
|
kiosks to that inventory and reuses everything the fleet already provides.
|
|
Nothing is built twice. Vendor instrument PCs (Keyence, vision controllers,
|
|
PLC HMIs) are appliances, not servers: they export into the NixOS estate
|
|
and never host a service of ours.
|
|
|
|
**Standing decision (October 2026): the whole plant runs ERPNext**,
|
|
self-hosted on a NixOS machine in the same clan. It is the master for
|
|
customers, prices, orders, money, items, stock and assets, and every tool
|
|
this plan builds (the factory application with its stations and customer
|
|
portal, the artist platform, the instrument integrations) reads from and
|
|
writes to it through one integration layer; none keeps a second master of
|
|
what ERPNext owns ([ERPNext](#erpnext-the-plants-erp)).
|
|
|
|
## What the fleet already provides
|
|
|
|
Read the clan's own runbook (`docs/` in the repo, served on `control`) for
|
|
detail; this is what the plant plan leans on.
|
|
|
|
| Fleet piece | Where | What the plant gets from it |
|
|
|---|---|---|
|
|
| Clan repo, clan-core 26.05, sops vars gated by YubiKey, `admins` group applied to every machine automatically | git.b4l.co.th/B4L/cnx-network-clan | one inventory, one deploy path (`clan machines install` / `update`), secrets minted with `clan vars generate` |
|
|
| ZeroTier mesh, controller on `control`, IPv6 ULA `/88`, `modules/mesh-hosts.nix` reads every peer's address from vars | `control` | admin SSH, deploys and scrapes for plant machines behind PPPoE NAT, exactly as `gw-cnx-1` is reached today; no port forwards, no VPS needed |
|
|
| `modules/router/`: `cnx.router.*` on Topton 1U boxes: PPPoE WAN, VLAN-filtering bridge, nftables with no inter-VLAN traffic, Kea, Blocky, CrowdSec, Omada, iperf3, hourly speed test, Caddy internal proxy with a real wildcard `*.<site><n>.cnx.network` via DNS-01 | site gateways | the plant router is a copy of `machines/gw-cnx-1/` with its own VLANs; the internal proxy gives every station a browser-trusted `https://factory.<site>1.cnx.network` |
|
|
| Addressing convention `10.<siteId>.<vlanId>.0/24`, VLAN ids 10 mgmt and 20 lan mandatory, 30 guest, 40 iot reserved | fleet-wide | the plant takes the next free `siteId` and adds its own VLANs above 40 |
|
|
| VictoriaMetrics (180 d), VictoriaLogs (30 d), vmalert, Grafana, blackbox probes | `control` | node_exporter on every plant machine scraped over the mesh, journald shipped to VictoriaLogs, host alerts from the existing rules; process data stays on site (below) |
|
|
| Knot DNS (`cnx.network`, `buildfor.life`, `cnx.email`) with per-consumer TSIG keys for ACME | `ns1`, `ns2` | the plant's public names and the gateway's wildcard cert; a brand domain for the customer portal is one zone file away |
|
|
| `web01`: public Caddy with `*.cnx.network`, backends dialed over the mesh | Hetzner | the customer portal is one `virtualHosts` entry pointing at the plant core's mesh address, the same way `grafana.cnx.network` works now |
|
|
| clan `borgbackup` service, `control` as server, repokey encryption, backups over the mesh | `control` | the pattern; the plant needs its own server because plant data is far larger than the DNSSEC keystore |
|
|
| chrony everywhere, fixed-offset time zones (`Etc/GMT-7` for Thailand) | fleet baseline | plant machines inherit; edge collectors additionally serve NTP to the PLCs |
|
|
| mdbook runbook in `docs/src/`, built and served on `control` | `control` | the plant's operational runbook goes there as a chapter; this document stays the planning source |
|
|
|
|
Known gaps in the fleet that the plant needs closed (all on the clan's own
|
|
TODO list or small extensions to existing modules):
|
|
|
|
- **Alert delivery**: vmalert currently uses `notifier.blackhole`. A boiler
|
|
alarm needs a phone. The plant brings an Alertmanager with Telegram or
|
|
LINE and SMS receivers; `control`'s vmalert can point at it too.
|
|
- **Off-site backup destination**: `control`'s disk is not sized for plant
|
|
images and audio. Add a Hetzner Storage Box or a storage VPS as a second
|
|
`borgbackup` destination (the clan service supports several per client).
|
|
- **Router module**: three small extensions, listed under Networks.
|
|
|
|
## Design decisions
|
|
|
|
1. **One clan, one repo, rebuildable from git.** Plant machines are entries
|
|
in `clan.nix`'s `fleet` set with a `machines/<name>/` directory each
|
|
(`configuration.nix`, `disko.nix`, `facter.json`), like every machine
|
|
already there. A failed box is replaced from a fresh disk with
|
|
`clan machines install`; a change is `clan machines update`. Nothing is
|
|
configured by hand on a running machine.
|
|
2. **Two stores, by shape of the data.** Everything with an identity
|
|
(orders, gates, genealogy, lots, press runs, per-cycle rows, QC
|
|
verdicts, events) lives in PostgreSQL on the plant core. Everything
|
|
that is a time series (1 Hz machine tags, zone environment, energy,
|
|
water) lives in a plant-local VictoriaMetrics, the same engine and
|
|
modules the fleet already runs on `control`, so dashboards, PromQL and
|
|
vmalert rules carry over unchanged. Per-cycle rows carry a snapshot of
|
|
the zone environment and the key press values, so the relational join
|
|
"this record, that cycle, those conditions" never needs the time-series
|
|
store; the curve behind the snapshot is one PromQL range query away.
|
|
(TimescaleDB in PostgreSQL is the fallback if that split proves
|
|
awkward; it is not the starting point.) Business documents (sales
|
|
orders, stock, invoices) are ERPNext's, decision 10.
|
|
3. **MQTT is the plant bus, Telegraf is the universal collector.** Every
|
|
machine, meter and sensor is read by a Telegraf instance on the edge
|
|
collector nearest it (Modbus, OPC UA, Siemens S7 and MQTT inputs are
|
|
stock plugins) and published to Mosquitto on the core. On the core,
|
|
Telegraf writes the stream to VictoriaMetrics (InfluxDB line protocol,
|
|
native to VM) and the factory application subscribes to the event
|
|
topics (cycle complete, alarm, instrument result) and writes rows.
|
|
Custom code exists only where no plugin does (vision results,
|
|
instrument CSV ingestion, the audio QC pipeline).
|
|
4. **Read-only toward machines.** The data platform never writes to a PLC
|
|
or an instrument. Control, safety and interlocks (fume extraction, steam
|
|
shutoff, boiler safety chain) stay in the vendor PLCs and hard wiring,
|
|
as [facility security](facility-security.md) already requires.
|
|
5. **Join keys are counters, not clocks.** Every press cycle carries the
|
|
machine's own cycle counter; every instrument result carries the part
|
|
serial it was read from. Wall-clock time is recorded (UTC, chrony from
|
|
the fleet baseline) but is never what two systems are joined on.
|
|
6. **Append-only, hash-chained.** Process events, gate approvals and QC
|
|
verdicts are written once and never updated; corrections are new events
|
|
that reference the old. Each event carries the hash of the previous one
|
|
so the log's integrity can be checked, which is what lets the customer
|
|
portal be trusted.
|
|
7. **The customer portal is a view, not a copy.** It reads the same
|
|
database through the same application with a customer role; there is
|
|
no exported report to keep in sync.
|
|
8. **The plant runs without the internet.** Everything that pressing,
|
|
QC and traceability need is on site. A WAN outage stops only the
|
|
customer portal, off-site backup, fleet monitoring and remote support,
|
|
and all four catch up when it returns.
|
|
9. **Operators see the data first.** Every live value the platform
|
|
collects is shown at the station where it can change the next cycle
|
|
(the press-side kiosk with cycle values against the recipe band, the
|
|
inspection reject reasons, the stamper's play count) before it is
|
|
rolled up into reports. Sibert's advice after Haarlem was exactly this:
|
|
live feedback loops that the staff use to improve production, rather
|
|
than only for management reporting, decide whether a plant succeeds
|
|
([exhibitor briefing](suppliers/making-vinyl-europe-2026-exhibitors.md#after-the-show)).
|
|
10. **ERPNext owns the business, the factory application owns the floor.**
|
|
ERPNext holds every business document (customer, sales order, work
|
|
order, job card, batch, quality inspection, stock entry, delivery
|
|
note, invoice, asset); the factory application holds what happens at
|
|
machine speed (cycles, per-record inspection, metrology, genealogy,
|
|
gates, the hash-chained log) and posts results to ERPNext in
|
|
business-sized pieces. ERPNext document names are the keys both sides
|
|
share ([ERPNext](#erpnext-the-plants-erp)).
|
|
|
|
## The systems on one page
|
|
|
|
| System | What it does | Runs on | Phase |
|
|
|---|---|---|---|
|
|
| Site gateway | PPPoE WAN, VLAN zoning, DHCP/DNS, NTP for the site, internal TLS proxy, CrowdSec | `gw-<site>-1` (existing `cnx.router` module) | Build |
|
|
| Plant core | PostgreSQL, Mosquitto, Telegraf persister, VictoriaMetrics + vmalert + Grafana, Alertmanager, factory application, file datasets on ZFS | `vinyl-core` | Build |
|
|
| Factory application | Orders, gates, genealogy, materials, press runs, QC verdicts, packaging, fulfillment, customer portal, audit log; the ERPNext integration layer | `vinyl-core` | Build (v1), Launch (v2) |
|
|
| ERPNext | Accounting with Thai VAT and withholding tax, customers, prices, sales and purchase orders, items, BOMs, work orders and job cards, stock and batches, incoming quality inspections, assets and maintenance, payroll | `vinyl-erp` | from the first capex invoice; manufacturing at commissioning |
|
|
| Edge collectors | Telegraf per machine zone, disk-backed store-and-forward, NTP to the OT segment, vision result receiver | `vinyl-edge-press`, `vinyl-edge-util`, `vinyl-edge-galv` | Build, Expand |
|
|
| Environmental monitoring | Zone temperature, humidity, PM, differential pressure; alarms; correlation with production | RS485 loops into the edge collectors | Build |
|
|
| Machine telemetry | Presses, extruders, hydraulics, boiler, chiller, compressor, heat pump, HVAC, PV, BESS, energy and water meters | edge collectors | Build |
|
|
| QC instrument integration | Optical inspection results and images, metrology CSV ingestion, PVC lab entries, audio QC scores, packaging scans | stations plus `vinyl-edge-press` | Launch, Expand |
|
|
| Station kiosks | Badge check-in and scanning at each manual step; kiosk browser on the factory application | `vinyl-st-*` | Build |
|
|
| Process alerting | Environment, utilities, reject rate, stamper life, PV/BESS, fire and BESS detection contacts, UPS, data silence; delivered to phones | `vinyl-core` (vmalert + Alertmanager) | Build |
|
|
| Fleet monitoring | Host metrics, journals, host alerts for every plant machine | `control` (existing) | Build |
|
|
| Backup and DR | ZFS snapshots and replication, Borg to the plant backup server and off site, PostgreSQL point-in-time recovery, restore drills | `vinyl-backup`, off-site destination | Build |
|
|
| Identity | Staff sign-on (OIDC), badge IDs for operators at stations, customer role | `vinyl-core` | Build |
|
|
| Client audio vault | Encrypted, isolated dataset for pre-release audio with checksums and access logging | `vinyl-core` | Build |
|
|
| Site systems integration | Access-control door events, fire panel and BESS detection contacts, UPS state | `vinyl-edge-util` | Build |
|
|
| Customer portal, public | TLS termination and forwarding over the mesh to the plant core | `web01` (existing) | Launch |
|
|
| CCTV | Recording on its own VLAN | vendor NVR appliance, or `vinyl-nvr` (Frigate) if wanted | Build |
|
|
|
|
## Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph L0[Field, isolated OT segments]
|
|
PLC[Press and extruder PLCs<br/>OPC UA / S7]
|
|
UT[Boiler, chiller, compressor,<br/>heat pump, HVAC<br/>Modbus / BACnet]
|
|
EN[Energy meters, Deye inverters,<br/>BESS, water meters<br/>Modbus / pulse]
|
|
ENV[Zone T / RH / PM / dP sensors<br/>Modbus RTU]
|
|
GV[Rectifiers, bath sensors<br/>Modbus]
|
|
VIS[Vision controller<br/>TCP results, SFTP images]
|
|
INST[Keyence, XRF, lab instruments<br/>CSV export]
|
|
end
|
|
subgraph L1[Edge, NixOS, dual-homed]
|
|
EP[vinyl-edge-press<br/>Telegraf + vision receiver]
|
|
EU[vinyl-edge-util<br/>Telegraf + dry-contact I/O]
|
|
EG[vinyl-edge-galv<br/>Telegraf]
|
|
ST[vinyl-st-* stations<br/>kiosk, scanners, ingestion]
|
|
end
|
|
subgraph L2[Plant core, NixOS, servers VLAN]
|
|
MQ[Mosquitto]
|
|
VM[(VictoriaMetrics<br/>vmalert, Grafana)]
|
|
DB[(PostgreSQL)]
|
|
APP[Factory application]
|
|
AM[Alertmanager]
|
|
ERP[(ERPNext<br/>vinyl-erp)]
|
|
end
|
|
subgraph GW[gw-site-1]
|
|
PX[Caddy internal proxy<br/>factory.site1.cnx.network]
|
|
end
|
|
subgraph L3[Fleet, Hetzner, over the mesh]
|
|
CTL[control: node metrics,<br/>journals, host alerts]
|
|
WEB[web01: public proxy]
|
|
BK[off-site borg destination]
|
|
CUST[Customer portal]
|
|
PH[Phones: alerts]
|
|
end
|
|
PLC --> EP
|
|
VIS --> EP
|
|
UT --> EU
|
|
EN --> EU
|
|
ENV --> EU
|
|
GV --> EG
|
|
INST --> ST
|
|
EP --> MQ
|
|
EU --> MQ
|
|
EG --> MQ
|
|
MQ --> VM
|
|
MQ --> APP
|
|
APP --> DB
|
|
APP -->|REST, outbox| ERP
|
|
ERP -.->|webhooks| APP
|
|
ST --> PX --> APP
|
|
VM --> AM --> PH
|
|
APP -.mesh.-> WEB --> CUST
|
|
L2 -.mesh.-> CTL
|
|
L2 -.mesh.-> BK
|
|
```
|
|
|
|
Three rules keep this simple: field devices sit on isolated OT segments
|
|
whose only other member is their edge collector; edge collectors publish to
|
|
the core and nothing else; stations and office clients reach the
|
|
application only through the gateway's proxy. Everything that leaves the
|
|
site rides the ZeroTier mesh, never an open port.
|
|
|
|
## Machines
|
|
|
|
All NixOS, all entries in the clan inventory. Names follow the fleet
|
|
convention (`gw-<city>-<n>` for gateways, short role names for servers)
|
|
with a `vinyl-` prefix so the plant's machines are recognisable in a
|
|
multi-site inventory. `<site>` is the city code chosen at site selection;
|
|
Mueang Chi is in Lamphun province, so `lpn` is the working assumption and
|
|
the gateway is `gw-lpn-1`, giving internal names under `*.lpn1.cnx.network`.
|
|
Hardware classes are planning positions; models are chosen at purchase.
|
|
|
|
| Machine | Role | Hardware class | Where |
|
|
|---|---|---|---|
|
|
| `gw-lpn-1` | Site gateway: `cnx.router` with the plant VLANs, internal proxy for the factory application and Grafana, NTP for the site | Topton 1U N300, 4x 2.5G (fleet standard); a 10G SFP+ variant if the core server link warrants it | IT room |
|
|
| `vinyl-core` | PostgreSQL, Mosquitto, Telegraf persister, VictoriaMetrics + vmalert + Grafana, Alertmanager, factory application, ZFS datasets for files (audio captures, inspection images, metrology raw data, CoA scans, client audio) | 1U/2U server, ECC RAM 64 to 128 GB, mirrored NVMe for OS and database, HDD mirror or raidz for files, dual PSU on UPS | IT room |
|
|
| `vinyl-erp` | ERPNext (Frappe web, workers, scheduler, socket.io), MariaDB, Redis, the Thai localisation app and our `cnx_erp` app | 1U server, 32 GB RAM, mirrored NVMe, on UPS; or a VM on `vinyl-core` if its hardware has room | IT room |
|
|
| `vinyl-backup` | Borg server for the plant clients, ZFS receive target for `vinyl-core`'s datasets, PostgreSQL streaming replica, pgBackRest repository | tower or 2U, large HDD pool | separate room from `vinyl-core` (archive side of the building) |
|
|
| `vinyl-edge-press` | Telegraf (OPC UA / S7 to presses and extruders, Modbus to hydraulics and label oven), vision result receiver, SFTP target for inspection images, chrony serving the OT segment, disk-backed MQTT buffer | fanless DIN-rail IPC, 2 LAN, 2 to 4 RS485 | press hall control cabinet |
|
|
| `vinyl-edge-util` | Telegraf (Modbus to boiler, chiller, compressor, heat pump, energy meters, inverters, BESS; BACnet or Modbus to the HVAC BMS), zone environment loops, dry-contact I/O for fire panel, BESS detection and UPS | same class, more RS485 | electrical room or utility yard cabinet |
|
|
| `vinyl-edge-galv` | Telegraf (rectifiers, bath sensors, extraction status) | same class, corrosion-protected enclosure outside the wet area | galvanics service corridor (Expand phase) |
|
|
| `vinyl-st-cutting` | Session log entry, lacquer registration, scribe ID | mini PC, touch screen, badge reader | cutting room |
|
|
| `vinyl-st-press1`, `vinyl-st-press2` | Operator check-in, stamper mount/unmount scan, PVC lot and label lot scan, test pressing hold, reject reasons | mini PC, touch screen, badge reader, 2D scanner | at each press |
|
|
| `vinyl-st-lab` | PVC lot gate entry; reads instruments with serial export, manual entry otherwise; CoA attachment | mini PC, RS232/USB to instruments | material lab |
|
|
| `vinyl-st-metrology` | SMB share the Keyence and XRF PCs export into; ingestion watcher; part serial scan | mini PC | metrology bench (Expand phase) |
|
|
| `vinyl-st-audioqc` | Needle-drop capture and analysis pipeline | mini PC with audio interface, turntable, phono stage | listening/QC room (Expand phase) |
|
|
| `vinyl-st-pack1` | Runout code camera read, jacket barcode scan, BOM check, barcode verifier | mini PC, camera, scanner, verifier | packaging |
|
|
| `vinyl-st-ship` | Consignment assembly, carrier label and tracking capture | mini PC, label printer | shipping |
|
|
| `vinyl-nvr` | Frigate CCTV recording on the cctv VLAN (optional; a vendor NVR appliance is the alternative) | small server with GPU or Coral accelerator | IT room |
|
|
|
|
Office audio intake and gate approvals happen from ordinary staff laptops
|
|
on the `lan` VLAN through the same proxy; no dedicated machine. Stations
|
|
run a kiosk browser (`cage` plus Firefox or Chromium) pointed at the
|
|
application; badge readers and scanners are USB HID keyboard devices, so
|
|
there is no driver work. Sixteen plant machines at launch, about twenty-one
|
|
at maturity; every one is a directory and a line in the inventory.
|
|
|
|
## Networks
|
|
|
|
The plant is site `<siteId>` (next free number in the fleet's site table;
|
|
`cnx` is 1 and the public-WiFi site is 2, so 3 is the working assumption)
|
|
and owns `10.<siteId>.0.0/16`. VLANs follow the fleet convention; the
|
|
ids above 40 are proposals.
|
|
|
|
| VLAN | id | Members | `allowWan` | Rule |
|
|
|---|---|---|---|---|
|
|
| `mgmt` | 10 | switches, APs, Omada, IPMI, admin laptops on site | yes | fleet convention: trusted, reaches everything |
|
|
| `lan` | 20 | office laptops, printers | yes | proxy to factory app and Grafana, internet |
|
|
| `guest` | 30 | visitors, vendor laptops | yes | internet only |
|
|
| `servers` | 100 | `vinyl-core`, `vinyl-backup`, the servers-side NIC of every edge collector | yes (mesh, nix, journald, off-site backup) | no inbound from other VLANs; the gateway's own Caddy dials the app here |
|
|
| `stations` | 110 | `vinyl-st-*`, instrument PCs (Keyence, vision setup laptop), label printer | yes (mesh, deploys, journald) | proxy to the factory app only; `dhcp.reservations` for every station |
|
|
| `cctv` | 120 | cameras, NVR | no | reachable from `mgmt`; NVR UI via the proxy if wanted |
|
|
| `ot-press` | 210 | press and extruder PLCs and HMIs, vision controller, OT NIC of `vinyl-edge-press` | no | isolated segment, see below |
|
|
| `ot-util` | 220 | boiler, chiller, compressor, heat pump, HVAC BMS, meters, inverters, BESS BMS, OT NIC of `vinyl-edge-util` | no | isolated segment |
|
|
| `ot-galv` | 230 | rectifiers, bath transmitters, OT NIC of `vinyl-edge-galv` | no | isolated segment |
|
|
|
|
**OT segments are islands, not routes.** Each edge collector is dual-homed:
|
|
one NIC on `servers` (an access port on the cabinet switch), one NIC on
|
|
its OT segment. The OT segment lives on the cabinet's industrial switch;
|
|
if it must span cabinets it rides the building trunk as a VLAN with
|
|
`dhcp.enable = false` and `allowWan = false`, and the router's default
|
|
no-inter-VLAN policy keeps it closed. Either way the gateway forwards
|
|
nothing into or out of an OT segment; the collector is the only bridge,
|
|
and it only publishes. Zone sensors on RS485 loops terminate directly on
|
|
the collectors and have no IP presence at all. Cheap ESP32/MQTT sensor
|
|
nodes are acceptable for extra density in non-critical zones (on
|
|
`stations`, publishing to the core), never for alarmed zones.
|
|
|
|
**Stations never route to `servers`.** They reach
|
|
`https://factory.lpn1.cnx.network`, which Blocky resolves to the gateway
|
|
and Caddy forwards to `vinyl-core` on the servers VLAN, with the real
|
|
wildcard certificate the router module already issues. `proxy.allowVlans`
|
|
gains `stations`. The same proxy serves `grafana.lpn1.cnx.network` to the
|
|
office. This keeps the fleet's trust model intact: no VLAN talks to
|
|
another VLAN; the router talks to its backends.
|
|
|
|
Router module extensions the plant needs (small, and useful for the other
|
|
sites too):
|
|
|
|
1. **NTP served by the gateway** to every VLAN (chrony `allow` for the
|
|
site's `/16`, UDP 123 in the per-VLAN firewall set) so stations and
|
|
the NVR sync locally; today the gateway is only a chrony client.
|
|
2. **Static-only VLANs**: a `dhcp.enable = false` VLAN should still
|
|
accept `reservations` semantics for documentation, or simply be
|
|
documented as address-by-hand; either is fine, decide when writing it.
|
|
3. **Second WAN**: a 4G/5G modem as failover for the PPPoE session
|
|
(`wan.backupInterface` with a metric and a health check). Not needed
|
|
for launch because the plant runs without the internet; needed before
|
|
the customer portal is promised as always-on.
|
|
|
|
Vendor remote support (Newbilt and Pheenix both offer remote monitoring)
|
|
goes through an admin's SSH port-forward across the mesh into the relevant
|
|
edge collector and from there to the one PLC HMI they need, time-boxed and
|
|
logged. No machine ever gets a vendor's own cellular router or cloud agent
|
|
on an OT segment.
|
|
|
|
## What changes in the clan repo
|
|
|
|
Following the "Adding a site" procedure in the clan runbook, plus the
|
|
plant's own modules. Everything below is one pull request per machine
|
|
class, in this order.
|
|
|
|
```
|
|
clan.nix
|
|
fleet: + gw-lpn-1, vinyl-core, vinyl-backup, vinyl-edge-press,
|
|
vinyl-edge-util, vinyl-st-press1, vinyl-st-lab,
|
|
vinyl-st-pack1, vinyl-st-ship (then the rest as they come)
|
|
borgbackup:
|
|
roles.server.machines.vinyl-backup = { };
|
|
roles.client.machines.vinyl-core.settings.destinations = {
|
|
plant = { repo = "borg@vinyl-backup:vinyl-core"; };
|
|
offsite = { repo = "<storage-box or bk host>:vinyl-core"; };
|
|
};
|
|
roles.client.machines.vinyl-edge-press = { ... plant only ... };
|
|
# side benefit: control can now back up its VictoriaMetrics TSDB to
|
|
# vinyl-backup, closing the open item in the clan TODO
|
|
(no `internet` entries: plant machines sit behind PPPoE like gw-cnx-1
|
|
and are reached over the mesh)
|
|
|
|
modules/mesh-hosts.nix + every plant machine in the name list
|
|
modules/monitoring/server.nix
|
|
+ node_exporter targets for every plant machine
|
|
+ a second Grafana datasource "vinyl-vm" pointing
|
|
at [vinyl-core mesh]:8428 so fleet dashboards
|
|
can show plant series
|
|
modules/monitoring/alerts.nix
|
|
unchanged; host rules already cover new targets
|
|
machines/ns1/configuration.nix
|
|
+ dns-acme-gw-lpn-1-knot generator, keyFiles entry,
|
|
acl_acme_gw_lpn_1 scoped to _acme-challenge.lpn1
|
|
(copy the gw-cnx-1 blocks)
|
|
modules/dns/acme-gw-secret.nix
|
|
reused as-is with "gw-lpn-1"
|
|
modules/web-proxy.nix + virtualHosts."portal.<domain>" -> [vinyl-core mesh]:8080
|
|
(second cert + scoped TSIG key if <domain> is
|
|
not cnx.network, following the acme_web01 pattern)
|
|
|
|
machines/gw-lpn-1/ copy of gw-cnx-1: site "lpn", siteId 3, the VLAN
|
|
table above, proxy.services.factory and .grafana,
|
|
proxy.allowVlans = [ "mgmt" "lan" "stations" ]
|
|
machines/vinyl-core/ imports modules/vinyl/{core,metrics,factory-app,
|
|
alertmanager}.nix + monitoring/exporters.nix
|
|
machines/vinyl-backup/ imports modules/vinyl/backup.nix + exporters.nix
|
|
machines/vinyl-edge-*/ imports modules/vinyl/edge.nix with the cabinet's
|
|
Telegraf inputs and the OT NIC config
|
|
machines/vinyl-st-*/ imports modules/vinyl/station.nix with the kiosk URL
|
|
|
|
modules/vinyl/core.nix PostgreSQL (ZFS dataset, pg_hba for the app and
|
|
the replica), Mosquitto (per-client passwords from
|
|
vars generators), Telegraf persister (MQTT in,
|
|
VictoriaMetrics out), ZFS datasets and sanoid
|
|
modules/vinyl/metrics.nix VictoriaMetrics + vmalert + Grafana for the plant,
|
|
a sibling of monitoring/server.nix with the
|
|
process rules and dashboards under
|
|
modules/vinyl/dashboards/
|
|
modules/vinyl/alertmanager.nix
|
|
receivers: Telegram or LINE for the shift group,
|
|
SMS gateway for the critical set, email digest;
|
|
tokens from vars generators
|
|
modules/vinyl/factory-app.nix
|
|
the cnx-factory package as a flake input (like
|
|
nixos-mailserver is today), systemd service,
|
|
migrations on start, DB password from vars,
|
|
OIDC client secret from vars
|
|
modules/vinyl/edge.nix Telegraf with per-machine input lists, MQTT output
|
|
with disk buffer, chrony `allow` on the OT NIC,
|
|
networkd config for the two NICs, SFTP chroot for
|
|
vision images (press collector only)
|
|
modules/vinyl/station.nix cage + browser kiosk, auto-login, HID readers,
|
|
SMB share and ingestion watcher where needed
|
|
modules/vinyl/backup.nix borg server role settings, syncoid receive,
|
|
PostgreSQL replica, pgBackRest repo
|
|
modules/router/… the three extensions above
|
|
docs/src/vinyl.md the plant chapter of the runbook, + SUMMARY.md
|
|
```
|
|
|
|
Operational notes that already bit the fleet and apply here: `git add` a
|
|
new `machines/` directory before evaluating (untracked files are invisible
|
|
to the flake); run `clan vars generate <machine>` (YubiKey) before the
|
|
first install and again after adding a generator; check `facter.json` for
|
|
the real NIC names after install; the `clan` wrapper in the dev shell
|
|
updates machines sequentially so YubiKey prompts are not buried.
|
|
|
|
Repository layout, decided: infrastructure lives in `cnx-network-clan`;
|
|
the factory application gets its own repository `cnx-factory` in the B4L
|
|
organisation and is consumed as a flake input; this repository stays
|
|
documents and links to both. That satisfies the README's promise that the
|
|
project repository "hosts the software" without mixing Nix, Rust and
|
|
planning prose in one history.
|
|
|
|
## Data acquisition, source by source
|
|
|
|
Every row is also a purchasing clause: the "requirement" column is what
|
|
goes into the RFQ, per the traceability principle that machine data access
|
|
is bought, not retrofitted. Rates are planning values.
|
|
|
|
### Presses and extruders
|
|
|
|
| Source | Signals | Interface | Rate | Requirement to write into the RFQ |
|
|
|---|---|---|---|---|
|
|
| Newbilt Classic 12" automatic (Siemens PLC, vendor states SCADA/OPC UA) | cycle counter, machine state, cycle time, mold top/bottom temperatures, steam valve state, press pressure and force, extruder zone temperatures, screw speed, melt temperature and pressure, cake weight if measured, label feed state, alarm codes | OPC UA server on the PLC, read by Telegraf `inputs.opcua` | per cycle plus 1 Hz | OPC UA server licensed and enabled, full node list with units delivered with the machine, cycle counter exposed as a node, read access for our collector, no cloud agent required for warranty, PLC clock accepting NTP |
|
|
| Newbilt HRP 12 semi-automatic and VEX 50 extruder (PLC type unconfirmed) | as above where the PLC has them; on a relay-logic machine at least press closed/open, steam on/off, cycle count | OPC UA or Siemens S7 (`inputs.s7comm`); fallback: our own 4 to 20 mA sensors on a Modbus TCP I/O module and a cycle count from the mold-closed limit switch | per cycle plus 1 Hz | state what the PLC is and which of the two paths applies; the fallback sensor set is priced separately |
|
|
| Pheenix AD12 (Beckhoff PLC, remote monitoring supported) | same tag family as the Classic | OPC UA (Beckhoff TF6100) or ADS; `inputs.opcua` | per cycle plus 1 Hz | OPC UA license included, tag list delivered, remote monitoring routed through our mesh, not the vendor's |
|
|
| Hydraulic aggregate | system pressure, oil temperature, tank level, filter differential pressure | via the press PLC if exposed, else own transmitters on the I/O module | 1 Hz | ask whether the aggregate's sensors are on the PLC |
|
|
| Label dry oven | chamber temperature, timer state | Modbus if the controller has it, else a thermocouple on the I/O module | 1/min | |
|
|
| PVC grinder and regrind scale | run state, batch weight | scale with RS232 to `vinyl-st-press1`, run contact to I/O | per batch | scale must have a serial output |
|
|
|
|
### Utilities and energy
|
|
|
|
| Source | Signals | Interface | Rate | Requirement |
|
|
|---|---|---|---|---|
|
|
| Steam boiler (Thai-sourced) | pressure, steam temperature, feedwater temperature, drum level, burner or heater state, fuel or kW, blowdown events, water treatment conductivity and hardness alarm | Modbus RTU or TCP on the boiler controller | 1 Hz | Modbus register map delivered with the boiler; Thai boiler-regulation inspection data logged too |
|
|
| Steam accumulator | pressure | pressure transmitter to I/O | 1 Hz | |
|
|
| Chiller and cooling tower | supply and return temperatures, flow, compressor power, tower fan state, makeup water | Modbus | 1 Hz | register map |
|
|
| Feedwater heat pump | source and sink temperatures, power | Modbus | 1/10 s | register map |
|
|
| Compressor | line pressure, run hours, dew point | Modbus | 1/10 s | register map |
|
|
| HVAC / BMS | AHU states, setpoints, supply temperatures, filter differential pressure, positive-pressure room dP | BACnet/IP or Modbus from the BMS the HVAC contractor supplies | 1/min | the RFQ names BACnet/IP or Modbus TCP read access as a deliverable, with point list |
|
|
| Electrical | main incomer, feeders for press hall, chiller, HVAC, boiler, galvanics, packaging, IT; PV inverters (Deye, Modbus RTU/TCP register map is public); BESS state of charge, power, cell temperatures (through the Deye battery registers or the Gobel BMS RS485) | Modbus energy meters on each feeder; inverters direct, no cloud dongle | 1/10 s | meters specified in the electrical design, not added later |
|
|
| Water | mains or well meter, process tank level, boiler and cooling tower makeup meters, galvanics rinse recycle, fire tank level | pulse counters and level transmitters on I/O | 1/min | meters in the plumbing design |
|
|
|
|
### Environment
|
|
|
|
| Zone | Sensors | Alarm tier |
|
|
|---|---|---|
|
|
| cutting room | T, RH, PM2.5/PM10, room dP | tight, positive pressure |
|
|
| galvanics clean room | T, RH, PM, room dP | tight, positive pressure |
|
|
| master and stamper archive | T, RH, PM, water leak | tightest |
|
|
| press hall (two points at launch, four at maturity) | T, RH, PM | process spec 20 to 26 °C, 35 to 60 % RH |
|
|
| material lab and PVC store | T, RH | process |
|
|
| packaging and finished goods | T, RH, PM | process |
|
|
| listening/QC room | T, RH | comfort |
|
|
| IT room | T, RH, water leak | tight |
|
|
| office | T, RH, PM | comfort |
|
|
| outdoor reference | T, RH, PM2.5 | reference for burning-season episodes |
|
|
|
|
Industrial RS485 Modbus RTU transmitters on wired loops to the nearest edge
|
|
collector, one reading per minute, with a reference-grade PM instrument in
|
|
the archive or cutting room to check the cheaper sensors against. Every
|
|
zone alarm has two thresholds (warning, action) and a rate-of-change alarm
|
|
for the archive.
|
|
|
|
### Galvanics (Expand phase)
|
|
|
|
Rectifier current, voltage and ampere-hours per cell (Digital Matrix: ask
|
|
what the control rings and rectifiers expose; Pheenix Alpha: its plating
|
|
units are PLC-controlled with a network connection, protocol unstated, see
|
|
[galvanics and stamper prep](equipment/galvanics.md); Modbus is the
|
|
requirement),
|
|
bath temperature, pH and conductivity from transmitters, filtration and
|
|
agitation pump states, fume extraction running status (the interlock itself
|
|
stays hard-wired), rinse water conductivity and evaporator state. Titration
|
|
results are entered at a station against the bath ID. On the Pheenix
|
|
units, the machine's own alerts (filter cartridge change, nickel pellet
|
|
top-up, anode basket cleaning) are logged as bath events.
|
|
|
|
### Site systems
|
|
|
|
Access-control door events (badge ID, door, direction, time) from the
|
|
controller's API or export; fire alarm panel zone and fault contacts, BESS
|
|
off-gas and early-smoke detection, UPS on-battery and low-battery (NUT on
|
|
`vinyl-core` talking to the UPS) as dry contacts on `vinyl-edge-util`.
|
|
CCTV stays on its own VLAN with its own retention; it is never joined to
|
|
production data.
|
|
|
|
### What produces no telemetry
|
|
|
|
The Sibert stamper-prep machines (their current specifications, 2022 and
|
|
2023, list no data or network interface; Sibert's 2009 optical-disc punch
|
|
offered barcode and LAN options, and Sibert offered to carry features of
|
|
its fully automatic VSPa punch across to the VSP, so ask) and the shrink wrapper produce no machine data, and neither does the mastering chain on the Agnew route. They get
|
|
station check-ins: who, when, which part, which parameters, with files
|
|
attached (cutting parameters, DAW session export). The room environment
|
|
around them is logged by the sensors above.
|
|
|
|
If the lathe is a Sillitoe SDMS instead, its Space / Time Control software
|
|
(a vendor application on the cutting room's own PC, which stays an
|
|
appliance) writes files worth collecting: the master cue sheet (catalogue
|
|
number, side, matrix number, engineer, groove diameters per track), an
|
|
Open Groove export of the groove itself, the project file, and a
|
|
stylus-hours counter. They are dropped on an SMB share on the cutting
|
|
station and attached to the cutting session against the lacquer ID, the
|
|
same watcher pattern as the metrology exports below. See
|
|
[vendor landscape](suppliers/vendor-landscape.md).
|
|
|
|
## QC instrument integration
|
|
|
|
| Instrument | How results reach the database | Key | Open integration item |
|
|
|---|---|---|---|
|
|
| Inline optical inspection at the press unloader (Keyence or Cognex class, custom integration) | vision controller sends one result string per record over TCP to a receiver on `vinyl-edge-press`; images pushed by SFTP to the collector, synced to the core's image dataset | press ID + cycle counter, passed from the PLC to the vision trigger | the cycle counter hand-off between PLC and vision controller is an integrator deliverable and an acceptance test |
|
|
| Stamper metrology (Keyence VK-X, VR-6000, XRF gauge, profilometer) | vendor Windows PCs export CSV and raw 3D to an SMB share on `vinyl-st-metrology`; a watcher parses and posts | back-side Data Matrix serial, scanned at the bench before measurement | export format per instrument confirmed with Keyence Thailand |
|
|
| PVC material lab (moisture analyzer, spectrophotometer, balance, later torque rheometer, DSC, FTIR) | instruments with RS232/USB export are read at `vinyl-st-lab`; the rest are typed in with the printout photographed; CoA PDF attached | compound lot ID (printed label scanned at intake) | which benchtop models have serial export |
|
|
| Audio QC station | capture service on `vinyl-st-audioqc` records the needle-drop (PipeWire, FLAC to the core's capture dataset); analysis pipeline computes spectral-envelope match to the source master, click/pop count, noise floor, channel balance; scores posted | record key (stamper + press + cycle + side), typed or picked from the run list | first-of-kind pipeline, developed in Expand phase against launch-phase recordings |
|
|
| Packaging assembly check | camera with darkfield lighting reads the scribed dead-wax code (a handheld scanner will not read a scribed code on black vinyl reliably); handheld scanner reads the jacket barcode; barcode verifier grades it | order + record key | confirm dead-wax code readability with a sample before the packaging station is designed |
|
|
| Label print inspection | same vision family as the record label check; result per label lot | label lot ID | shared with the record inspection budget |
|
|
|
|
## The factory application
|
|
|
|
Working name `cnx-factory`, its own repository. Server-rendered web
|
|
application in Rust (axum, SQLx, minijinja templates, HTMX) on PostgreSQL,
|
|
the stack B4L already uses for its other in-house tooling, with the same
|
|
append-only ledger and hash-chained audit log pattern. It subscribes to
|
|
the MQTT event topics (cycle complete, machine alarm, instrument result)
|
|
and owns every write to the identity tables; Telegraf owns the writes to
|
|
VictoriaMetrics. Packaged as a flake so the clan repo pins it as an input.
|
|
|
|
**Demo.** [Factory console and order status](factory-console.html) (test
|
|
site `/demos/factory/`) shows what the application would look like on a
|
|
simulated plant day: the press floor with live counters per press, work
|
|
orders per station in the kiosk form described below (badge in, scan
|
|
what the step asks for, confirm), the optical inspection stream, audio samples, the material lab gate with a
|
|
lot on hold, stamper metrology and genealogy, the hash-chained event log,
|
|
the ERPNext link (every posting the floor makes: sales orders in by
|
|
webhook, job cards, batches and quality inspections, stock entries,
|
|
downtime entries, delivery notes and invoices, through the outbox, with
|
|
the ERPNext documents on each order and work order),
|
|
and the customer portal of the same orders (progress through the three
|
|
gates, copies pressed so far, the lab certificate of the compound lot,
|
|
listening checks). All orders, values and IDs are simulated; limits are
|
|
illustrative until the acceptance specifications are written.
|
|
|
|
Modules, in build order:
|
|
|
|
1. **Orders and releases**: customer, order, release, sides, formats, run
|
|
quantity, bill of materials (record, inner sleeve, jacket, inserts,
|
|
shrink), due dates.
|
|
2. **Audio intake and the client vault**: upload via portal or SFTP with
|
|
checksums, versions, handler, technical QA form; files in an encrypted,
|
|
access-logged dataset separate from everything else.
|
|
3. **The three artist gates**: master plan, master, test pressing; each is
|
|
an event with who approved, when, from which portal session or on whose
|
|
verbal instruction, and nothing downstream is schedulable without it.
|
|
4. **Mastering and cutting sessions**: engineer, chain settings, lacquer
|
|
IDs, dead-wax scribe text and fiducial applied, microscope check result.
|
|
5. **Metal genealogy**: lacquer to father to mother to stamper, or bridged
|
|
supplier plus batch; per-part metrology history; stamper lifecycle
|
|
(mounted on which press, play count, retired why), archive location
|
|
(box and shelf position in the mobile shelving,
|
|
[facility security](facility-security.md#archive-room-mobile-shelving)),
|
|
stamper sets grown per mother.
|
|
6. **Materials**: PVC lots with the lab gate and CoA, regrind batches,
|
|
label lots with print inspection and oven pre-dry, carton and sleeve
|
|
lots with incoming QC.
|
|
7. **Press runs and cycles**: a run binds order, stamper pair, press,
|
|
operator, shift, PVC lot in the hopper, regrind batch, label lot; each
|
|
cycle event from the PLC becomes a row keyed by press + counter with
|
|
the key press values and the zone environment snapshot; the record key
|
|
is derived from it per side.
|
|
8. **QC verdicts**: optical per record, audio per sampled record, lab per
|
|
lot, packaging per unit; reject reasons; test-pressing hold state.
|
|
9. **Finishing, packaging, fulfillment**: assembly checks, consignment
|
|
against the BOM, carrier and tracking per shipment or dropship end
|
|
customer.
|
|
10. **Customer portal**: per-order timeline of everything above, scoped so
|
|
one customer sees only its own orders; PDF export of the trace. The
|
|
public front of order intake (price calculator, uploads with
|
|
automatic checks, fan-funded campaigns) is planned in
|
|
[artist platform](artist-platform.md), including why its public
|
|
side should not depend on the plant's WAN link.
|
|
11. **Reports and signals**: yield per stamper, reject rate drift, audio
|
|
score trend versus play count (the automatic retirement signal),
|
|
energy per record, environment excursions overlaid on runs (the
|
|
curves fetched from VictoriaMetrics for the run's window).
|
|
|
|
Core entities: `customer`, `order`, `release`, `side`, `audio_asset`,
|
|
`gate_approval`, `session`, `lacquer`, `father`, `mother`, `stamper`,
|
|
`stamper_measurement`, `press`, `press_run`, `press_cycle`, `record`,
|
|
`optical_result`, `audio_result`, `pvc_lot`, `regrind_batch`, `label_lot`,
|
|
`carton_lot`, `lab_result`, `packaging_check`, `shipment`, `operator`,
|
|
`shift`, `event` (the hash chain).
|
|
|
|
Where a module touches a business document, the document is ERPNext's
|
|
and the module holds its production side: the order is an ERPNext sales
|
|
order, a run is a work order, each station step is a job card, a PVC lot
|
|
is a batch with its quality inspection, a shipment is a delivery note
|
|
([ERPNext](#erpnext-the-plants-erp)).
|
|
|
|
Stations use the same application in kiosk mode: an operator taps a badge
|
|
(the same card the access-control system issues, so one ID) and scans
|
|
whatever the step asks for. No forms longer than one screen.
|
|
|
|
## ERPNext: the plant's ERP
|
|
|
|
Decided October 2026: **the whole plant runs ERPNext**, the open-source ERP
|
|
on the Frappe framework (GPLv3, no licence fee; version 15 is the current
|
|
stable line). It closes the item this plan had left for after launch, and
|
|
it sets a rule for everything we build: **every tool has to interact with
|
|
ERPNext**, through the integration described here.
|
|
|
|
### Who owns what
|
|
|
|
| Object | Master | The other side keeps |
|
|
|---|---|---|
|
|
| Customers, contacts, addresses | ERPNext | the factory application scopes the portal by the ERPNext customer; the artist platform links each account to one |
|
|
| Price list, item prices, pricing rules | ERPNext | the artist-platform calculator reads them; the demos' price list becomes an ERPNext price list |
|
|
| Quotations, sales orders, invoices, payments (fan pre-orders in the ring-fenced account included) | ERPNext | the artist platform creates them; the factory application reads the order |
|
|
| Items and BOMs: one item per release and format (record, sleeves, jacket, labels, compound per copy), raw materials and packaging | ERPNext | the factory application reads the BOM of the run |
|
|
| Work order per press run; job cards per operation (cutting, plating, stamper prep, test pressings, pressing, packing); workstations (presses, lathe, plating cells, Sibert line, packing) | ERPNext | the stations' work orders are these job cards: kiosk scans and times post back as time logs and completed quantity |
|
|
| Stock: compound lots as batches, label and carton lots, finished goods, warehouses including the archive | ERPNext | the factory application knows which lot sat in which hopper for every cycle and posts the consumption |
|
|
| Incoming lab gate per compound lot | Quality Inspection in ERPNext, with a template carrying the parameters and limits of [quality automation](quality-automation.md) | instrument files and the CoA scan in the factory application |
|
|
| Metal parts: lacquers, fathers, mothers, stampers | serial numbers in ERPNext for location and value (warehouse: archive, rack, press) | genealogy, metrology, play counts and stamper sets per mother in the factory application |
|
|
| Gates, per-cycle rows, per-record optical results, audio scores, telemetry, environment, the hash-chained event log, client audio | factory application and VictoriaMetrics only: thousands of rows a day are not ERP documents | ERPNext gets the totals: good, rejected and scrap to regrind per job card and run |
|
|
| Stops | Downtime Entry in ERPNext per stop (workstation, reason, from, to) | the PLC detail |
|
|
| Deliveries | delivery notes in ERPNext | carrier and tracking per consignment on both sides |
|
|
| Assets: presses, lathe, galvanics, Sibert line, utilities, PV and BESS, IT | ERPNext asset register, depreciation, maintenance schedules | telemetry and run hours |
|
|
| Purchasing, suppliers, accounting, Thai VAT and withholding tax, payroll | ERPNext, with the HR app for payroll | nothing |
|
|
|
|
### How every tool interacts with it
|
|
|
|
1. **One integration layer.** A typed client for the Frappe REST API
|
|
(`/api/resource/<DocType>` and whitelisted methods) lives in
|
|
`cnx-factory` and is the only path in or out. Each tool authenticates
|
|
as its own integration user (API key and secret in clan vars) whose
|
|
role grants only the document types it writes. ERPNext pushes changes
|
|
(submitted sales orders and work orders, purchase receipts, items and
|
|
prices) to the factory application with Frappe's webhooks.
|
|
2. **Outbox, not live calls.** Every posting is written to an outbox
|
|
table in the same transaction as the floor event and sent by a worker
|
|
with retries; each carries the event's ID in a custom field, so a
|
|
retry never creates a second document. ERPNext down for an upgrade
|
|
stops nothing on the floor; the outbox catches up (decision 8
|
|
extended).
|
|
3. **Business-sized postings.** Per job card, per run, per lot, per
|
|
stop, per shift. Never per record or per cycle.
|
|
4. **ERPNext names are the shared keys** (sales order, work order, job
|
|
card, batch, serial number). The factory application keeps its own IDs
|
|
for records, cycles and parts and stores the ERPNext name beside them;
|
|
the record key on the sleeve never changes.
|
|
5. **Customisation as code.** Custom fields, document types, print
|
|
formats and webhooks ship as one Frappe app in our Gitea (working name
|
|
`cnx_erp`), installed and upgraded by the deployment, never clicked
|
|
into production by hand.
|
|
6. **Nightly reconciliation** of totals per work order and per batch in
|
|
both directions; a mismatch is an alarm, not a silent fix.
|
|
7. **The public side never reaches ERPNext directly.** The artist
|
|
platform's public process (see [artist platform](artist-platform.md))
|
|
posts through the plant's API over the mesh; payment-provider webhooks
|
|
land on the public process and become payment entries through the
|
|
same outbox. No ERPNext endpoint is published.
|
|
8. **New tools follow the same rule.** A dashboard, a script or a kiosk
|
|
app built later uses the same client and an integration user of its
|
|
own. A tool that cannot read from or write to ERPNext this way is not
|
|
finished.
|
|
|
|
### Hosting on NixOS
|
|
|
|
- ERPNext has no package or module in nixpkgs (the request,
|
|
NixOS/nixpkgs#241921, has been open since 2023). Community Nix packaging
|
|
exists: Avunu's [frappe-nix](https://github.com/Avunu/frappe-nix) (a
|
|
NixOS module `services.frappe` with per-site systemd units, a uv2nix
|
|
Python environment and a database snapshot before every migration) and
|
|
pub.solar's erpnext-nix. Plan: start from frappe-nix, pinned as a flake
|
|
input in the clan repo. Fallback: the official frappe_docker images
|
|
declared as `virtualisation.oci-containers` on the same NixOS machine,
|
|
still declared in the clan, never a hand-built Docker host. Both
|
|
checked October 2026.
|
|
- **MariaDB**, ERPNext's standard database (10.11 on the version 15
|
|
stack), with Redis. PostgreSQL support in Frappe is growing but is not
|
|
ERPNext's main path, so the plant runs two database engines. Backups:
|
|
Frappe's site backup plus mariabackup to `vinyl-backup` and the
|
|
off-site Borg set, and the quarterly restore drill includes ERPNext.
|
|
- **On site** (decision 8): machine `vinyl-erp`, reached by the office
|
|
through the gateway proxy at `erp.lpn1.cnx.network`; staff sign on with
|
|
the plant's OIDC provider (Frappe supports custom OAuth providers for
|
|
login); the external accountant over the mesh.
|
|
- **Upgrades** run on a restored copy first; the clan pins the ERPNext
|
|
version, so a major upgrade is a reviewed change, not a surprise.
|
|
|
|
### Thai localisation and a partner
|
|
|
|
- [erpnext_thailand](https://github.com/ecosoft-frappe/erpnext_thailand)
|
|
(Ecosoft, Bangkok; version 15; Frappe Cloud marketplace, checked
|
|
October 2026): Thai VAT including service VAT at payment, withholding
|
|
tax and its certificates, PND and purchase and sales tax reports for
|
|
the Revenue Department, billing notes, deposit invoices and Bank of
|
|
Thailand exchange rates.
|
|
- **Ecosoft** (ecosoft.co.th) is a Frappe certified partner and the
|
|
maintainer of that app; ask it for an implementation and support quote
|
|
alongside doing it in-house. Its site shows an ERPNext rollout at a
|
|
firm named Lamphun Plastpack.
|
|
- **Payroll**: Thai social security and PND 1 are not in that app's
|
|
list; check what Frappe HR covers with configuration and what Ecosoft
|
|
uses.
|
|
- **BOI**: machinery and raw materials imported under promotion have to
|
|
be accounted for ([BOI](boi.md)); the asset register and the stock
|
|
ledger are where that record lives. Check the raw-material reporting
|
|
the BOI asks for against what ERPNext produces before go-live.
|
|
|
|
### Phasing and cost
|
|
|
|
- **Now:** a dev ERPNext next to the dev trio behind `gw-cnx-1`, the
|
|
`cnx_erp` app skeleton, and the integration client in `cnx-factory` v0
|
|
against it.
|
|
- **From the first capex invoice:** accounting, purchasing and the asset
|
|
register, so the BOI asset list and depreciation start there. The
|
|
build's cost entries in proplan (project CNX-01) are the source for
|
|
capitalising the plant at handover.
|
|
- **Commissioning and launch:** items, BOMs, workstations, inspection
|
|
templates, work orders and job cards; stations post job cards.
|
|
- **Expand:** serial numbers for in-house metal parts; payroll if it did
|
|
not start earlier.
|
|
|
|
Estimates: no licence; `vinyl-erp` 2,000 to 4,000 USD of hardware (none
|
|
if it runs as a VM); ERPNext configuration (chart of accounts, Thai tax,
|
|
items, BOMs, workstations, inspection templates, print formats) 1 to 2
|
|
person-months or a partner quote; the integration and the `cnx_erp` app
|
|
1.5 to 2.5 person-months; Nix packaging and the clan module 0.5 to 1
|
|
person-month.
|
|
|
|
## Observability and alerting
|
|
|
|
Two tiers, split by where the data is:
|
|
|
|
- **Process, on site** (`vinyl-core`: VictoriaMetrics, vmalert,
|
|
Grafana, Alertmanager): zone environment excursions and rate of change,
|
|
boiler pressure and level, chiller return temperature, compressed-air
|
|
pressure, hydraulic pressure, reject rate per run above baseline,
|
|
stamper approaching its play-count limit, PV or BESS faults, fire and
|
|
BESS detection contacts, UPS on battery. Works with the WAN down.
|
|
- **Hosts, on the fleet** (`control`): node_exporter and journald from
|
|
every plant machine over the mesh, the existing `ScrapeTargetDown`,
|
|
`RootFilesystemFull`, `BackupJobFailed`, `BackupStale` rules apply to
|
|
them without changes.
|
|
- **Data-silence alarms** in both tiers: a collector that has published
|
|
nothing for five minutes is an alarm, whatever the last value was.
|
|
|
|
Routing: Telegram or the LINE Messaging API for the shift group, email
|
|
digest daily, SMS through a Thai gateway for the critical set (boiler,
|
|
fire, BESS, archive climate) with an on-call rota. `control`'s vmalert can
|
|
drop `notifier.blackhole` and point at the same Alertmanager over the
|
|
mesh, which closes the fleet's own alert-delivery item. The animated
|
|
[plant operation page](plant-operation.html) is the design target for the
|
|
live overview: once the series exist it is fed from VictoriaMetrics
|
|
instead of its simulation.
|
|
|
|
## Backup, retention and recovery
|
|
|
|
- **Hourly ZFS snapshots** on `vinyl-core` (sanoid), replicated to
|
|
`vinyl-backup` (syncoid) for the file datasets and the PostgreSQL data
|
|
directory.
|
|
- **Nightly Borg** (clan `borgbackup`) from `vinyl-core` and the edge
|
|
collectors to `vinyl-backup`, and from `vinyl-core` to the off-site
|
|
destination as a second entry in `destinations`; repokey encryption as
|
|
the fleet already does.
|
|
- **PostgreSQL point-in-time recovery**: pgBackRest with WAL archiving to
|
|
`vinyl-backup`, its repository included in the off-site Borg set; a
|
|
streaming replica on `vinyl-backup` for fast failover.
|
|
- **VictoriaMetrics**: `vmbackup` snapshots to `vinyl-backup` nightly; the
|
|
fleet's own open item (backing up `control`'s TSDB) gets a destination
|
|
at the same time.
|
|
- **ERPNext**: Frappe's site backup and mariabackup to `vinyl-backup`
|
|
nightly, in the off-site Borg set, restored in the same drill.
|
|
- **Configuration**: the clan repo, `cnx-factory` and the `cnx_erp` app in
|
|
Gitea; any machine is rebuilt from them.
|
|
- **Restore drill quarterly**: restore the database to a scratch VM and
|
|
open a random order's trace in the portal. A backup that has never been
|
|
restored does not count. The `BackupStale` rule already watches the
|
|
timers.
|
|
|
|
| Data | Retention |
|
|
|---|---|
|
|
| genealogy, orders, gates, QC verdicts, event log | life of the customer relationship, never deleted while the metal parts exist |
|
|
| per-cycle press rows | forever (small) |
|
|
| 1 Hz machine telemetry | 2 years in VictoriaMetrics (`retentionPeriod`); per-cycle rows carry the summary forever |
|
|
| zone environment | 5 years (separate VM retention filter, or a second small instance) |
|
|
| energy and water | forever (small; also summarised per shift into PostgreSQL) |
|
|
| audio captures and inspection images | 2 years, longer for any record with a complaint |
|
|
| client audio | per contract; default deleted 12 months after final delivery, or on request |
|
|
| access-control events | 1 year |
|
|
| CCTV | 30 days |
|
|
| host logs and metrics | fleet defaults: 30 d logs, 180 d metrics on `control` |
|
|
|
|
PDPA applies to badge events, CCTV and customer contact data: a written
|
|
retention policy, role-based access, and export/deletion on request. The
|
|
customer portal shows operator roles, never operator names.
|
|
|
|
## Security
|
|
|
|
- OT segments have no gateway interface and no internet; edge collectors
|
|
are the only bridge and they only publish.
|
|
- Stations and office reach the application only through the gateway's
|
|
Caddy, with the wildcard certificate the fleet already issues; the
|
|
servers VLAN accepts nothing from other VLANs.
|
|
- Staff sign-on through an OIDC provider on `vinyl-core` (Pocket ID,
|
|
passkeys; the fleet has no identity provider yet, so this is the first)
|
|
with roles: admin, engineer, operator, sales, customer. Stations
|
|
authenticate by badge for attribution, not for privilege.
|
|
- Secrets only in clan vars (YubiKey-gated, `admins` group); nothing in
|
|
plain text in either repo.
|
|
- Client audio: separate encrypted ZFS dataset, SFTP or portal upload with
|
|
checksums, every read logged, a written policy sales can show a label.
|
|
- UPS on `vinyl-core`, `gw-lpn-1`, the switches, the edge collectors and
|
|
the press PLCs (already required by facility security); NUT triggers a
|
|
clean shutdown before the battery is exhausted.
|
|
- Admin access rides the mesh, as for every fleet machine; public SSH is
|
|
never opened.
|
|
|
|
## Procurement clauses
|
|
|
|
To go into every RFQ that leaves from [roadmap](roadmap.md) items 4 to 8:
|
|
|
|
1. **Presses (Newbilt, Pheenix)**: OPC UA (or S7/ADS) read access licensed
|
|
and enabled; the full tag list with units and the cycle counter as a
|
|
tag; a defined hand-off of the cycle counter to the inspection trigger;
|
|
remote support through our mesh; no cloud agent as a warranty
|
|
condition; PLC clock accepting NTP.
|
|
2. **Boiler, chiller, compressor, heat pump (Thai vendors)**: Modbus RTU or
|
|
TCP with the register map delivered; alarm contacts wired to a terminal
|
|
strip for us.
|
|
3. **HVAC contractor**: a BMS with BACnet/IP or Modbus TCP read access and
|
|
a point list; zone sensor wiring and room dP sensors in scope; sensor
|
|
loops terminated in the edge cabinets.
|
|
4. **Electrical contractor**: Modbus energy meters on the feeders listed
|
|
above; RS485 to the inverters and BESS BMS without cloud dongles;
|
|
structured cabling, a VLAN-capable industrial switch in every cabinet,
|
|
and Omada-managed switches and APs for the building so the gateway's
|
|
existing Omada controller covers the site.
|
|
5. **Galvanics (Digital Matrix, Pheenix Alpha)**: what the rectifiers,
|
|
control rings and PLCs expose, and how; Modbus preferred; remote support
|
|
through our mesh, no vendor cloud. **Sibert**: whether the VS machines
|
|
can take a stamper serial in and write the centring result out, and
|
|
which features of the fully automatic VSPa can move to the VSP.
|
|
6. **Metrology (Keyence Thailand)**: confirmed CSV/3D export paths and
|
|
whether the instrument software can write to a network share unattended.
|
|
7. **Vision integrator**: result string format, image push over SFTP,
|
|
cycle counter correlation, and an acceptance test defined in records not
|
|
in features.
|
|
8. **Access control and fire**: API or export for door events; dry
|
|
contacts for every alarm class; BESS detection outputs.
|
|
|
|
## Build order
|
|
|
|
Aligned with the [roadmap](roadmap.md) phases.
|
|
|
|
**Now (roadmap Phase 0 and 1): design and dev fleet.**
|
|
|
|
- Put the procurement clauses above into the Newbilt re-quote, the
|
|
Pheenix, Digital Matrix and Thai utility RFQs.
|
|
- In the clan repo: reserve the site id and VLAN table, write
|
|
`modules/vinyl/{core,metrics,edge,station}.nix`, and stand up a dev
|
|
trio at the Chiang Mai site behind `gw-cnx-1` (a `vinyl-core` stand-in,
|
|
one edge collector with a Modbus and OPC UA simulator, one station).
|
|
This proves the modules, the mesh path and the proxy before a single
|
|
plant machine exists, and the machines move to the plant when it does.
|
|
- Land the three router-module extensions on `gw-cnx-1` first, where they
|
|
are also useful.
|
|
- Stand up a dev ERPNext beside the dev trio (frappe-nix module, or the
|
|
container fallback) and build the integration client against it from
|
|
`cnx-factory` v0 onward.
|
|
- Start `cnx-factory` v0 against a press simulator (the state machine in
|
|
the plant operation page is the seed: same states, same cycle timing) so
|
|
orders, gates, genealogy and press cycles can be exercised end to end
|
|
without hardware; package it as a flake from the first commit.
|
|
- Wire the Alertmanager and one real receiver; point `control`'s vmalert
|
|
at it.
|
|
|
|
**Build and commission (roadmap Phase 3).**
|
|
|
|
- `gw-lpn-1`, switches, `vinyl-core` and `vinyl-backup` installed before
|
|
the machinery arrives, so vendor commissioning engineers plug into a
|
|
finished network; ns1 gets the gateway's ACME key at the same time.
|
|
- Zone sensor loops, meters and edge cabinets installed by the electrical
|
|
and HVAC contractors during fit-out; environment logging is live before
|
|
pressing starts, which gives a baseline.
|
|
- Machinery commissioning includes data acceptance: every tag on the list
|
|
readable, the cycle counter increments in the database with each cycle,
|
|
data-silence alarms proven by pulling a cable.
|
|
- Application v1 live: intake, gates, bridge-stamper genealogy, press runs
|
|
and cycles, PVC lot gate, packaging scan, shipments; stations at press,
|
|
lab, packaging and shipping.
|
|
- Off-site backup destination live and a first restore drill done.
|
|
|
|
**Launch (roadmap Phase 4).**
|
|
|
|
- Optical inspection integrated and correlated; reject reasons flowing.
|
|
- Customer portal opened to the first labels through `web01`.
|
|
- Energy and environment dashboards used for the first PV and HVAC
|
|
tuning.
|
|
|
|
**Expand (roadmap Phase 5).**
|
|
|
|
- `vinyl-edge-galv`, bath sensors, rectifier data, metrology ingestion at
|
|
`vinyl-st-metrology`; genealogy switches from bridged to in-house.
|
|
- Audio QC station and pipeline; retirement signal live.
|
|
- Automated press line joins the same collector with the same tag family.
|
|
- Biomass boiler on the same Modbus path as its predecessor.
|
|
|
|
## Rough cost
|
|
|
|
Planning estimates, September 2026, excluding the QC instruments (those
|
|
are in quality automation) and excluding the vendors' OPC UA license
|
|
options, which come back with the quotes. Fleet pieces already paid for
|
|
(`control`, `ns1`, `ns2`, `web01`, the mesh) cost the plant nothing extra.
|
|
|
|
| Block | Estimate (USD) |
|
|
|---|---|
|
|
| `gw-lpn-1` (Topton 1U, fleet standard) | 300 to 500 |
|
|
| `vinyl-core` and `vinyl-backup` servers, UPS, rack | 8,000 to 15,000 |
|
|
| `vinyl-erp` server (none if it runs as a VM) | 2,000 to 4,000 |
|
|
| Omada switches and APs for the building, industrial switches in the cabinets, cabling allowance | 4,000 to 8,000 |
|
|
| 3 edge collectors with I/O modules and enclosures | 3,000 to 6,000 |
|
|
| 8 station kiosks with readers, scanners, one barcode verifier, one darkfield camera | 8,000 to 15,000 |
|
|
| zone environment sensors (12 to 16 points), room dP, reference PM instrument | 4,000 to 8,000 |
|
|
| energy meters (10 feeders), water meters, level transmitters | 3,000 to 6,000 |
|
|
| audio QC hardware (turntable, arm, cartridge, phono stage, interface) | 2,000 to 5,000 |
|
|
| off-site backup storage, per year | 100 to 500 |
|
|
| **Hardware total** | **~34,000 to 68,000** |
|
|
| `cnx-factory` to v1 (orders through shipments, stations, portal) | 4 to 6 person-months |
|
|
| ERPNext: configuration with Thai tax, the integration and the `cnx_erp` app, Nix packaging | 3 to 5.5 person-months, or a partner quote for the configuration |
|
|
| Clan modules, router extensions, collectors, dashboards and rules to commissioning | 1 to 2 person-months |
|
|
| Vision, metrology and audio pipeline integration (Launch and Expand) | 3 to 5 person-months, plus integrator fees in the vision quote |
|
|
|
|
Headcount note: [scale](scale.md) lists no IT or automation role in the
|
|
15 to 25 people. Running this needs one automation and IT engineer from
|
|
Build onward, part-time at launch; add it to the headcount.
|
|
|
|
## Open questions
|
|
|
|
- Site code and id: `lpn` / 3 assumed; confirm when the site is chosen and
|
|
the fleet site table is updated.
|
|
- Semi-automatic press PLC: does the HRP 12 expose data, or is the sensor
|
|
fallback needed? (Ask in the Newbilt re-quote.)
|
|
- Cycle counter hand-off to the vision system: PLC output, network
|
|
message, or time correlation at the unloader? Decided with the vision
|
|
integrator, tested at acceptance.
|
|
- HVAC BMS protocol and who owns the zone sensors: HVAC contractor or us.
|
|
- Off-site backup destination: Hetzner Storage Box (cheapest) or a storage
|
|
VPS that is also a mesh peer (uniform tooling). Data residency for
|
|
customer audio may argue for a Bangkok provider instead.
|
|
- Customer portal domain: the plant's brand domain (new zone on ns1/ns2,
|
|
second cert on `web01`) or a name under `cnx.network` for launch.
|
|
Internal testing has its own domain, `cnxvinyl.lol` (registered
|
|
2026-10-02, DNS on Cloudflare): a static demo site built from
|
|
[`site/`](../site/README.md) and served by the buildfor.life web server,
|
|
not a plant service. The brand domain decision stays open.
|
|
- Portal availability during a plant WAN outage: accept it (the portal is
|
|
a view of live data) or add a read replica on the fleet side later.
|
|
- ERPNext (decided October 2026, see its section): own machine or a VM
|
|
on `vinyl-core`; frappe-nix or the container fallback; Ecosoft or
|
|
in-house for the configuration; payroll localisation; the BOI
|
|
raw-material reporting; when the company's books move into it.
|
|
- Dead-wax code readability on black and coloured vinyl with a camera:
|
|
test with samples before the packaging station is designed.
|