The open stack: from firmware to tape

Research note, October 27, 2026, by Alexander Grabowski (B4L). Sources were retrieved on October 27, 2026. Figures are cited, and our own estimates are marked.

This note covers the software and hardware of the investor's facility, from server firmware to tape, and how much of it can be open. B4L prepared it as consultant; the investor would build, own, and run the facility. Money is given in THB, with USD in parentheses at 33 THB per USD, the B4L planning rate, and euro prices are converted at the ECB rate of 37.821 THB per EUR (October 26, 2026).[494] Prices are foreign list or reseller prices, without Thai VAT or duty, unless they are marked as Thai.

In short

• Almost every layer can run on open software: SONiC on white-box switches (about 156 models from about 27 vendors), Linux or NixOS, Incus or Proxmox, Ceph, and Prometheus.[391] What stays closed is NVIDIA's CUDA, its user-space driver and GPU firmware, and the firmware of drives, tape drives, and most servers.[508]
• A 64x800G white-box switch with enterprise SONiC lists at USD 46,800 (1.54 million THB), and NVIDIA's SN5600 at USD 74,457 (2.46 million THB).[509][510] Ethernet already takes more than two thirds of switch sales for AI clusters.[511] Our switch bill for a 1 MW GPU hall comes to about 23 million THB (USD 0.70 million) with white boxes and 34 million THB (USD 1.03 million) with SN5600s, before optics (our estimate).
• OPNsense is BSD-licensed, but Deciso's largest 1U appliance moves 21 Gbps port to port and about 7.5 Gbps with intrusion prevention.[512][513] At 100G, the open design filters on switches and Linux routers, detects attacks with FastNetMon, and sends floods to upstream scrubbing, which True IDC sells in Thailand.[514][515]
• Ceph, under the Linux Foundation, provides S3 storage with an Object Lock mode that not even an administrator can undo through S3, while the community edition of MinIO is archived.[516][517][518] One petabyte on Ceph plus two copies on LTO-9 tape costs about 144 THB (USD 4.4) per TB-month in hardware and power (our estimate, staff excluded), against 185 THB at Hetzner and 770 THB at NIPA Cloud.[519][413]
• Germany's BSI wants geo-redundant sites about 200 km apart and never under 100 km.[520] Chiang Mai and the Korat/Chaiyaphum belt are 460 to 520 km apart (our calculation). Banks on a foreign cloud must keep backups in Thailand or a third country.[428]
• Monitoring can be fully open. Prometheus, VictoriaMetrics, Thanos, and NVIDIA's DCGM exporter are Apache-2.0, and Grafana and Loki are AGPL.[521][522][523] NetBox is the inventory tool to standardize on, as openDCIM and Ralph fade.[524][525]

What counts as open

The aim is a stack as open as possible, from the operating system to the switches and firewalls, with the most open alternative named wherever a closed part cannot be avoided. We test "open" on four points (our definition):

1. License. The source is under an OSI-approved license (Apache, MIT, BSD, GPL, AGPL, LGPL, or MPL). Source-available licenses and source behind a subscription do not count.
2. Governance. A neutral foundation (Linux Foundation, CNCF, OpenInfra) is safer than a single company, as MinIO shows.
3. Vendor choice. The software runs on hardware from several makers (ONIE switches, OCP racks).
4. Firmware. The server's management controller (BMC) and boot firmware are open. Few products meet this test today.

The table is our judgment, built on the sources cited below.

Servers, firmware, and the operating system

Open hardware and firmware

Open Rack V3 defines a rack with a 48 V busbar that takes both OCP and standard 19-inch units, so parts from different vendors fit together.[526] Southeast Asia's first OCP Experience Center opened in Singapore in June 2024, and Open Computing Singapore distributes OCP gear there; QCT's nearest office is also in Singapore.[527][528] We found no Thai channel for OCP servers. Our reading is that phase 1 should use 19-inch servers and that the hall should get a power design ready for ORv3.

Open firmware lags behind. IBM ships OpenBMC on its Power servers, managed through Redfish.[529] In April 2024 Supermicro announced its "intention" to adopt OpenBMC and demonstrated AMD's openSIL with coreboot on one board.[530] openSIL is a proof of concept, "NOT for use in production firmware".[531] Our advice is to require Redfish on every server, to prefer OpenBMC where it is offered, and not to count on open boot firmware.

Hetzner shows what owning the server design buys: servers of its own design without unneeded parts, kept for about eight years through repair and resale.[490] The business hosting note sets out the benchmark.

NixOS and clan.lol

NixOS describes every machine in code, so servers can be rebuilt identically (our summary). Flying Circus, a German managed-hosting firm, runs its cloud on its own NixOS platform in an ISO 27001 facility.[532] We found no large HPC center or GPU cloud running NixOS.

GPUs are the weak point. CUDA is unfree and off by default in NixOS, and prebuilt packages come from a community cache.[533] The CUDA team, which is not an official NixOS project, tests on an RTX 4000 Ada and a GTX 1080.[534] It says that support for GB200-class clusters "can only be sustainably offered on a commercial basis".[535] Grace Hopper and Blackwell GPUs must use NVIDIA's open kernel modules, while the user-space driver stays unfree.[536] Our reading is that NixOS suits CPU, storage, network, and hosting servers. For B200 or GB300 nodes, the investor should budget for paid Nix support and its own tests, or use a distribution with vendor support, such as Rocky Linux 10, which is supported to May 2030.[537]

clan.lol manages a NixOS fleet declaratively and peer to peer, without a central controller, and it has secrets and remote installation built in.[538]

What follows is B4L's own experience, not a description of the investor's facility. B4L runs its own servers as a NixOS fleet managed with clan.lol, with services in Incus guests, Kanidm for identity, Prometheus-style monitoring, and a NetBird mesh. That proves the pattern at small scale, but not yet at 1 to 2 MW.

Virtualization and orchestration

The rescue system note builds on these parts: iPXE boots a NixOS rescue image over a provisioning network, Redfish arms a one-time network boot, Incus gives VPS customers a console and an ISO boot, and a thin control service on NixOS ties them to the portal, with Ironic standalone kept as the option for a larger, mixed fleet.

AMD or NVIDIA

NVIDIA's open kernel modules are MIT/GPLv2, and Blackwell and later GPUs run only with them, on top of closed GSP firmware.[553] CUDA and the user-space driver stay closed.[508] AMD licenses ROCm per component, with open parts on GitHub and some proprietary binaries.[554] Its MI355X platform is an industry-standard UBB 2.0 board with eight OAM modules, and GIGABYTE's Thai site lists a 256-GPU MI355X pod at up to 70 kW per rack, without a price.[555][556] A secondary source put an MI300X at USD 10,000 to 15,000 in early 2024, against USD 25,000 to 40,000 for an H100, with a hidden cost for porting CUDA code.[557] Our reading is that AMD is the more open path and a second source of supply, and that the Thai models should first be tested on rented AMD GPUs.

Network: white-box switches with SONiC

SONiC, white boxes, and prices

SONiC moved to the Linux Foundation in April 2022. Microsoft founded it and chairs its board and its technical committee.[558][559] The community list counts about 156 switch models from about 27 vendors: 104 on Broadcom chips, 14 on NVIDIA, 13 on Marvell, and 10 on Intel Tofino (the research stream's count).[391] ONIE, the open installer on white boxes, lets the buyer load any network OS.[560]

The 800G white box costs about 63% as much as the SN5600 (our arithmetic). The two pages leave unclear whether the SN5600 also needs the separate Cumulus license; we assume it does not.

In Thailand, Throughwave sells Edgecore switches from 1G to 800G with SONiC; Photon State distributes, installs, and supports Edgecore with enterprise or community SONiC (its claim); and SWS Group carries Edgecore. None of them publishes prices.[565][566][567] Paid SONiC support exists, also without public prices: Dell's Enterprise SONiC bundles, Broadcom's enterprise SONiC with one to seven years of support from STORDIS, and Aviz's 24x7 support priced by device size.[568][569][570] Hedgehog runs SONiC through Kubernetes with cloud-style tenant networks and can be installed air-gapped.[571]

The GPU fabric

InfiniBand is in effect a single-vendor technology, about 99% NVIDIA by one secondary source.[572] Dell'Oro reported in March 2026 that Ethernet took more than two thirds of switch sales for AI clusters in 2025, mostly at 800G.[511] Version 1.0 of the Ultra Ethernet specification came out on June 11, 2025, and version 1.0.3, of July 16, 2026, is current.[573]

SONiC already runs large AI fabrics. Microsoft's Fairwater uses it for up to 512,000 GPUs;[574] SAKURA internet's SAKURAONE (800 H100s on 800G Ethernet) ranks 49th on the TOP500 and is the only top-100 system with a fully open network stack;[575] and Alibaba uses it in its HPN7.0 AI cluster.[576] NVIDIA sells and supports SONiC on its Spectrum switches and says Spectrum-X supports it, and Dell ships Spectrum-X switches with its SONiC.[577][578][579] Adaptive routing on Spectrum-X needs BlueField-3 cards in the servers (secondary source).[580]

Tenant networks

RFC 8365 defines EVPN with VXLAN as the overlay for a multi-tenant data center.[581] SONiC routes EVPN-VXLAN per VRF (a separate routing table for each tenant), and FRR does the same on Linux hosts.[582][583] Our design gives each customer one VRF from the leaf switch to the hypervisor, with the same FRR everywhere.

A sketch for phase 1 and for the hall

Switch bill of materials (our estimate). The estimate assumes 64 GPU servers of 8 GPUs per MW (about 15 kW each, as in the summary) and 400G per GPU. Each 64x800G switch runs as 128x400G, and each leaf has 64 ports down and 32x800G up, so the fabric is non-blocking. Each GPU server has two 100G front-end ports on AS7726-32X leaves (24 down, 8 up). There are six 25G top-of-rack switches per MW for storage and hosting, and one OPNsense pair. The prices come from the table above.

The arithmetic for this table and for the storage example below is in model/open_infra_costs.py in the repository. The GPU fabric makes up 80% of the bill, and the whole bill comes to about 45,000 THB per GPU with white boxes and 66,000 THB with SN5600s. That is 2 to 3% of the 1.5 to 2.3 million THB per GPU of an 8x B200 server (our arithmetic; server prices in the summary). White boxes save about 11 million THB per MW at list prices. The bill leaves out optics and cables (unpriced in our evidence, and large at 800G), management switches, support, router servers, VAT, and duty.

Firewalls, routing, and DDoS

OPNsense fits phase 1, where CMU's exchange port is 10 Gbit (option A), but it does not fit a 100G edge. At 100G the most open option (our judgment) is NixOS routers with nftables and FRR, and that option needs three things:

1. Filtering on the switches. Stateless filters and BGP run at line rate on the border and leaf switches, and stateful firewalls sit with each tenant or service (OPNsense or nftables in the tenant's zone) rather than in one big box.
2. A fast software data plane where routers forward 100G. That means user-space forwarding (VPP/DPDK). Netgate claims about 14 million packets per second per core, so about 10 cores would handle 100 Gbps of the smallest packets (vendor claim).[591]
3. People. We found no host that names NixOS and nftables as its edge, and no Thai provider that names its firewall stack, so the investor's team must own this work.

DDoS. FastNetMon Community (GPLv2) detects an attack in about 2 seconds from a port mirror, 4 from sFlow, or 10 to 30 from NetFlow, and then announces BGP routes that blackhole the target or divert its traffic to scrubbing.[514] Floods larger than the uplinks must be cleaned upstream. True IDC sells DDoS protection from a local scrubbing center, without a public price.[515]

Storage: Ceph for S3, tape for the copy nobody can delete

Ceph

The Ceph Foundation sits under the Linux Foundation, and a leadership team of developers, rather than the funding members, sets the project's direction; most of the code is LGPL.[516][592] Its S3 gateway (RGW) supports lifecycle rules, versions, storage classes, and policies, while bucket replication is partial.[593] In Object Lock compliance mode, no user, not even an RGW administrator, can delete an object or shorten its retention through S3.[517] Someone with administrator access to the cluster itself is not stopped, which is why an offline tape copy is needed (our reading).

Ceph runs at large scale. CERN has more than 100 PB in over 20 clusters at three locations, with a backup S3 cluster at a second site;[597] DigitalOcean runs more than 250 PB raw;[598] and the UK's STFC runs 110 PB raw on 8+3 coding.[599]

Why not MinIO, Garage, or SeaweedFS

MinIO moved to AGPLv3 in 2021.[600] In May 2025 it stripped its community console and moved logins to its paid product; it stopped community Docker images in October 2025 and entered maintenance mode in December 2025.[601][602][603] As of October 27, 2026, the repository is archived and "no longer maintained".[518] Garage (AGPLv3) replicates across sites over the internet but has no S3 versioning yet, and so no Object Lock (our inference).[604][605] SeaweedFS is Apache-2.0 but is run by one vendor that sells an enterprise edition.[606] Of these options, only Ceph is governed by a foundation and has Object Lock (our judgment).

Hardware and drives

The 45Drives Storinator XL60 holds 60 directly wired drives in 4U, up to 1.92 PB raw, and 45Drives builds its clusters on Ceph.[607][608] Its undated price list starts the 30-bay Q30 at USD 2,999 and the XL60 at USD 6,930, without stating CPU, memory, or drives.[609] B4L's own experience is with a 45Drives Storinator holding about 200 TB on ZFS, which it runs for itself. Supermicro's 90-bay 4U SSG-542B-E1CR90 weighs 102 kg, and at Ceph's 6 GiB per disk it would need over 1 TB of RAM, so nodes of 24 to 60 bays are easier (our arithmetic).[610] Backblaze stopped building its open Storage Pods and now buys 60-drive Supermicro servers.[611] Remanufactured OCP storage racks with OpenBMC hold up to 11.5 PB.[612]

Drives are scarce. Seagate launched its 30 TB Exos M at USD 599.99 (about USD 20 per TB) in July 2025; on October 27, 2026, US retail prices were USD 46 to 55 per TB.[613][614] Enterprise lead times reached about two years, and in June 2026 a Thai distributor showed no price or stock for a 24 TB drive.[615][616] A 122.88 TB QLC SSD tripled in price within nine months.[617] Backblaze's figures for the second quarter of 2026 show quarterly failures of 1.73% across 354,415 drives, but 0.55% for a 22 TB WD model and 3.3% for a 24 TB Seagate, so the choice of model matters.[618]

The power figures are our calculation and cover the drives or the library only. Even 435 W per PB costs only about 22,000 THB a year at the study's 3.81 THB/kWh and a PUE of 1.5 (our arithmetic), so drive prices and staff, rather than power, decide the cost of storage.

Tape

LTO-9 holds 18 TB per cartridge and LTO-10 holds 30 TB (a 40 TB LTO-10 cartridge followed), and LTO-10 drives cannot read older generations.[623][624][625] The roadmap was cut in November 2025: LTO-14 now goes to 365 TB native instead of 576 (Blocks & Files' reading).[626] Only Sony and Fujifilm make the cartridges.[627]

US street prices on October 23, 2026, were USD 92 to 129 per LTO-9 cartridge, USD 259 to 300 per 30 TB LTO-10 cartridge, and USD 23,422 to 24,950 per external LTO-10 drive.[622] In Thailand, an LTO-9 cartridge lists at 4,477 THB (VAT basis not stated) and HPE's external LTO-9 drive at 248,000 THB excluding VAT.[628][629] IBM's sample prices are USD 28,000 for a TS4300 with 40 slots and two LTO-9 drives, and USD 54,500 with 80 slots and four drives.[630][631] At tens of petabytes, one IBM Diamondback frame holds 46.4 PB of LTO-10.[632]

The format and the software can be fully open. LTFS, which makes a tape readable as a file system, is ISO/IEC 20919:2021, with a BSD reference implementation; Bareos (AGPLv3) drives libraries and WORM tape; and Ceph's lifecycle rules can move objects to a tier for "S3 Glacier or Tape storage services".[633][634][635] Drive and library firmware stay closed (our reading). Tape needs 15 to 32 °C and 20 to 80% humidity, with slow changes.[636] We recommend LTO-9 for now, because it is cheaper per TB, and a later move to LTO-10 can keep one LTO-9 drive for the old tapes.

A sized example: 1 PB usable plus two tape copies

Filling the same 12 servers to 30 drives each (about 5 PB) brings Ceph to about 62 THB per TB-month, before the extra memory and the larger library. Drives at Seagate's 2025 launch price would save about 27 THB per TB-month. The estimate leaves out staff, floor space, transit, software support, the off-site vault, VAT, and duty.

Our reading is that at 1 PB this cost leaves room for staff, space, and margin under every list price except that of Glacier Deep Archive. Against Hetzner and Backblaze, the service would compete on location and on sitting next to the customer's agents (business hosting) rather than on price.

Backup and a second site

The 3-2-1-1-0 rule asks for three copies on two media, with one off site, one immutable or offline, and zero errors in tested restores.[642] CISA advises offline, encrypted, and tested backups, because ransomware seeks out and deletes or encrypts the backups it can reach.[643] The example above meets the rule (our design). Ceph with Object Lock holds the online copy, a tape in the library is the second medium, an exported tape is the copy kept off site and offline, and Bareos restore tests supply the zero.

BSI recommends geo-redundant data centers about 200 km apart and never under 100 km, with at most one of them on the same river system unless it sits well above historic floods. Beyond about 100 km of fiber, synchronous replication struggles, so the second site is replicated asynchronously.[520] The Bank of Thailand's 2023 rules ask banks for a recovery site that no single power failure or disaster can hit together with the main site and that shares no utilities with it, for backup media kept outside the main center, and for contingency tests at least once a year. A bank on a public cloud abroad must keep backups in Thailand or a third country.[428] Those rules make a market for a Thai backup service (our reading).

All of this points to two sites (our judgment). CMU ITSC in Chiang Mai and the investor's own data center in the Korat/Chaiyaphum belt would be 460 to 520 km apart in a straight line (our calculation from the coordinates in site selection). They are unlikely to share one flood, quake, or grid fault, and they suit asynchronous S3 replication and a tape vault. BOI's cloud promotion also needs two ISO 27001 data centers in Thailand linked at 10 Gbps or more;[250] ITSC's ban on private circuits is the open point (see business hosting).

Monitoring and inventory software

What the AGPL means here. The AGPL requires anyone who runs a modified version on a network server to offer that version's source code to the server's users.[550][656] Our reading, which is not legal advice, is that running Grafana or Loki unmodified for the investor's own operations creates no duty, but that customer dashboards on a modified Grafana would oblige the investor to offer those changes to those customers. A lawyer should look at the question before a product is built on AGPL code.

The IT equipment reports its telemetry through four kinds of source:

• GPUs. NVIDIA's DCGM and its Prometheus exporter are Apache-2.0. DCGM caps power per GPU group and flags thermal excursions and driver errors, such as a GPU dropping off the bus.[658][659] The driver underneath is closed (our reading).
• Servers and cooling units. OpenBMC implements Redfish's telemetry model, and Redfish also models coolant distribution units; older BMCs are read with the MIT-licensed IPMI exporter.[660][661][662]
• Network. SONiC streams counters over gNMI, and the Apache-2.0 SNMP exporter covers UPS units and PDUs.[663][664]
• Storage. Ceph's manager serves Prometheus metrics, and the smartctl exporter reads drive health.[665][666]

Sensors, outside data feeds, and the generator are covered in the facility systems note. B4L's own experience here is Prometheus-style monitoring of its fleet and a public river-monitoring service, water.buildfor.life, that ingests live Thai sensor feeds into its own dashboards and alerts.

What is still closed, and how to live with it

• NVIDIA's CUDA, user-space driver, and GPU firmware are closed; only the kernel modules are open.[553] NVIDIA AI Enterprise (USD 4,500 per GPU a year) is optional for an open stack.[464]
• Switch chips and network cards are closed. SONiC reaches the chip through SAI, the Switch Abstraction Interface specified at the OCP.[558] We found no source on the chip makers' software licenses, so we treat that software, like network-card firmware, as closed (our assumption).
• Drive, tape-drive, and library firmware is closed in every product we saw (our reading), and tape media come from only two makers.
• BMCs and boot firmware are mostly closed: in our evidence only IBM ships OpenBMC, and openSIL is a proof of concept.

We would live with these closed parts in four ways (our judgment):

1. Use the open parts that exist: NVIDIA's open kernel modules, OpenBMC where it is offered, and SONiC in place of a closed network OS.
2. Keep a second source: AMD for GPUs, two drive makers, and two switch vendors that both run SONiC.
3. Write it into contracts: Redfish on every server, ONIE on every switch, and firmware security updates for the planned life.
4. Keep data in open formats: S3 for objects, LTFS on tape, and Prometheus and Redfish for telemetry, so that any closed part can be swapped without moving the data.

What this means for the investor

What follows is our judgment.

Open software works everywhere except in the GPU software and the firmware. That closed core is the same for every competitor, and everywhere else open software removes per-switch, per-socket, and per-seat licenses.

Phase 1 at CMU ITSC needs little: two 25G switches and two OPNsense appliances for about 0.8 million THB, with NixOS, Incus, Prometheus, and Grafana. One ZFS storage server with an LTO-9 drive is enough there. Ceph starts in the investor's own hall, where 8+3 coding needs at least 12 servers.

The hall's GPU fabric should be Ethernet with SONiC. White boxes save about 11 million THB per MW against NVIDIA's switches at list prices. The investor should buy a SONiC support contract, and should get the optics quoted before deciding.

Storage is a sound business line. It costs about 144 THB per TB-month in hardware and power at 1 PB with two tape copies, against 770 THB at NIPA and 743 THB at AWS Bangkok, although staff decide the real cost.

The two sites should be far apart. ITSC plus a hall in the Korat/Chaiyaphum belt meets BSI's distance and the Bank of Thailand's separation, and it is what BOI's cloud promotion asks for.

Open software shifts money from licenses to people. The investor needs engineers for NixOS, SONiC, FRR, and Ceph, or support contracts that cover them. Part of the saving can go back to the projects themselves; the note on funding open-source projects proposes a rule, the recipients and the Thai tax treatment.

Open questions