Rescue and reinstall: a system for the investor's servers

Research note, October 28, 2026, by Alexander Grabowski (B4L). Sources were retrieved on October 28, 2026. Figures are cited, and our own estimates are marked.

This note covers the rescue and reinstall system that the investor's first data center would run for its customers. When a dedicated server or a VPS no longer boots, or its owner is locked out, the customer should be able to start a small Linux system over the network, repair or copy what is on the disks, reinstall, and see the screen in a browser, all without a ticket. The note compares what Hetzner and three other hosts offer, proposes a design from open-source parts on the study's NixOS stack, weighs adopting a provisioning project against writing a thin service, and covers security, the Thai rules, cost and phasing. B4L prepared it as consultant; the investor would build, own and operate the data center and sell these services. Money is in THB, with USD in parentheses at 33 THB per USD, the B4L planning rate, and EUR at 37.821 THB, the ECB rate of October 26, 2026.[494]

In short

• A rescue system is a small operating system that a server boots over the network in place of its own disk. Hetzner arms it for one boot from its Robot panel or API, starts a Debian-based system in memory, and admits the customer with a one-time root password or an SSH key; unused for 60 minutes, the activation lapses.[1226]
• The investor can build the same service from open parts: iPXE with HTTPS and signed images, a NixOS rescue image, nixos-anywhere and disko for installs, and Redfish to set a one-time network boot and power-cycle the server.[1227][1228][1229]
• Customers would see their server's screen in the browser through a proxy, while the management controllers (BMCs) stay on a closed network. BMC firmware has a long record of flaws, and one in AMI's MegaRAC was confirmed as exploited in June 2025.[1230]
• VPS rescue needs little new machinery, because Incus already gives serial and graphical consoles and boots a VM from an ISO volume.[1231][1232]
• Between customers every drive would be sanitized with its own erase command, under a program that follows NIST SP 800-88 Revision 2 of September 2025.[1233]
• We recommend a thin control service on NixOS over adopting Tinkerbell or Ironic, with Ironic standalone kept in reserve for a multi-vendor fleet (our judgment). It takes 12 to 18 person-months of platform engineering, 0.96 to 2.7 million THB, and two small servers at 420,326 to 458,308 THB (est.).

What Hetzner offers

Hetzner's rescue system is a Debian-based live Linux that a dedicated server boots over the network with PXE and runs in memory, without touching the drives. The customer arms it in Robot, Hetzner's panel for dedicated servers, which shows a root password or lets the customer pick a stored SSH key; the login is as root on port 22 or 222. Arming only schedules the next boot, so the customer reboots the server or uses Robot's reset tab. The activation holds for one boot and lapses if the server is not rebooted within 60 minutes.[1226] Robot's web service offers the same call, with SSH key fingerprints as an option.[1234] Hetzner Cloud works the same way for virtual servers: "Enable rescue" arms a network boot for 60 minutes, "Enable rescue & power cycle" restarts the server into it at once, and rescue ends with the next restart.[1235] The API action, enable_rescue, returns the root password.[1236]

Reinstalls start from the rescue system. Hetzner's installimage script opens a configuration file in which DRIVE lines name the disks to be wiped, SWRAID and SWRAIDLEVEL set software RAID 0, 1, 5, 6 or 10, and partition lines may include LVM; a file at /autosetup runs the install unattended. An install takes 1 to 5 minutes, from standard images, older unsupported ones or the customer's own.[1237]

For the screen, Hetzner has two answers. Its technicians attach a KVM-over-IP console, a Lantronix Spider or a Raritan Dominion, whose virtual media mounts ISO files over SMB only; the consoles are few, so customers book them ahead.[893] The first three hours are free, each further three-hour session costs EUR 10 (378 THB), and remote hands cost EUR 21 (794 THB) per 15 minutes.[893][855] The vKVM rescue needs no technician: it starts the installed operating system in a KVM virtual machine, with the server's own disks and MAC address, and shows it in a web console, at the cost of half the server's memory.[1238]

Resets come in five kinds, depending on the server: CTRL+ALT+DEL, an automatic hardware reset, a press of the power button (which also switches a powered-off server on), a long press that forces it off, and a manual power cycle in which a technician unplugs the server.[1239][1234] The web service also sends Wake-on-LAN packets, ten an hour.[1234] The NixOS installer in the design below already works from this rescue system; nixos-anywhere names Hetzner's bare metal among its targets.[1228]

Three other hosts

OVHcloud boots every dedicated server through iPXE on every start. iPXE asks an internal boot service which script to run, so the panel's setting decides whether the server boots its disk, the rescue system or the customer's own script, and OVHcloud tells customers never to move PXE from the top of the boot order.[1240] Its rescue menu includes an iPXE shell, and the setting stays until the customer switches it back to disk.[1241] Its IPMI console gives KVM and serial over LAN in the browser.[1242] Scaleway boots Elastic Metal servers into an Ubuntu-based rescue system in memory.[1243] Leaseweb power-cycles into rescue by default, makes a new root password per request and a new SSH host key per session, and calls the customer's URL when the system is up.[1244] It switches the server's power port, falling back to IPMI,[1245] and it overwrites disks that pass a SMART check with zeros before reuse and degausses failed ones.[1246]

The design

The system we propose copies Hetzner's customer-facing behavior: one-shot activation, a one-time password or SSH key, a reset menu and a reinstall script. It replaces the console cart with a console in the browser. For dedicated servers it has five parts: a provisioning network, a rescue image, an installer, a small control plane and a console proxy. VPS use the same control plane but rescue through Incus.

Provisioning network and iPXE

Each dedicated server already sits on its own segment, because the first data center gives every tenant its own routing table (a VRF) at the leaf. The leaf relays DHCP from that segment to two provisioning servers. DHCP snooping, which Dell's SONiC supports, treats customer ports as untrusted and drops any DHCP server reply arriving on one, so no server can answer another's boot request.[1250] The provisioning servers answer only MAC addresses listed in NetBox, and only while a boot is armed (our design).

The network boot itself is set through Redfish, the standard management API of server BMCs. Its boot override can be set to Once: on the next boot the system starts once from the chosen target, Pxe or UefiHttp, and the override falls back to Disabled.[1229] That is Hetzner's one-shot rule enforced by the hardware, and it leaves the customer's boot order alone. OVHcloud's model, PXE first on every boot, puts the provisioning service in the path of every restart, so we keep it only as a fallback for BMCs without a reliable override.

The investor would build its own iPXE, mostly GPL version 2 or later, with HTTPS and only the investor's private root certificate trusted.[1251][1227] Its embedded script requires trusted images and verifies every kernel and initrd before running it; iPXE allows untrusted images by default, so the requirement has to be built into the binary.[1252] Each server fetches its own boot script over HTTPS with a token that works once.

Secure Boot needs a decision. Microsoft signed iPXE's shim in November 2025, so a signed iPXE can start with only the standard Microsoft keys,[1253] but a NixOS kernel carries no Microsoft signature, and the community NixOS installer images ask for Secure Boot to be disabled.[1254] We would deliver servers with Secure Boot off. For customers who turn it on, the control plane switches it off through Redfish for the rescue boot and back on afterward, as Ironic's Redfish driver can, and says so to the customer.[1255] A signed chain of the investor's own, with lanzaboote for NixOS, can follow.[1256]

The rescue image

NixOS builds a netboot image from its installer's netboot module: an iPXE script, a kernel and an initrd.[1257][1254] The rescue image would be one more machine in the investor's clan repository, built reproducibly from the same pinned packages as the fleet, signed for iPXE, and also produced as an ISO for virtual media and VPS rescue (our design).

Each activation gets a random root password, shown once in the portal, plus the SSH keys in the customer's account. The image makes fresh host keys at boot and reports their fingerprints, which the portal shows before the first login; Hetzner and OVHcloud instead tell customers to edit their known_hosts files.[1235][1241]

Reinstalls

NixOS installs run through nixos-anywhere, which connects over SSH, recognizes a running NixOS installer, partitions the disks with disko and installs the system in one unattended command.[1228] disko declares the layout in code, from GPT to LVM, mdadm, LUKS and ZFS.[1258] The portal would offer a few layouts and accept a customer's own flake. Other systems use a small installer in the rescue image, driven by a short file of drives, RAID level, partitions, image and keys, much like Hetzner's /autosetup,[1237] writing Debian, Ubuntu, Rocky Linux and AlmaLinux images that the investor tests each month (our design). Windows and other ISOs go through Redfish virtual media, whose InsertMedia action attaches an image by URI, and the browser console.[1229]

The control plane

The control plane is a small service on the two provisioning servers, deployed with clan, which manages NixOS machines declaratively without a central controller.[538] It is the only thing that talks to BMCs. Its parts, all our design, are these:

1. A REST API for the portal and customers' scripts, with customer sign-in through Keycloak and staff sign-in through Kanidm.[458][459]
2. A state machine per server, kept in PostgreSQL.
3. A Redfish client for power, one-time boot and virtual media, with one set of BMC credentials per server in OpenBao.[460]
4. NetBox as the record of servers, MAC and BMC addresses, switch ports and owners.[644]
5. An HTTPS endpoint for the per-server iPXE scripts.
6. An append-only audit log, kept at least 90 days.
7. Webhooks that report when a rescue or install is ready, like Leaseweb's callback.

Vendor quirks are the main risk. Supermicro covers end-to-end Redfish only under its SFT-DCMS-SINGLE license,[892] and Dell iDRAC firmware before 4.40.10.00 on Intel or 6.00.00.00 on AMD needs a non-standard call to boot from virtual media.[1255] The server purchase should include whatever license enables Redfish virtual media, and the first fleet should come from one vendor.

The state machine

Redfish reset types include On, ForceOff, GracefulShutdown, GracefulRestart, ForceRestart, Nmi, ForceOn and PushPowerButton, which cover Hetzner's software, hardware and power-button resets.[1229] Hetzner's last resort, a technician pulling the cord, maps onto the switched outlets of the rack PDUs, which is how Leaseweb power-cycles.[1245]

A browser console for dedicated servers

When a customer opens the console, the control plane checks ownership, logs the session and starts a console worker on the provisioning servers, which alone have routes into the management network. The worker opens the BMC's HTML5 console through Redfish and relays it to the browser with noVNC, a JavaScript VNC client licensed mainly under the MPL 2.0, over a websocket protected by a short-lived token.[1259] Ironic's graphical console already works this way: a container per session drives the BMC's HTML5 console through Redfish, and ironic-novncproxy checks a time-limited token. Ironic warns that the VNC server in that container is unauthenticated and unencrypted, so it must be reachable only from the proxy's network; the same rule applies here.[1260]

OpenBMC servers make this simpler, because bmcweb offers the KVM as VNC over a websocket and the host's serial console as a websocket, with mutual TLS.[1261] On older BMCs the serial console runs over IPMI serial over LAN, which needs the RMCP+ (lanplus) interface.[1262] The console page also carries the reset buttons and an "attach ISO" field for virtual media. No BMC ever gets an address that a customer or the internet can reach; staff use the admin VPN, and customers reach only the proxy.

Rescue for VPS through Incus

Incus already has the parts. Its console attaches from the first moment of boot, and a VM's graphical console works even when the guest agent is not running.[1231] The API takes a console type of "console" or "vga", so the portal can relay either to the browser over the TLS-encrypted REST API.[1263][884] For a rescue, the control plane attaches the rescue ISO as a storage volume with a boot priority, so the VM starts from it with the customer's disk attached, and afterward detaches it.[1232] Before any rescue or reinstall it takes a snapshot with a short expiry. Snapshots sit in the instance's own storage pool, so they undo mistakes but are no backup.[1264] Project limits keep the extra volumes within each customer's quota.[882]

Tests on arrival, burn-in and erasure between customers

Every new server would pass an intake run before it is sold: an inventory, a check of firmware against the investor's baseline, Memtest86+, and stress on CPU, memory, disks and network.[1265] Ironic's burn-in steps show the pattern, with stress-ng for CPU and memory, fio for disks and network, and gpu-burn for GPUs, and CPU and memory runs of 24 hours by default, to bring out early failures before production.[1266] Results go into NetBox.

When a customer leaves, the server is cleaned in four steps (our design):

1. The BMC and BIOS settings are reset to the baseline, and their firmware is checked against known hashes or flashed again. On some ASPEED BMCs the host can write the BMC's memory ("pantsdown", CVE-2019-6260), so a departing tenant could have left something behind.[1267]
2. NVMe drives get the Sanitize command, with crypto erase or block erase.[1268] SATA drives get ATA Secure Erase on a plain SATA port, because RAID and SAS controllers and USB bridges can break the long command.[1269] Drives without a working secure erase are overwritten, and a drive that fails any step is destroyed. Ironic's erase step also prefers hardware erase, falls back to overwriting, which can take hours or days, and fails the cleaning rather than leave data.[1270]
3. Each drive's serial number, method, result and time are recorded. NIST SP 800-88 Revision 2, final since September 26, 2025, asks for an enterprise sanitization program and refers to IEEE 2883 for the techniques, apart from cryptographic erase.[1233][1271]
4. A short burn-in confirms that the server works before it returns to stock.

Build or adopt

Smaller tools fill single roles: Pixiecore packs network booting into one binary beside an existing DHCP server, Matchbox matches machines to boot profiles, and netboot.xyz is a menu of installers in iPXE.[1278][1279][1280] They help in tests and a pilot rack, but none holds customer state.

We recommend the thin service, for three reasons. Every option still needs the customer-facing half written for this data center: ownership, identity, the Thai logs, the portal, billing and Incus. The hardware half at opening is small: one vendor, tens of servers and a handful of Redfish calls. And the service fits a stack of NixOS, clan, NetBox, OpenBao and Keycloak without bringing Kubernetes. Its hardware layer should be a driver interface, so that Ironic standalone can slot in as a second driver once the fleet mixes vendors or passes about 200 servers (our trigger); Ironic's lack of multi-tenancy would not matter, because only the service talks to it. We would not adopt Tinkerbell, because it brings Kubernetes and its code has only just been merged into one repository.

Security

Management controllers

BMCs have been a weak point for more than a decade. CISA warned in 2013 that IPMI gives an attacker what amounts to physical access, enough to reboot the server, install a new operating system or take its data, and advised keeping it on a management VLAN and disabling "cipher 0", which many devices enable by default and which skips authentication.[1281] In 2019 "pantsdown" showed that the host could write into the memory of common ASPEED BMCs.[1267] In June 2025 CISA confirmed attacks on CVE-2024-54085, an authentication bypass in AMI's MegaRAC firmware, used by HPE, Asus and ASRock, that lets a remote attacker take over and even brick a server; Eclypsium had found more than 1,000 exposed servers in March.[1230]

The design answers with five rules. BMCs live only on the 1G management network that the first data center already plans. Only the provisioning servers and the staff VPN reach them. IPMI over the network is off wherever Redfish does the job. Every BMC has its own credentials in OpenBao, changed after each customer. BMC firmware is patched on a schedule and its logs are collected.

The boot path and rescue sessions

A rogue DHCP server could hand a booting server a hostile image; per-customer segments and DHCP snooping stop it. A compromised web server could serve a forged image; iPXE refuses anything not signed with the investor's key, and the key stays in OpenBao.[1252] A stolen boot script could be replayed; each token works once. Each activation has its own password, the rescue image accepts SSH only, and it powers off after a set time without a login (our design). A server blocked for abuse can still boot the rescue system, with its network limited to the console, so the owner can clean it without reaching the internet. Staff never log in to a customer's rescue session.

Thai rules

A host that rents out servers keeps traffic data for at least 90 days, and up to two years on an official order, under section 26 of the Computer Crime Act.[837] The MDES notification of August 13, 2021, covers providers that rent out computer systems, requires them to prove each user's identity, and keeps them responsible even when someone else stores the logs.[838][898] The server packages note already plans 90-day retention of traffic data and address assignments.

The rescue system adds records of its own, which the control plane writes to the audit log (our design): who armed a rescue, reinstall or console session, with the verified account, source address and time; every power action and boot override; every boot script fetched; every SSH login to a rescue system, with source address and key fingerprint; and every console session and attached image. The rescue system uses the server's own addresses, so its traffic appears in the server's flow records. The log is append-only and kept at least 90 days, longer on an official order, and the data that identifies a customer stays until 90 days after the service ends.

What it costs

The provisioning servers can be small because they only serve boot files, keep logs and relay consoles. Part of the engineering overlaps the 1 to 2 million THB that the server packages note budgets for the customer platform, console and rebuild included; on our reading that budget is low once the rescue engine is counted, so the platform should carry the higher figure here.

Phasing

How it ties into the packages

The server packages note includes a web KVM console with every dedicated server, reached through the BMC behind a VPN. The proxy keeps the BMC closed while sparing customers the VPN; staff keep theirs. The console stays included without an hour limit, where Hetzner charges EUR 10 (378 THB) for each three hours after the first three.[855] Rescue and reinstalls are free and unlimited on every package. Remote hands stay at 600 THB per 15 minutes. On the VPS lines the console, the rescue ISO and the snapshot taken before a rescue are included, while snapshots a customer keeps cost 1.00 THB per GB a month, as in the packages. The intake burn-in and the erasure between customers are part of what the dedicated-server setup fee and the 12-month term recover.

What this means for the investor

What follows is our judgment.

A rescue system is part of the product. Customers who rent whole servers expect to recover them at three in the morning without calling anyone, and Hetzner, OVHcloud, Scaleway and Leaseweb all offer it. Opening hall B without one would leave the dedicated servers short of the market.

The system is small. Two entry servers and one to one and a half person-years of platform engineering build it from iPXE, NixOS, nixos-anywhere, disko, noVNC and the Redfish standard. The investor should write the thin control service and keep Ironic in reserve for a mixed fleet.

The browser console is where the investor can do better than Hetzner, whose consoles are booked ahead, attached by a technician and charged after three hours. That depends on BMCs bought with full Redfish and kept on a closed network, so the purchase specification for servers should name both.

Erasure between customers is a promise that customers and auditors will ask about. A program that follows NIST SP 800-88 Revision 2, with a record for every drive, answers the question before it is asked.

Open questions