nix: CA bundle for tests in the sandbox; README runbook

README.md

@@ -0,0 +1,206 @@
+# mailmig
+
+Moves mail from a **Stalwart** server to **Simple NixOS Mailserver** (SNM) in
+three stages, in this order:
+
+1. **Users** – read accounts, aliases, quotas and mailing lists from Stalwart's
+   management API and render them for SNM (Nix data + password hashes), so
+   every mailbox exists on the new server before any mail moves. Password
+   hashes are carried over where Dovecot can verify them: users keep their
+   passwords.
+2. **Mail** – copy every folder over IMAP (master-user login on both sides, no
+   user passwords needed). Re-runnable: each run copies only what is missing,
+   so you can bulk-copy early and run delta passes after the switch.
+3. **DNS** – compute MX/SPF/DKIM/DMARC/SRV per domain, diff against what is
+   published (Cloudflare API), apply with a rollback journal, verify via DoH.
+
+Single static binary, Rust, no state besides the files listed below.
+
+## Build
+
+```sh
+cargo build --release            # target/release/mailmig
+nix build                        # ./result/bin/mailmig
+nix flake check -L               # unit tests + VM end-to-end test (needs KVM)
+```
+
+## Files
+
+| file | written by | contents |
+|---|---|---|
+| `mailmig.toml` | you (`mailmig init`) | servers, logins, folder/DNS settings |
+| `plan.toml` | `inventory`, then you | accounts, aliases, forwards, notes – **review and edit** |
+| `secrets.json` | `inventory` | password hashes (mode 0600; never commit) |
+| `out/` | `users` | `mail-accounts.nix`, `hashes/`, `clan-vars-set.sh`, `dovecot-master-user.nix` |
+| `reports/*.json` | `sync`, `verify` | per-account, per-folder counts and errors |
+| `dns-journal/*.json` | `dns apply` | records before the change + what was applied (for rollback) |
+
+All of them are git-ignored.
+
+## Runbook
+
+### 0. Prepare both servers
+
+**Stalwart (source)** – an account that may open any mailbox:
+
+* 0.15: master user in the config, login `"<account>%<master>"`:
+  ```toml
+  [authentication.master]
+  user = "mailmig"
+  secret = "<long random secret>"
+  ```
+* 0.16+: give a normal account the `impersonate` permission (via a role); the
+  password must be its main password, not an app password.
+
+and admin credentials (or an API key) for the management API.
+
+**SNM (destination)** – a temporary Dovecot master user. `mailmig users`
+writes `out/dovecot-master-user.nix`; import it on the mail host while
+migrating and create the passwd file it points at:
+
+```sh
+printf 'mailmig:{CRYPT}%s\n' "$(mkpasswd -m sha-512)" > /var/lib/dovecot-master-users
+```
+
+### 1. Inventory → plan
+
+```sh
+mailmig init                     # writes mailmig.toml; edit hosts/logins
+export MAILMIG_STALWART_ADMIN_PASSWORD=...
+mailmig inventory --raw raw.json
+```
+
+Prints a summary and a list of notes (accounts without addresses, app
+passwords, TOTP, groups, lists with unknown members, catch-all settings).
+Edit `plan.toml`: drop admin/service accounts, set `migrate_mail = false`
+where only the account must exist, adjust quotas. `--from-raw raw.json`
+rebuilds the plan offline. Stalwart ≤ 0.15 is read over REST
+(`/api/principal`), ≥ 0.16 over JMAP (`x:Account`, `x:MailingList`); `kind = "auto"`
+picks one from `/.well-known/jmap`.
+
+### 2. Users on the new server
+
+```sh
+mailmig users                    # add --generate-missing to mint passwords
+```
+
+* `out/mail-accounts.nix` – `{ domains; accounts.<login> = { aliases; quota; }; forwards; }`
+* `out/hashes/<login>.hash` – Dovecot-ready hash per account
+* `out/clan-vars-set.sh` – loads those hashes into clan vars
+
+Passwords that carry over: `$6$`/`$5$` (sha-crypt), `$2b$`/`$2y$` (bcrypt),
+`$y$` (yescrypt), `$1$`, `{SSHA}`/`{SHA*}`-style schemes, Argon2 (needs Dovecot
+with libsodium), and plaintext secrets (hashed by mailmig). Others (PBKDF2,
+scrypt) and accounts without a password are listed as needing a new one.
+App passwords and TOTP do not exist in SNM.
+
+**cnx-network-clan** (`modules/mail.nix`): copy `mail-accounts.nix` next to it
+and merge it into the existing account list, so each migrated address gets
+its clan vars generator like any other mailbox:
+
+```nix
+let
+  migrated = import ./mail-accounts.nix;
+  accounts = [ "postmaster@cnx.email" ] ++ builtins.attrNames migrated.accounts;
+  # ... genName / passwdGenerators / loginAccounts unchanged ...
+in
+{
+  mailserver.domains = [ "cnx.email" ] ++ migrated.domains;
+  mailserver.accounts = lib.recursiveUpdate loginAccounts (
+    lib.mapAttrs (_: a: { inherit (a) aliases; quota = a.quota or null; }) migrated.accounts
+  );
+  mailserver.forwards = migrated.forwards;
+}
+```
+
+Then `clan vars generate mx1` (creates the generators), `bash out/clan-vars-set.sh`
+(replaces each generated hash with the Stalwart one, so the user's old
+password keeps working), `clan machines update mx1`. Accounts without a
+carried hash keep the generated passphrase (`clan vars get mx1 mail-passwd-…/passphrase`).
+
+Copy the new DKIM keys for the DNS step: `scp 'mx1:/var/dkim/*.txt' dkim/`.
+
+### 3. Check logins
+
+```sh
+export MAILMIG_SRC_MASTER_PASSWORD=... MAILMIG_DST_MASTER_PASSWORD=...
+mailmig check
+```
+
+### 4. Copy mail (old server keeps running)
+
+```sh
+mailmig sync --dry-run           # what would be copied
+mailmig sync                     # bulk copy, 4 accounts in parallel (-j)
+mailmig sync --sync-flags        # later passes before the switch: also copy read/flagged state
+```
+
+* Folders: special-use folders map by attribute (Stalwart "Sent Items",
+  "Deleted Items", "Junk Mail" → SNM "Sent", "Trash", "Junk"); everything else
+  keeps its path with the destination's separator (`Projects/2024` →
+  `Projects.2024`; a `.` inside a name becomes `_`). Override with
+  `[folders.map]`, skip with `[folders.exclude]`. Shared/other-users
+  namespaces are skipped.
+* Each message keeps its flags (minus `\Recent`) and INTERNALDATE.
+  Messages flagged `\Deleted` are not copied (`skip_deleted`).
+* De-duplication is by Message-ID (or Date/From/Subject digest without one),
+  counted, so legitimate duplicates are preserved. Nothing is ever deleted on
+  either side.
+* Exit code 1 if any message failed (quota, size limit); details in the report.
+
+### 5. DNS
+
+```sh
+export CLOUDFLARE_API_TOKEN=...  # Zone:DNS:Edit on the mail zones
+mailmig dns plan                 # diff per domain, warnings
+mailmig dns apply --yes          # journal in dns-journal/
+mailmig dns verify               # public view via DNS-over-HTTPS
+mailmig dns rollback dns-journal/<domain>-<time>.json --yes
+mailmig dns zone                 # zone-file lines for Knot-served zones
+```
+
+Per domain mailmig manages exactly: MX, the SPF TXT (`v=spf1…`), `_dmarc`,
+`<selector>._domainkey` and the RFC 6186 SRV records (`_imaps`,
+`_submission(s)`). Mail SRV records that point at an old host and that SNM
+does not offer (`_pop3s`, `_jmap`, `_caldavs`, …) are deleted; everything
+else is left alone, but records still pointing at the old host (e.g.
+`autoconfig` CNAMEs) are reported. Per-domain SPF/DMARC/MX overrides:
+`[dns.overrides."domain"]` (keep `include:` for other senders such as a
+website or newsletter tool).
+
+**MTA-STS**: if a domain publishes `_mta-sts` with an `enforce` policy that
+lists only the old MX, senders that cached it refuse to deliver to the new
+MX until `max_age` runs out. `dns plan` shows the live policy. Fix that
+first (policy listing both MX hosts or `mode: testing`, bump the id, wait
+`max_age`), then switch MX.
+
+### 6. After the switch
+
+* Run `mailmig sync` (without `--sync-flags`) for a few days: mail that
+  still reaches the old server through cached MX records is copied over.
+* `mailmig verify` exits 0 when nothing is missing.
+* Mail clients: point IMAP/SMTP at the new host (`client_host`), or give the
+  new server a certificate for the old hostname and repoint that name.
+* Remove `dovecot-master-user.nix` and `/var/lib/dovecot-master-users`, and
+  the Stalwart master user.
+
+## Not migrated
+
+Sieve filters and vacation replies, contacts and calendars (SNM has no
+CardDAV/CalDAV), app passwords, TOTP, shared-folder ACLs, catch-all rules
+(reported as notes; SNM uses `catchAll` per account).
+
+## Tests
+
+`cargo test`: IMAP parser, modified UTF-7, folder mapping, de-duplication,
+hash classification, Stalwart REST/JMAP shapes, Nix rendering, DNS diff and
+zone output, Cloudflare API against a mock server (apply + rollback).
+
+`nix flake check` (`checks.e2e`): three VMs – Stalwart 0.15.5, SNM
+(nixos-26.05, Dovecot 2.4) importing the golden `tests/expected-mail-accounts.nix`
+and `tests/expected-dovecot-master-user.nix`, and a client running mailmig
+through the whole runbook: inventory over REST, hash carry-over (sha512-crypt,
+bcrypt, plaintext, none → generated), STARTTLS master login on Stalwart,
+implicit-TLS master login on Dovecot, folder mapping, flags/dates/duplicates/
+`\Deleted`/3 MB message, idempotent re-run, delta + flag pass, and a zone file
+from SNM's real DKIM keys.

flake.nix

@@ -31,6 +31,8 @@
           ];
         };
         cargoLock.lockFile = ./Cargo.lock;
+        # The sandbox has no system CA store; the HTTP client needs one to build.
+        SSL_CERT_FILE = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
         meta.mainProgram = "mailmig";
       };
     in