nix: CA bundle for tests in the sandbox; README runbook
This commit is contained in:
grabowski committed
1 parent
be16176b6e
commit
cbd85c1e30
2 files changed
+208
No files matched your search
@@ -0,0 +1,206 @@
|
||||
# mailmig
|
||||
|
||||
Moves mail from a **Stalwart** server to **Simple NixOS Mailserver** (SNM) in
|
||||
three stages, in this order:
|
||||
|
||||
1. **Users** – read accounts, aliases, quotas and mailing lists from Stalwart's
|
||||
management API and render them for SNM (Nix data + password hashes), so
|
||||
every mailbox exists on the new server before any mail moves. Password
|
||||
hashes are carried over where Dovecot can verify them: users keep their
|
||||
passwords.
|
||||
2. **Mail** – copy every folder over IMAP (master-user login on both sides, no
|
||||
user passwords needed). Re-runnable: each run copies only what is missing,
|
||||
so you can bulk-copy early and run delta passes after the switch.
|
||||
3. **DNS** – compute MX/SPF/DKIM/DMARC/SRV per domain, diff against what is
|
||||
published (Cloudflare API), apply with a rollback journal, verify via DoH.
|
||||
|
||||
Single static binary, Rust, no state besides the files listed below.
|
||||
|
||||
## Build
|
||||
|
||||
```sh
|
||||
cargo build --release # target/release/mailmig
|
||||
nix build # ./result/bin/mailmig
|
||||
nix flake check -L # unit tests + VM end-to-end test (needs KVM)
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
| file | written by | contents |
|
||||
|---|---|---|
|
||||
| `mailmig.toml` | you (`mailmig init`) | servers, logins, folder/DNS settings |
|
||||
| `plan.toml` | `inventory`, then you | accounts, aliases, forwards, notes – **review and edit** |
|
||||
| `secrets.json` | `inventory` | password hashes (mode 0600; never commit) |
|
||||
| `out/` | `users` | `mail-accounts.nix`, `hashes/`, `clan-vars-set.sh`, `dovecot-master-user.nix` |
|
||||
| `reports/*.json` | `sync`, `verify` | per-account, per-folder counts and errors |
|
||||
| `dns-journal/*.json` | `dns apply` | records before the change + what was applied (for rollback) |
|
||||
|
||||
All of them are git-ignored.
|
||||
|
||||
## Runbook
|
||||
|
||||
### 0. Prepare both servers
|
||||
|
||||
**Stalwart (source)** – an account that may open any mailbox:
|
||||
|
||||
* 0.15: master user in the config, login `"<account>%<master>"`:
|
||||
```toml
|
||||
[authentication.master]
|
||||
user = "mailmig"
|
||||
secret = "<long random secret>"
|
||||
```
|
||||
* 0.16+: give a normal account the `impersonate` permission (via a role); the
|
||||
password must be its main password, not an app password.
|
||||
|
||||
and admin credentials (or an API key) for the management API.
|
||||
|
||||
**SNM (destination)** – a temporary Dovecot master user. `mailmig users`
|
||||
writes `out/dovecot-master-user.nix`; import it on the mail host while
|
||||
migrating and create the passwd file it points at:
|
||||
|
||||
```sh
|
||||
printf 'mailmig:{CRYPT}%s\n' "$(mkpasswd -m sha-512)" > /var/lib/dovecot-master-users
|
||||
```
|
||||
|
||||
### 1. Inventory → plan
|
||||
|
||||
```sh
|
||||
mailmig init # writes mailmig.toml; edit hosts/logins
|
||||
export MAILMIG_STALWART_ADMIN_PASSWORD=...
|
||||
mailmig inventory --raw raw.json
|
||||
```
|
||||
|
||||
Prints a summary and a list of notes (accounts without addresses, app
|
||||
passwords, TOTP, groups, lists with unknown members, catch-all settings).
|
||||
Edit `plan.toml`: drop admin/service accounts, set `migrate_mail = false`
|
||||
where only the account must exist, adjust quotas. `--from-raw raw.json`
|
||||
rebuilds the plan offline. Stalwart ≤ 0.15 is read over REST
|
||||
(`/api/principal`), ≥ 0.16 over JMAP (`x:Account`, `x:MailingList`); `kind = "auto"`
|
||||
picks one from `/.well-known/jmap`.
|
||||
|
||||
### 2. Users on the new server
|
||||
|
||||
```sh
|
||||
mailmig users # add --generate-missing to mint passwords
|
||||
```
|
||||
|
||||
* `out/mail-accounts.nix` – `{ domains; accounts.<login> = { aliases; quota; }; forwards; }`
|
||||
* `out/hashes/<login>.hash` – Dovecot-ready hash per account
|
||||
* `out/clan-vars-set.sh` – loads those hashes into clan vars
|
||||
|
||||
Passwords that carry over: `$6$`/`$5$` (sha-crypt), `$2b$`/`$2y$` (bcrypt),
|
||||
`$y$` (yescrypt), `$1$`, `{SSHA}`/`{SHA*}`-style schemes, Argon2 (needs Dovecot
|
||||
with libsodium), and plaintext secrets (hashed by mailmig). Others (PBKDF2,
|
||||
scrypt) and accounts without a password are listed as needing a new one.
|
||||
App passwords and TOTP do not exist in SNM.
|
||||
|
||||
**cnx-network-clan** (`modules/mail.nix`): copy `mail-accounts.nix` next to it
|
||||
and merge it into the existing account list, so each migrated address gets
|
||||
its clan vars generator like any other mailbox:
|
||||
|
||||
```nix
|
||||
let
|
||||
migrated = import ./mail-accounts.nix;
|
||||
accounts = [ "postmaster@cnx.email" ] ++ builtins.attrNames migrated.accounts;
|
||||
# ... genName / passwdGenerators / loginAccounts unchanged ...
|
||||
in
|
||||
{
|
||||
mailserver.domains = [ "cnx.email" ] ++ migrated.domains;
|
||||
mailserver.accounts = lib.recursiveUpdate loginAccounts (
|
||||
lib.mapAttrs (_: a: { inherit (a) aliases; quota = a.quota or null; }) migrated.accounts
|
||||
);
|
||||
mailserver.forwards = migrated.forwards;
|
||||
}
|
||||
```
|
||||
|
||||
Then `clan vars generate mx1` (creates the generators), `bash out/clan-vars-set.sh`
|
||||
(replaces each generated hash with the Stalwart one, so the user's old
|
||||
password keeps working), `clan machines update mx1`. Accounts without a
|
||||
carried hash keep the generated passphrase (`clan vars get mx1 mail-passwd-…/passphrase`).
|
||||
|
||||
Copy the new DKIM keys for the DNS step: `scp 'mx1:/var/dkim/*.txt' dkim/`.
|
||||
|
||||
### 3. Check logins
|
||||
|
||||
```sh
|
||||
export MAILMIG_SRC_MASTER_PASSWORD=... MAILMIG_DST_MASTER_PASSWORD=...
|
||||
mailmig check
|
||||
```
|
||||
|
||||
### 4. Copy mail (old server keeps running)
|
||||
|
||||
```sh
|
||||
mailmig sync --dry-run # what would be copied
|
||||
mailmig sync # bulk copy, 4 accounts in parallel (-j)
|
||||
mailmig sync --sync-flags # later passes before the switch: also copy read/flagged state
|
||||
```
|
||||
|
||||
* Folders: special-use folders map by attribute (Stalwart "Sent Items",
|
||||
"Deleted Items", "Junk Mail" → SNM "Sent", "Trash", "Junk"); everything else
|
||||
keeps its path with the destination's separator (`Projects/2024` →
|
||||
`Projects.2024`; a `.` inside a name becomes `_`). Override with
|
||||
`[folders.map]`, skip with `[folders.exclude]`. Shared/other-users
|
||||
namespaces are skipped.
|
||||
* Each message keeps its flags (minus `\Recent`) and INTERNALDATE.
|
||||
Messages flagged `\Deleted` are not copied (`skip_deleted`).
|
||||
* De-duplication is by Message-ID (or Date/From/Subject digest without one),
|
||||
counted, so legitimate duplicates are preserved. Nothing is ever deleted on
|
||||
either side.
|
||||
* Exit code 1 if any message failed (quota, size limit); details in the report.
|
||||
|
||||
### 5. DNS
|
||||
|
||||
```sh
|
||||
export CLOUDFLARE_API_TOKEN=... # Zone:DNS:Edit on the mail zones
|
||||
mailmig dns plan # diff per domain, warnings
|
||||
mailmig dns apply --yes # journal in dns-journal/
|
||||
mailmig dns verify # public view via DNS-over-HTTPS
|
||||
mailmig dns rollback dns-journal/<domain>-<time>.json --yes
|
||||
mailmig dns zone # zone-file lines for Knot-served zones
|
||||
```
|
||||
|
||||
Per domain mailmig manages exactly: MX, the SPF TXT (`v=spf1…`), `_dmarc`,
|
||||
`<selector>._domainkey` and the RFC 6186 SRV records (`_imaps`,
|
||||
`_submission(s)`). Mail SRV records that point at an old host and that SNM
|
||||
does not offer (`_pop3s`, `_jmap`, `_caldavs`, …) are deleted; everything
|
||||
else is left alone, but records still pointing at the old host (e.g.
|
||||
`autoconfig` CNAMEs) are reported. Per-domain SPF/DMARC/MX overrides:
|
||||
`[dns.overrides."domain"]` (keep `include:` for other senders such as a
|
||||
website or newsletter tool).
|
||||
|
||||
**MTA-STS**: if a domain publishes `_mta-sts` with an `enforce` policy that
|
||||
lists only the old MX, senders that cached it refuse to deliver to the new
|
||||
MX until `max_age` runs out. `dns plan` shows the live policy. Fix that
|
||||
first (policy listing both MX hosts or `mode: testing`, bump the id, wait
|
||||
`max_age`), then switch MX.
|
||||
|
||||
### 6. After the switch
|
||||
|
||||
* Run `mailmig sync` (without `--sync-flags`) for a few days: mail that
|
||||
still reaches the old server through cached MX records is copied over.
|
||||
* `mailmig verify` exits 0 when nothing is missing.
|
||||
* Mail clients: point IMAP/SMTP at the new host (`client_host`), or give the
|
||||
new server a certificate for the old hostname and repoint that name.
|
||||
* Remove `dovecot-master-user.nix` and `/var/lib/dovecot-master-users`, and
|
||||
the Stalwart master user.
|
||||
|
||||
## Not migrated
|
||||
|
||||
Sieve filters and vacation replies, contacts and calendars (SNM has no
|
||||
CardDAV/CalDAV), app passwords, TOTP, shared-folder ACLs, catch-all rules
|
||||
(reported as notes; SNM uses `catchAll` per account).
|
||||
|
||||
## Tests
|
||||
|
||||
`cargo test`: IMAP parser, modified UTF-7, folder mapping, de-duplication,
|
||||
hash classification, Stalwart REST/JMAP shapes, Nix rendering, DNS diff and
|
||||
zone output, Cloudflare API against a mock server (apply + rollback).
|
||||
|
||||
`nix flake check` (`checks.e2e`): three VMs – Stalwart 0.15.5, SNM
|
||||
(nixos-26.05, Dovecot 2.4) importing the golden `tests/expected-mail-accounts.nix`
|
||||
and `tests/expected-dovecot-master-user.nix`, and a client running mailmig
|
||||
through the whole runbook: inventory over REST, hash carry-over (sha512-crypt,
|
||||
bcrypt, plaintext, none → generated), STARTTLS master login on Stalwart,
|
||||
implicit-TLS master login on Dovecot, folder mapping, flags/dates/duplicates/
|
||||
`\Deleted`/3 MB message, idempotent re-run, delta + flag pass, and a zone file
|
||||
from SNM's real DKIM keys.
|
||||
Reference in new issue
Block a user