Always-On Infrastructure Recovery Report

Project: Always-On Infrastructure Recovery & VPS Migration — Discovery Pass Date: 2026-09-23 (UTC) Author: Hermes (infrastructure discovery agent), for review Directive: "Always-On Infrastructure Recovery and VPS Migration Directive" (ChatGPT-authored prompt, supplied by Stephen) Scope: READ-ONLY DISCOVERY + PLANNING. No container, volume, key, config, DNS, firewall, port, or production service was created, modified, stopped, deleted, or migrated. Every command run was read-only. Companion file: infrastructure-inventory.json (machine-readable)

Evidence labels used throughout: - Observed — verified live from a system during this pass - Recovered — established from existing configuration/documentation/recovery kits - Inferred — likely, not yet verified - Recommended — proposed change or placement

00. Executive Summary

The headline: the immediate outage is not a LAN-addressing problem, and it is not a NAS-hosting problem. It is a single dead tunnel dependency — and it is recoverable on the VPS today.

Three facts drive everything below:

1. GoldenEye's LAN address changed, and that is mostly a non-issue. (Observed) GoldenEye moved from 192.168.0.184 (Wi-Fi) to 192.168.1.111 (wired enp1s0), gateway 192.168.1.1. But ~35 Cloudflare tunnel ingress rules and every cross-host service reference it resolves through 127.0.0.1 and Tailscale (100.112.235.92), not LAN IPs. The one genuinely broken dependency was LAN-local: golden-eye's media stack pointed at a local copy, never at the NAS.
2. The NAS is not offline — it is disconnected. (Observed) Tailscale reports ngaged-cloud (100.90.167.88, tag:storage) offline since 2026-09-20T23:41:53Z. It is absent from the new LAN (192.168.1.0/24 scan found no DSM 5000/5001, no 54321, no 445), and Synology DDNS ngaged.synology.me now resolves to 64.138.205.61, which accepts no ports at all (not even 22/443) — consistent with a stale DDNS record pointing at a residential IP where the NAS is not currently publishing. The NAS does not need to be "recovered"; it needs to be physically brought back online at the new location.
3. The critical infrastructure that must never depend on the NAS is currently failing for exactly that reason, and it is a small, fixable problem. (Observed) vault.nsystems.live, traefik.nsystems.live, and rustdesk.nsystems.live all return HTTP 530, because their DNS CNAMEs point at Cloudflare tunnel 6f717a56… ("markpress"), which reports down, 0 connections. That tunnel runs on the NAS. Every other nsystems.live hostname that resolves through golden-eye's shared-public tunnel (455ffa8a…, healthy, 4 connections) works fine.

The recovery assets are already on disk on GoldenEye (Observed + Recovered) — a Vaultwarden data snapshot (3 users, 1071 ciphers, rsa_key.pem, db dated 2026-08-21) and a RustDesk server keypair plus peer database, meaning the preservation-critical assets the directive protects are not locked on the offline NAS. This materially changes the risk profile: the NAS going offline did not destroy Vaultwarden or RustDesk identity.

Recommended direction (not executed): stand up Vaultwarden and RustDesk on the VPS as the always-on instances, restoring the recovered snapshot and the preserved keypair so no client reconfiguration is required; keep the NAS as storage + media plane. Critically — the VPS cannot host this safely today: disk is at 81% (9.3 GB free) and it runs a single 1 vCPU with 2.0 GB swap 55% consumed. A VPS capacity step is a prerequisite, not a nice-to-have.

Added 2026-09-23 — a second, larger program is now folded in. Stephen supplied a "Vector Infrastructure Fabric" upgrade plan (§19): replace most of Cloudflare's operational role with a self-hosted edge (Pangolin + NetBird or Headscale + Technitium). It is treated here as user-supplied intent, not verified fact, and every load-bearing claim was checked against vendor docs and the live host. Verdict: strategically sound, but it collides with three things measured on the box today — it needs UDP ports 51820/21820 while ufw is currently inactive; it needs TCP 80/443 which nginx already owns for five vhosts including two production sites; and Pangolin alone wants 2 GB RAM (82% of what's available) plus 12 GB disk against 9.4 GB free. It is therefore blocked behind a VPS resize/rebuild — a conclusion the plan itself reaches independently ("before we touch the VPS"), which is the strongest point of agreement between the two analyses. Details and the full claim-by-claim verification are in §19 and Appendix D.

01. Reachable Devices

(Observed — tailscale status, ping, TCP probes, SSH, this pass)

02. Offline / Unknown Devices

(Observed)

Unknown / unverified: rore-pc-1 hardware and services (no SSH key). NAS current physical state and whether its volumes survived the move.

03. Existing Services Discovered

VPS srv1388017 — 8 containers (Observed)

Also on VPS host: nginx (443/80/8088), postgresql@16 (loopback), academy-web.service (systemd, 127.0.0.1:8870, /opt/academy, least-privilege DB role academy_app), cloudflared-srv1388017-files.service (tunnel d0a6e41c, healthy), haproxy, fail2ban, tailscaled. No Traefik.

GoldenEye — 34 running / 40 total (Observed) — three WordPress dev stacks (vidal-studio, vyg-dev, rorecclesia, wp-plugin-dev: 8020/8021/8022 + MariaDB), ~12 game/static sites (loopback 8800–8897), obsidian-remote (8093), searxng (8888), filebrowser (ts:8080), homarr (ts:7575), faster-whisper (8000), foundry, kairos-landing, 2× cloudflared, ttyd (7681, LAN). No NAS mounts of any kind.

04. Existing Configuration Artifacts Recovered

(Recovered — ~/nas-modernization-recovery/ on GoldenEye. This is the single most valuable find this pass.)

docker-kit/vaultwarden/ — full Vaultwarden data snapshot, 8.9 MB - config/db.sqlite3 (3.44 MB, 2026-08-21 22:55) + -wal/-shm - config/rsa_key.pem — the instance's RSA signing key is present - config/attachments/ (empty), config/sends/, config/icon_cache/ (populated), config/tmp/ - docker-compose.yml + README.md (Traefik routing, DOMAIN=https://vault.nsystems.live, SMTP via Gmail, /alive healthcheck) - Content verified without exposing secrets: users=3, ciphers=1071, organizations=3, devices=67, folders=3. User records created 2024-09-18/19 and 2025-05-14; last updated 2026-08-17.

small-data/projects/vps/homelab/backup/stacks/admin-tools/rustdesk/data/ — RustDesk server identity, preserved - id_ed25519 (88 B, private) + id_ed25519.pub = ToP1MNOgUymtFbZyVuIUkMoZa9GxbGegAhWChKcZnZg= - db_v2.sqlite3 (+wal/shm) containing 6 registered peers - Integrity (for later verification of restored copies): priv sha256 f7d121a0…b50bf, pub sha256 8bc6d446…7ce47 - ⚠️ Key file mode is 644 — the private key is world-readable in the kit. (Observed) Recommend tightening to 600 at restore time.

docker-kit/compose/ — platform stack definitions: traefik (traefik.yml + compose), cloudflared (config + compose, tunnel 6f717a56), ddns, authelia; nsystems/gatus (3 config variants incl. Vaultwarden /alive checks); stacks/. Plus .secrets/ (smtp.env, authelia jwt/session/storage keys, acme.json, cloudflared-tunnel.json) and .secrets-excluded (placeholder — secrets deliberately not collected).

docker-kit/inspect/*.json — 59 full docker inspect dumps of NAS containers, including hbbs.json and hbbr.json (binds, ports, images, restart policy), vaultwarden.json, traefik.json, cloudflared.json, cf-ddns-nsystems.json. This is a complete container-definition record of the NAS.

dsm-kit/ — NGAGED_CLOUD-config-backup.dss (DSM config export), synoinfo.txt, packages.txt, users.txt, groups.txt, shares.txt, tasks.txt, crontab.txt.

Ingress truth on golden-eye: ~/.cloudflared/shared-public.yml (tunnel 455ffa8a, ~35 hostnames, protocol: http2, edge-ip-version: "4" — with a documented comment explaining the IPv6/QUIC failure mode that caused a prior mass-530 outage) plus ~20 single-site tunnel configs.

Prior documentation: ~/storage-compute-architecture-discovery.md (2026-08-27, 516 lines) + storage-compute-inventory.json; ~/agent-lab/workspaces/docker-ops/NETWORKING-AND-EXPOSURE.md + HOST-INVENTORY.md; ~/personal-cloud-project/ (backup-recovery-design, migration-plan, identity-design); ~/wiki/.

05. Services Currently Unavailable

(Observed)

06. VPS Resource Assessment

(Observed, live)

Incremental requirement for Vaultwarden + RustDesk: ~150–250 MB RAM, ~1–2 GB disk (Vaultwarden data 8.9 MB + image; RustDesk relay is ~4 MB RAM). RAM is fine; disk is the constraint. Recommended prerequisite: free ≥5 GB (image prune, log rotation, the 6.73 GB openwebui image) or grow the volume before migrating.

07. Recommended Always-On Services (VPS / control plane)

(Recommended — nothing moved)

08. Services That Should Remain Local (home)

(Recommended)

09. Hybrid Architecture Opportunities

(Recommended) — the directive's own emphasis, and the correct read of this environment:

1. Vaultwarden: VPS instance + NAS backup target. Control plane on VPS for availability; the existing NAS snapshot/backup discipline continues as the durable copy. Best of both.
2. RustDesk: VPS relay + preserved identity. Same keypair → existing devices reconnect with zero client changes (see §10).
3. Monitoring: VPS checks the home; home does not check the VPS. Split so each failure domain is observed from outside itself.
4. NAS as storage plane, not ingress plane. Keep ~100+ GB media/datasets home; move the small, critical, availability-sensitive services out.
5. Tailscale as the invariant management layer. Every cross-host reference should be a MagicDNS name or Tailscale IP, never a LAN IP — this is what made the move survivable.

10. RustDesk Recovery Status

Status: RECOVERY SUCCEEDED — existing identity is preserved. No replacement keypair is needed.

(Recovered + Observed) - Original host: NAS (ngaged-cloud), containers hbbs (ID/rendezvous) + hbbr (relay), both bind-mounted /volume1/docker/stacks/homelab/rustdesk/data:/root, image sha256:f89aca4c…, network rustdesk_default, ports 21115–21119 tcp + 21116 udp bound on all interfaces. (from docker-kit/inspect/hbbs.json / hbbr.json) - Public identity: rustdesk1.vectorhost.net → 31.187.72.46 (the VPS). rustdesk.nsystems.live → tunnel CNAME 6f717a56 (NAS) → 530. (Observed) - Keypair recovered: id_ed25519 + id_ed25519.pub = ToP1MNOgUymtFbZyVuIUkMoZa9GxbGegAhWChKcZnZg=, with db_v2.sqlite3 holding 6 peers. Located at ~/nas-modernization-recovery/small-data/projects/vps/homelab/backup/stacks/admin-tools/rustdesk/data/. - Current state: no RustDesk listener on the VPS or golden-eye; all 21115–21119 probes closed; relay effectively down since the NAS went offline. - Historical note (Recovered): hbbr was observed EXITED once before (since 2026-06-07) while hbbs kept running — the relay has a fragile history independent of this move. - ⚠️ Key hygiene (Observed): private key is mode 644. Tighten to 600 on restore.

Recommended procedure (NOT executed — see §18): recover keys from kit → verify sha256 matches recorded values → deploy hbbs+hbbr on VPS with preserved /root data → restore id_ed25519 (+600) and db_v2.sqlite3 → open 21115–21119 tcp / 21116 udp with firewall scope → repoint rustdesk1.vectorhost.net (already correct) or publish via tunnel → validate: server key on a client equals the recovered pubkey → test an existing client connects with NO reconfiguration → only then retire the NAS copy. Rollback: the NAS copy is untouched until step "retire"; reverting = restart the NAS containers.

11. Vaultwarden Recovery Status

Status: RECOVERED SNAPSHOT IN HAND — a full data snapshot exists on GoldenEye. Accounts and vault contents are preserved.

(Recovered + Observed) - Original host: NAS, bind /volume1/docker/stacks/homelab/vault/config:/data, no host ports exposed (ingress purely via Traefik), DOMAIN=https://vault.nsystems.live, /alive healthcheck. (from inspect/vaultwarden.json + the kit compose) - Snapshot age: db.sqlite3 2026-08-21 22:55 → ~33 days old as of 2026-09-23. (Observed — this is the report's most important caveat, see §14/Security.) - Contents preserved: 3 users, 1071 ciphers, 3 organizations, 67 devices, 3 folders, rsa_key.pem present, icon cache populated, attachments/ empty. - Known-good integrations (Recovered): bw CLI worked against this instance (940 items readable via the hermes-nas container through a loopback TLS shim vault-tls on 127.0.0.1:8443, bypassing Cloudflare Access); SMTP via Gmail app password verified with a delivered test email; 2FA had been removed at the instance level because CF Access enforced it at the edge (user UUID a18567e7-eaea-48f7-baa3-1630e4b03ae1). - Recovery assets on golden-eye (Recovered): encrypted secret vaults + helpers — ~/.secrets.vault.yaml (Vaultwarden master password), ~/.secrets.nas.yaml, ~/.secrets.cf.yaml, ~/.secrets.gmail.yaml; sops/age key ~/.config/sops/age/keys.txt; helpers ~/bin/nas-env; ~/bin/retired-nas-vault/. CF token at ~/.cloudflare/.env. - ⚠️ Correction to a stale assumption (Observed — important, and subtle): the current session's standing note says secrets are obtained via a vault.home broker in a design profile. The broker DOES exist and is live and healthy — ~/.hermes/profiles/design/vault/vault.py (sops/age-encrypted, values never printed unless explicitly requested), and it is the correct secret source for this migration. What is retired is the NAS-specific helper: ~/bin/nas-vault has been moved to ~/bin/retired-nas-vault/. So: - ✅ Use the broker for host-local tasks — python3 ~/.hermes/profiles/design/vault/vault.py run NAME -- <cmd>. - ❌ Do not rely on ~/bin/nas-vault or the NAS Vaultwarden path — the NAS is offline, so anything sourcing secrets from the NAS is dead. - Names-only inventory read this pass (no values): CF_API_TOKEN, CF_DNS_API_TOKEN, VAULTWARDEN_MASTER_PASSWORD, GMAIL_SMTP_PASSWORD, CHURCH_WP_ADMIN_USERNAME/_PASSWORD, DASHBOARD_ADMIN_TOKEN, FACEBOOK_ADMIN_TOKEN, URPOST_FACEBOOK_PAGE_ID/_TOKEN, YOUTUBE_CLIENT_ID/_SECRET/_REFRESH_TOKEN, HOMARR_PASSWORD, PAGE_BUILDER_ACCESS, rorecclesia_wp_admin, wp_plugin_dev_admin. - Note for the migration: the broker's CF_API_TOKEN / CF_DNS_API_TOKEN may carry write scope, unlike ~/.cloudflare/.env (which returned no account and could not even list tunnels). Re-test scope via the broker before assuming DNS changes are blocked — but a DNS write still requires your explicit approval (§18). - Blocking constraint (Recovered, still true): bw refuses plain HTTP, and the HTTPS path was behind Cloudflare Access — so bw/Bitwarden clients cannot sync through the CF-Accessed hostname without a service token. A VPS-hosted instance should either use a Tailscale-only/private TLS endpoint for CLI/API sync and CF Access only for the web vault, or provision a CF service token. - ⚠️ Never print secrets. No secret value was read, printed, or written during this pass; only locations, sizes, modes, and non-secret counts are recorded.

Recommended procedure (NOT executed): verify snapshot integrity → snapshot the snapshot → deploy vaultwarden on VPS with /data restored → restore rsa_key.pem + db.sqlite3 → set DOMAIN + SMTP + ADMIN_TOKEN from secret vault → private validation (local /alive, login, item count == 1071) → external validation via CF Access → client validation (bw sync, web vault) → monitoring → keep NAS copy until proven → retire. Do not deploy a clean Vaultwarden as a substitute.

12. Tailscale Topology

(Observed)

• Self (golden-eye): 100.112.235.92, tag:agent.
• Peers & tags: srv1388017 tag:runtime; NGAGED_CLOUD tag:storage (offline); SADE-LAPTOP tag:control; rore-pc-1, rore-pc-wsl, MSI laptops, Pixel/A52 — untagged (user-owned).
• Direct paths confirmed: golden-eye ↔ srv1388017 (103 ms), ↔ rore-pc-1 (47 ms), ↔ sade-laptop (13 ms). Mobile peer reaches golden-eye directly over IPv6.
• Two peers are - (no path shown): golden-eye and srv1388017 show themselves as - — expected for self plus peers with no active session; both are reachable in practice (verified by SSH and ping).
• ⚠️ VPS reports: "Tailscale can't reach the configured DNS servers. Internet connectivity may be affected." — and golden-eye's Tailscale log shows a gateway/self-IP change (gw=192.168.1.1 self=192.168.1.111) plus Tailscale is hijacking DNS (a previously-documented fix on the VPS). Investigate Tailscale DNS configuration on both hosts — MagicDNS is the intended replacement for LAN IPs, so its resolution path must be healthy.
• ACL consideration (Recovered pitfall): a previously diagnosed failure mode is that ACL rules naming only untagged sources drop every tag:-carrying peer — traffic dies inside the tunnel before the remote OS sees it. With golden-eye tag:agent, the VPS tag:runtime, and the NAS tag:storage, any new VPS-hosted service must have its ACL grant written explicitly. Verify with a real inbound packet/log line, not by the console's save button.
• Addressing correction already achieved: Tailscale IPs/MagicDNS are already the primary cross-host reference in ingress config. This is why the LAN change broke so little.

13. DNS / Network Dependencies Needing Correction

(Observed)

Classification of 192.168.x.x references (Recommended): convert to Tailscale MagicDNS or Tailscale IPs for cross-host admin traffic (SSH, APIs, monitoring); keep LAN addresses for high-bandwidth local-only traffic (NAS ↔ golden-eye media, SMB); use Docker internal DNS for intra-host container-to-container; use public DNS only for genuinely public endpoints. Do not blindly replace LAN addresses.

14. Security Concerns Discovered

(Observed unless noted)

1. 🔴 Stale Vaultwarden snapshot as the recovery source — ~33 days old. Any item added/rotated between 2026-08-21 and the NAS going offline (2026-09-20) is at risk of loss. This is the single most urgent data-integrity concern. If the NAS powers on and is reachable, capture a FRESH snapshot BEFORE migrating anything.
2. 🔴 Cloudflare Access gate is on the critical path but cannot pass traffic. vault.nsystems.live 302→nlivesystems.cloudflareaccess.com; since the origin tunnel is down, authentication cannot help. Also: CF Access + bw CLI incompatibility is a designed dead end for programmatic access.
3. 🟠 Vaultwarden 2FA had been removed at the instance level (Recovered), compensated only by CF Access at the edge. If moved behind a non-CF path (e.g. Tailscale-only), this compensating control disappears — re-enable instance 2FA or document the new compensating control explicitly.
4. 🟠 RustDesk private key file mode 644 in the recovery kit; also its presence in a general-purpose recovery directory. Tighten to 600, and treat the kit directory as secret material.
5. 🟠 NAS administration surface. Golden-eye's standing note (a DSM firewall that drops inbound unless explicitly allow-listed, and no passwordless sudo for tailscale cert) is a good posture — but confirm whether NAS SSH:54321 and DSM 5000/5001 were ever exposed publicly via DDNS. (Inferred: DDNS was used for RustDesk, suggesting port-forwarding at the old location — verify no stale port-forward rules exist on the new router.)
6. 🟠 VPS disk at 81% is an operational-safety issue: a full disk takes down Postgres/nginx and the academy sites, i.e. a new always-on outage created while fixing the old one.
7. 🟡 Golden-eye over-exposes to its LAN. ttyd on 0.0.0.0:7681, WordPress 8020/8021/8022 on all interfaces, faster-whisper 8000, searxng 8888, space-formations 8124. On a shared/new network, scope these to loopback or Tailscale. (Observed: ss -tlnp)
8. 🟡 A .secrets corpus sits inside nas-modernization-recovery/docker-kit/compose/ (smtp.env, authelia jwt/session/storage keys, acme.json) alongside an unencrypted Vaultwarden DB and RSA key. Encrypt the kit or scope its permissions.
9. 🟡 534-restart container (cloudflared-vidal) — an unreviewed, endlessly-retrying process with a missing config; noise that masks real alerts.
10. 🟡 rore-pc-1 reachable on tailnet with no working SSH key — desired (admin access) but currently impossible; a deliberate one-time key install is needed (see the administrators_authorized_keys pitfall for Windows admin accounts).
11. Least-exposure target for new VPS services: Vaultwarden → public but CF-Access-protected for the web vault, plus a Tailscale-only private TLS endpoint for CLI/API sync; RustDesk → public (required by design) with ports scoped to 21115–21119 and firewall rules audited; monitoring/DNS → Tailscale-only; databases → Docker/internal network only. Never expose the Docker socket, Postgres, NAS admin, or management ports publicly.

15. Proposed Target Architecture

(Recommended — refined to what actually exists)

                    Internet
                        │
        ┌───────────────┴────────────────┐
        │   Cloudflare (DNS · Access · Tunnels)
        └───────────────┬────────────────┘
                        │
        ┌───────────────▼─────────────────────────┐
        │  VPS  srv1388017   31.187.72.46         │   ← ALWAYS-ON
        │  100.123.166.61  (tag:runtime)          │      control plane
        │  nginx · postgres · academy-web         │
        │  + Vaultwarden      (small, critical)   │
        │  + RustDesk hbbs/hbbr (public relay)    │
        │  + Monitoring, DDNS                     │
        └───────────────┬─────────────────────────┘
                        │
                  Tailscale tailnet  (tag:agent / runtime / storage / control)
                        │
     ┌──────────────────┼──────────────────────────────┐
     │                  │                              │
┌────▼─────────┐  ┌─────▼──────────────────┐   ┌───────▼──────────┐
│ golden-eye   │  │  NAS  NGAGED_CLOUD     │   │ rore-pc-1        │
│ 192.168.1.111│  │  (tag:storage)         │   │ (church PC)      │
│ tag:agent    │  │  physical — BRING BACK │   │ active, no ssh   │
│ compute/dev  │  │  ONLINE                │   │ key yet          │
│ 34 containers│  │  ── storage plane ──   │   └──────────────────┘
│ Hermes, WP,  │  │  Plex/Jellyfin/*arr,   │
│ games, Kairos│  │  qbit, TubeArchivist,  │    clients: sade-laptop
│              │  │  hermes-nas agent      │    (tag:control), Pixel 9a
│ NO NAS mounts│  │  media/datasets (~100s GB)
└──────────────┘  └─────┬──────────────────┘
                        │
                  Local Storage
              (SHR RAID-1 · USB backup)

Placement rationale (the goal is explicitly NOT "move everything to the VPS"):

16. Proposed Migration Sequence

(Recommended — parallel, non-destructive, rollback at every step. Nothing below was executed.)

Phase 0 — Prerequisites (blocking) 0.1 Bring the NAS physically back online at the new location and confirm it joins the new LAN + Tailscale. This is the highest-value next action: it (a) enables a fresh Vaultwarden snapshot to close the ~33-day gap, (b) lets us verify the RustDesk db_v2 peers, (c) unblocks ~30 tunnel hostnames. 0.2 Capture a fresh Vaultwarden snapshot the moment the NAS is up — before any migration. 0.3 Free ≥5 GB on the VPS (or grow the volume). Verify ufw/firewall state. 0.4 Verify Tailscale DNS/MagicDNS health on golden-eye + VPS. 0.5 Confirm no stale port-forwards on the new router.

Phase 1 — Recover, don't rebuild 1.1 Verify kit integrity (Vaultwarden db + rsa_key.pem; RustDesk keypair sha256 against §10 values). 1.2 Encrypt the recovery kit; tighten RustDesk key to 600; restrict ttyd/WP ports to loopback/Tailscale.

Phase 2 — Vaultwarden to VPS (parallel deploy) 2.1 Deploy Vaultwarden on the VPS (data restored from the kit) binding loopback only, behind nginx. 2.2 Private validation: /alive and login and item count must equal 1071 (or the fresh-snapshot count). 2.3 External validation via CF Access; re-enable instance 2FA or document the compensating control. 2.4 Client validation: web vault + bw sync (Tailscale-only private endpoint for CLI). 2.5 Monitoring (Gatus /alive). Rollback: stop the VPS container; CF CNAME unchanged means the NAS copy remains authoritative.

Phase 3 — RustDesk to VPS (parallel deploy) 3.1 Deploy hbbs/hbbr with the recovered keypair + peer DB; open 21115–21119 tcp / 21116 udp. 3.2 Validate: the server key a client displays equals the recovered pubkey ToP1MNOg…ZnZg=. 3.3 Client compatibility test: an existing device connects with ZERO reconfiguration. This is the acceptance gate. 3.4 Monitoring. Rollback: restore the NAS containers (untouched).

Phase 4 — Ingress convergence 4.1 Repoint vault/traefik/rustdesk hostname CNAMEs from tunnel 6f717a56 to a VPS- or golden-eye-hosted tunnel (or serve via the VPS public IP + nginx). 4.2 Verify HTTP 200 from outside; then DNS cutover; monitor. 4.3 Retire stale tunnels (37 down) and clean up DNS.

Phase 5 — Decommission (last, explicit approval) 5.1 Stop NAS hbbs/hbbr and NAS vaultwarden only after VPS instances have run clean for an agreed soak period, with the NAS copy retained as backup. 5.2 Update documentation; close the LAN-IP reference drift.

Rollback for every production step: the original is never destroyed first; reverting a VPS service = stop the VPS container + leave/park the DNS record; reverting a DNS cutover = restore the prior CNAME (both recorded).

17. Information That Cannot Be Recovered Without Another Machine Online

(Explicit — nothing here was guessed or inferred as fact)

Requires the NAS online: 1. Any Vaultwarden changes between 2026-08-21 and 2026-09-20 — the most consequential gap. 2. The current RustDesk db_v2.sqlite3 and keypair as of the shutdown (kit copy is from a 2025 backup path; whether the live NAS data is byte-identical to the kit copy is UNVERIFIED — must be diffed before trusting the kit). 3. The complete NAS Docker runtime state (33 containers), volumes, and compose at shutdown. 4. Confirmation that the media/datasets (534 GB media, 752 GB vMix, 823 GB homes) survived the move. 5. Backup staleness: whether Hyper Backup to USB is current, and whether a restore was ever tested (backup exists ≠ restore verified). 6. DSM firewall rules, port-forwards, and whether snapshots now exist (the prior audit found effectively zero snapshot protection — only an empty @snapshot/vMix).

Requires rore-pc-1 SSH access: its hardware, services, and whether the church stream pipeline is affected.

Requires the old router or records: whether port-forward rules for the NAS exist/existed.

Unknown (not yet probed): VPS ufw detail; golden-eye's current full docker inspect; whether any host has a NAS mount (all evidence says none); golden-eye ↔ NAS LAN throughput at the new location.

18. Actions Requiring Approval

The following are explicitly NOT authorized by the read-only discovery directive. Each needs Stephen's explicit approval:

1. Any migration/deployment of Vaultwarden or RustDesk to the VPS (Phase 2/3).
2. Any DNS change — repointing vault/traefik/rustdesk CNAMEs (Phase 4). Note: the Cloudflare API token at ~/.cloudflare/.env was used read-only (zones, tunnels, DNS list). It is scoped such that tunnel create returned account= empty and 0 tunnels; do not assume it can perform writes.**
3. Opening firewall ports on the VPS for RustDesk (21115–21119 tcp / 21116 udp).
4. Any Tailscale ACL edit (admin console, outside the CLI).
5. Touching the NAS beyond read-only once it is online — including re-enabling, re-deploying, or retiring its stacks.
6. Freeing VPS disk (image prune / log rotation) — routine but potentially destructive.
7. Re-enabling Vaultwarden instance 2FA (changes the login flow; and note CF Access currently compensates).
8. Retiring anything — NAS hbbs/hbbr, NAS vaultwarden, academy-db container, 37 stale tunnels, exited containers.
9. Relocating/encrypting the recovery kit — changes where Vaultwarden + RustDesk keys live; needs a deliberate, recorded destination.
10. Confirming whether a fresh snapshot is required before migration — if the NAS cannot be brought up, the decision to migrate on a 33-day-old snapshot is Stephen's to make explicitly.

19. User-Supplied Upgrade Plan — "Vector Infrastructure Fabric"

Status: INTENT, NOT OBSERVATION. (Plan — user-supplied) Everything in this section is Stephen's proposal, supplied 2026-09-23. It is not a verified observation about the environment, and it is deliberately fenced off from the Observed / Recovered / Inferred material above so the audit trail stays clean. Where it makes factual claims, those were independently checked and are tagged (Verified) / (Refuted) / (Unverifiable).

19.1 What the plan proposes

Replace most of Cloudflare's operational role with a self-hosted edge stack, organised as three planes:

• Public edge (VPS): Pangolin — a self-hosted tunneled reverse proxy that bundles Traefik (ingress/TLS), Gerbil (WireGuard tunnel manager/relay) and Badger (forward-auth middleware).
• Private network plane: Headscale (open-source Tailscale control server, keeps the existing Tailscale client ecosystem) or NetBird (WireGuard-based, integrated ZTNA, local user management since v0.62).
• Service plane: Vaultwarden, RustDesk, WordPress, Hermes, n8n, monitoring, storage.

Plus a replacement mapping: Cloudflare Tunnel→Pangolin/Newt, Access→Pangolin identity, proxy→Pangolin/Traefik, DNS→Technitium, SSL→Let's Encrypt, Tailscale→Headscale/NetBird, monitoring→Prometheus/Grafana/Gatus, WAF→Coraza/ModSecurity.

Stated governance rule: "Public applications enter through Pangolin. Private applications travel through the mesh. Backend services stay inside Docker networks."

Stated sequencing: make Cloudflare the last dependency removed — CURRENT → TRANSITION (Cloudflare + Pangolin + mesh + VPS control plane, coexisting) → SELF-HOSTED CORE → OPTIONAL EXTERNAL EDGE.

Explicit non-goals (and these are correct): do not try to replace Cloudflare's CDN or its anycast DDoS absorption; keep an upstream provider for volumetric attacks; self-host internal DNS before public authoritative DNS.

19.2 Claim-by-claim verification (Verified against vendor docs + the live host, 2026-09-23)

19.3 Conflicts with the environment as measured — these block implementation

(A) 🔴 Port conflicts on TCP 80/443 — the plan cannot be installed alongside what runs today.

(Observed, live) Pangolin's gerbil container publishes 80:80 and 443:443 on all interfaces, and Traefik must bind them. But on this VPS:

• nginx owns 0.0.0.0:80 and 31.187.72.46:443 ([::]:80 also nginx).
• nginx serves five vhosts: academy-app, academy-report (production, always-on), default, hls, infra-recovery (this report), plus stream (the vhost that owns the specific 31.187.72.46:443 socket).
• Pangolin cannot start until nginx yields 80/443, which means either (a) migrating every vhost into Traefik (and re-issuing certs), or (b) a coordinated cutover window. There is no "coexist quietly" path — the plan's TRANSITION phase as described ("Cloudflare + Pangolin + mesh") is not achievable on 80/443 without resolving this first.

(B) 🔴 Firewall is inactive — and the plan adds two public WireGuard UDP ports.

(Observed) ufw status → Status: inactive. iptables -L INPUT shows policy ACCEPT with only a Tailscale chain. Pangolin requires UDP 51820 (site tunnels) and UDP 21820 (client tunnels) open, and its docs stress: "by tunneling out to the VPS, you are effectively including the VPS in your security boundary." Today every port on this box is open by default. Adding two more public UDP listeners to an inactive firewall is a security regression, not a neutral change. A firewall baseline must precede Pangolin (§14 item 6, §18 item 5 territory).

⚠️ Additional trap: haproxy.service is running (Hostinger image default, ~1.3 KB stock config, active since 2026-09-14) and would contend for the same ingress ports in some configurations. It should be identified and retired deliberately rather than discovered mid-cutover. ModemManager and two snap.cups services are also running pointlessly on a VPS and broaden the attack surface.

(C) 🔴 Capacity — the whole stack does not fit, and Pangolin alone strains the box.

(Observed + Verified requirements)

Pangolin alone = 2048 MB — 82% of available RAM at idle — and it needs 12 GB disk against 9.4 GB free, so it does not fit at all today. The plan's full stack needs ~1.9× the available RAM and ~2.7× the free disk. This is on a 1 vCPU box that has already burnt through 55% of its 2 GB swap at idle.

(Recoverable without touching live sites: 3.37 GB of reclaimable Docker images + 266 MB build cache — the 6.73 GB openwebui image and the 3.15 GB unhealthy camofox image dominate. That buys ~3.6 GB and helps disk; it does not solve RAM.)

(D) 🟠 Subnet overlap between Pangolin's default mesh range and the live tailnet.

(Verified by computation) Pangolin's default subnet_group is written in its docs as 100.89.137.0/20 — which has host bits set and is not a valid network boundary; the real aligned block is 100.89.128.0/20 (100.89.128.0–100.89.143.255). That block sits entirely inside Tailscale's 100.64.0.0/10 CGNAT range, which is where every current device's address lives (golden-eye .112.235.92, NAS .90.167.88, VPS .123.166.61, …). No peer collides at this moment, but the ranges overlap, so Tailscale can allocate into that block on any new device or re-IP. Pangolin's docs say to change this before registering the first Gerbil. Recommended: move it out of 100.64.0.0/10 entirely (e.g. into a 10.x block) if Pangolin and Tailscale are to coexist during the transition — which is the plan's own sequencing.

(E) 🟠 Self-hosting the private-network plane trades a resilient control plane for a single point of failure.

(Verified) Self-hosted NetBird = single relay, DIY HA. Headscale = "a single tailnet run by people comfortable operating it," and does not replicate Tailscale's global DERP relay network. Today the tailnet's control plane and relays are Tailscale's, and the tailnet is precisely what made this move survivable (§01, §13). Moving it onto the same single 1 vCPU VPS that also hosts Pangolin, DNS, monitoring and WAF concentrates the entire control plane into one failure domain — the opposite of the directive's own goal. This is the plan's most significant architectural risk and it is not addressed in the proposal.

19.4 Where the plan is right, and where it needs strengthening

Right — and it agrees with this report's independent findings: 1. Separating the three planes (public edge / private mesh / service) is a genuinely better model than "expose everything through public DNS," and it directly addresses the failure observed here. 2. Cloudflare last, not first. Making it the final dependency removed is the correct risk posture and is exactly why this pass recommends parallel, reversible migration (§16). 3. Not trying to become Cloudflare. Correct on the CDN and volumetric-DDoS points — "no Docker container absorbs 50 Gbps." 4. The move-resilience property is real. Outbound-only tunnels from golden-eye/NAS to a static VPS mean the next move changes nothing important. That is precisely the property that failed this time. 5. "Before we touch the VPS" — the plan independently lands on the same gate as §06/§16 Phase 0. 6. Technitium internal-first. Correct: leave public authoritative DNS at a provider until the rest matures.

Needs strengthening before it can be sequenced: 1. No capacity or resize plan. The stack needs ~1.9× RAM and ~2.7× disk. State the target VPS spec explicitly (Pangolin recommends 2 vCPU / 2 GB / 20 GB minimum; the three-plane stack with monitoring wants 4 vCPU / 4 GB / 40 GB). 2. No port-ownership cutover plan for 80/443. This is the hardest single step and it currently has no design. It needs a window, a migration of five nginx vhosts into Traefik, and cert re-issuance. 3. No firewall baseline. ufw inactive must become a default-deny baseline before two public UDP ports are added. 4. No answer to the control-plane concentration risk (§19.3 E). Coexist with Tailscale long enough to prove the mesh, or run the mesh control plane somewhere other than the VPS that hosts everything else. 5. Pangolin vs NetBird vs Headscale is left open — a decision, not a finding. Recommend: evaluate NetBird on its own merits for the mesh, but keep Tailscale running until the replacement has survived a real incident. The mesh is the one component you least want to be debugging during an outage. 6. Subnet plan missing (§19.3 D). 7. No rollback per step. §16 already models this; the plan does not.

19.5 How this folds into the existing migration sequence

The plan is not a competing sequence — it is a later phase that depends on Phase 0. Ordered correctly:

§16 Phase 0  (prerequisites: NAS online, fresh Vaultwarden snapshot,
              VPS disk freed / RESIZED, firewall baseline, Tailscale DNS)
      │
      ├─► §16 Phase 1–3   (immediate outage fix: Vaultwarden + RustDesk
      │                    recovered onto the VPS, identity preserved)
      │                    ← THIS is what fixes the 530s. Do this first.
      │
      └─► §19 Phase A     (VPS RESIZE / rebuild — the gate for everything below)
                 │
                 ├─► §19 Phase B   Pangolin installed, coexisting, serving
                 │                 NEW hostnames only. nginx keeps production.
                 ├─► §19 Phase C   Port cutover: migrate vhosts into Traefik,
                 │                 re-issue certs, retire nginx. Requires a window.
                 ├─► §19 Phase D   Firewall default-deny + open 51820/21820.
                 ├─► §19 Phase E   Internal DNS (Technitium); public DNS stays CF.
                 ├─► §19 Phase F   Mesh evaluation: NetBird vs Headscale, run in
                 │                 parallel with Tailscale. No cutover yet.
                 └─► §19 Phase G   Monitoring on the VPS (§07 already recommends it).
                                   Cloudflare removal LAST, if ever.

The critical sequencing insight: §16 Phase 1–3 (Vaultwarden + RustDesk recovery) is independent of and much cheaper than the Fabric program, and it is what actually resolves the current outage. Do not gate the outage fix behind a VPS rebuild. Fix the 530s on the existing box; treat the Fabric as the follow-on programme it is.

19.6 Open decisions this plan creates

1. Target VPS spec — resize in place or move provider? (Pangolin + 3-plane stack wants 4 vCPU / 4 GB / 40 GB.)
2. Mesh choice — NetBird (integrated ZTNA, ~1 GB, no IdP) vs Headscale (keeps Tailscale clients) vs stay on Tailscale (best resilience, no sovereignty gain).
3. Do you accept the control-plane concentration risk (§19.3 E), or does the mesh run outside the VPS?
4. 80/443 cutover window — acceptable downtime for nginx→Traefik, including the two production academy sites.
5. Pangolin's subnet_group — must move out of 100.64.0.0/10 while Tailscale coexists.
6. Firewall baseline — approving default-deny on the VPS (needed regardless of Pangolin).
7. Is DNS sovereignty actually wanted? Internal-only is low-risk; public authoritative DNS at Cloudflare is "enormous global redundancy for essentially no operational effort." Deferring that is recommended.
8. Scope confirmation — is any of §19 to be executed now, or is it planning input for the post-resize programme? (Current answer: planning only; nothing in §19 has been executed.)

Appendix A — Method & Read-Only Guarantee

Every action this pass was read-only: tailscale status/ip/netcheck/status --json, ip/ss/lscpu/free/lsblk/df, docker ps -a/version/inspect/logs, systemctl list-units/list-timers, TCP port probes, bounded ping, curl HEAD/GET against public endpoints, dig, getent, ip neigh, TCP-connect LAN scan, read-only SSH ss/hostname/uname/free/df/crontab -l/systemctl cat, grep/find over local docs, sqlite3 count-only queries against recovery-kit databases (no secret values read), and read-only Cloudflare API GETs. No docker rm/stop/restart, no compose up/down, no key generation, no DNS write, no firewall change, no port change, no service stop, no data migration, no Tailscale reset, no OS reinstall, no credential rotation. No secret value appears in this report; only locations, sizes, modes, and non-secret counts.

Appendix B — Corrections to Prior Assumptions

Appendix C — ChatGPT Architecture Handoff (self-contained)

Context: A homelab operator moved house. The NAS (which hosted Vaultwarden, RustDesk relay, the Cloudflare tunnel 6f717a56 serving vault/traefik/rustdesk hostnames, plus the media stack) is physically offline since 2026-09-20 and absent from the new LAN. The VPS and golden-eye are healthy. The operator wants always-on services to stop depending on home hardware.

Nodes: golden-eye = Ubuntu, 2C4T Pentium 5405U, 7.1 GB RAM, 913 GB disk (16% used), Wi-Fi→now wired 192.168.1.111, TS 100.112.235.92 tag:agent, 34 containers, no NAS mounts. VPS srv1388017 = Ubuntu, 1 vCPU / 3.8 GB RAM / 48 GB disk at 81%, swap 55%, TS 100.123.166.61 tag:runtime, 63-day uptime, nginx + Postgres 16 + academy-web (8870), 8 containers, no Traefik. NAS NGAGED_CLOUD = DS220+, Celeron J4025, 17.4 GB RAM, 3.5 TB btrfs (69%), OFFLINE, TS 100.90.167.88 tag:storage, 33 containers when up, /volume1/docker/{stacks,data}, media 534 GB.

Recovered assets (on golden-eye, ~/nas-modernization-recovery/): Vaultwarden db.sqlite3 + rsa_key.pem + icon cache (3 users / 1071 ciphers, dated 2026-08-21); RustDesk id_ed25519+.pub (ToP1MNOg…ZnZg=) + db_v2.sqlite3 (6 peers); 59 docker inspect JSONs; DSM config backup; Traefik/cloudflared/Gatus/Authelia compose + configs. Secret sources that work: the live vault.home broker at ~/.hermes/profiles/design/vault/vault.py (holds CF_API_TOKEN, CF_DNS_API_TOKEN, VAULTWARDEN_MASTER_PASSWORD, …) — the NAS-specific ~/bin/nas-vault is retired, and anything sourced from the NAS is dead. Never print secrets.

Verified failure: vault.nsystems.live + traefik.nsystems.live → CNAME → tunnel 6f717a56 → down/0 conns → HTTP 530. rustdesk.nsystems.live same; rustdesk1.vectorhost.net → 31.187.72.46 with no listener. Working hostnames all resolve through golden-eye's shared-public tunnel 455ffa8a (healthy). 42 tunnels total, 5 healthy.

Decisions requested: 1. Confirm VPS-hosted Vaultwarden + RustDesk with clients otherwise unchanged — and the exact VPS disk remediation first. 2. How to reconcile the ~33-day-old Vaultwarden snapshot: wait for the NAS (and take a fresh one) vs migrate now and accept the gap. 3. Ingress: repoint vault/traefik/rustdesk CNAMEs to a VPS-hosted tunnel, serve Vaultwarden via the VPS public IP + nginx, or both (CF Access for web, Tailscale-only for CLI)? 4. Whether to relocate the ~35 shared-public hostnames off golden-eye (which flaps) to the VPS — and the cost. 5. Backup target for the new always-on services (they are currently in the failure domain they must survive). 6. Tailscale ACL structure for tag:agent / tag:runtime / tag:storage / tag:control, given tagged peers are dropped unless explicitly granted.

Constraints: read-only until approved; never print secrets; never add listen [::]:443 on the VPS (tailscaled holds 100.123.166.61:443); do not regenerate the RustDesk keypair — it is preserved; do not deploy a clean Vaultwarden — restore the existing data; keep media/*arr on the NAS (iGPU + same-filesystem hardlinks); prefer parallel migration + rollback over in-place.

Appendix D — Verification Record for the Supplied Upgrade Plan (§19)

Every load-bearing factual claim in the supplied plan was checked on 2026-09-23 against primary sources (vendor documentation) and the live host. Claims are recorded with their verdict so a future reader does not have to re-derive trust.

Sources consulted: - Pangolin official docs — System Architecture (component reference: Traefik/Gerbil/Badger/Newt/Olm), Choosing a VPS (specs), DNS & Networking (ports, subnet_group, docker port publishing). - NetBird official docs — Self-hosted vs Cloud-hosted (updated 2026-08-27: v0.62 local users, 4–5 containers, ~1 GB RAM, single relay, DIY HA). - Technitium DNS Server project documentation (authoritative + recursive, Docker, HTTP API, DoH/TLS/QUIC, clustering, DNSSEC). - Independent industry assessment of Headscale production readiness (July 2026).

Live host measurements taken for this verification:

Result: 6 of 8 load-bearing claims fully verified; 2 verified-but-qualified (Pangolin's resource floor is higher than "modest" suggests relative to this box; Headscale is production-ready only "for its intended scope" and does not replicate Tailscale's DERP/MagicDNS parity in full). No claim in the plan was found to be factually false. The blockers are integration conflicts with this specific host, not errors in the plan's reasoning — which is why §19.4 separates "right" from "needs strengthening."

End of report. Discovery and analysis only — no migration, shutdown, deletion, or reconfiguration was performed or authorized. The supplied upgrade plan (§19) is recorded as user intent, independently verified, and NOT executed.