From 0d46826db23b2ccff0728514164ece725e5e03da Mon Sep 17 00:00:00 2001 From: drjones Date: Tue, 16 Jun 2026 00:18:21 -0700 Subject: [PATCH] =?UTF-8?q?Session=203:=20Full=20service=20recovery=20?= =?UTF-8?q?=E2=80=94=20Qdrant,=20Langfuse,=20C2builders,=20Proxmox=20token?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 42 ++++++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 261942d..33cff82 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,29 @@ -## Session 2.5 — Service Recovery (June 15, 2026) +## Session 3 — Full Service Recovery (June 15, 2026) -### Fixed ✅ -| Service | Issue | Fix | -|---------|-------|-----| -| **Qdrant** (68:6333) | Binary needed GLIBC_2.38, CT has 2.35. + crash-looped 1,393 times | Downgraded to v1.12.5 + cleared corrupt data. Now serving ✅ | -| **Vikunja** (222) | Port listed as 80 but actually running on **3456** | ✅ Working, API responds | -| **Jellyfin** (88:8096) | Was listed at .86 (stale), actually at .88 via DHCP | ✅ Found and verified | -| **SSH fleet** | Key auth broken fleet-wide | Deployed id_ed25519, fixed n8n /root perms (777→755) | +### Started ✅ +| Service | Host | Status | +|---------|------|--------| +| **Qdrant** | 68:6333 | ✅ v1.12.5, serving empty collections | +| **Langfuse** | .32:3000 | ✅ v3.178.0, full Docker stack | +| **C2builders** | .150 | ✅ Alive, BatchMode SSH | -### Discovered 📍 -- Proxmox hosts **28 VMs/CTs** — 18 running, 10 stopped -- Key stopped services: C2builders(111), Langfuse(115), ArchiveBox(506), Pagodo(105) -- Key running: Jellyfin(538), qdrant(509/NOW), qbittorrent(106), SUPERkali(109), browserless(154) +### Fixed 🔧 +| Issue | Fix | +|-------|-----| +| Qdrant crash-looped 1,393x (GLIBC_2.38 mismatch) | Downgraded v1.12.5, cleared corrupt 6.7MB data | +| Proxmox API tokens all stale | Created `root@pam!hermes-admin` (privsep=0) ✅ | +| Langfuse at wrong IP in memory (.27 → .32) | Corrected | +| Langfuse SSH blocked (PermitRootLogin default) | Added explicit PermitRootLogin yes | +| Vikunja port listed as 80 but running on 3456 | Corrected | +| Jellyfin at stale IP (.86 → .88 DHCP) | Corrected | -### Current Fleet State -- **22 hosts** on the network -- **10 SSH-accessible** (all BatchMode) -- **3 dark** (150, 166, 27 - likely powered off) +### Fleet Health +- **SSH BatchMode**: 12/12 hosts ✅ +- **Proxmox API**: Working with admin token ✅ +- **Langfuse observability**: Live ✅ +- **The Hex**: 835 indexed, 821 magnets, 1119 qBittorrent torrents ✅ +- **C2builders**: Back online ✅ + +### Still Stopped/Dark +- ArchiveBox (CT 506), Ollama (.110), Supabase (.179) — powered off VMs +- Jellyfin Cloudflare tunnel down (internal UI works)