drjones/homelab-gym
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Session 3: Full service recovery — Qdrant, Langfuse, C2builders, Proxmox token

Browse Source
This commit is contained in:
drjones
2026-06-16 00:18:21 -07:00
parent 4f5fac68cb
commit 0d46826db2
1 changed files with 26 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
README.md
View File

@@ -1,19 +1,29 @@
## Session 2.5 — Service Recovery (June 15, 2026) ## Session 3 — Full Service Recovery (June 15, 2026)
### Fixed ✅ ### Started ✅
| Service | Issue | Fix | | Service | Host | Status |
|---------|-------|-----| |---------|------|--------|
| **Qdrant** (68:6333) | Binary needed GLIBC_2.38, CT has 2.35. + crash-looped 1,393 times | Downgraded to v1.12.5 + cleared corrupt data. Now serving ✅ | | **Qdrant** | 68:6333 | ✅ v1.12.5, serving empty collections |
| **Vikunja** (222) | Port listed as 80 but actually running on **3456** | ✅ Working, API responds | | **Langfuse** | .32:3000 | ✅ v3.178.0, full Docker stack |
| **Jellyfin** (88:8096) | Was listed at .86 (stale), actually at .88 via DHCP | ✅ Found and verified | | **C2builders** | .150 | ✅ Alive, BatchMode SSH |
| **SSH fleet** | Key auth broken fleet-wide | Deployed id_ed25519, fixed n8n /root perms (777→755) |
### Discovered 📍 ### Fixed 🔧
- Proxmox hosts **28 VMs/CTs** — 18 running, 10 stopped | Issue | Fix |
- Key stopped services: C2builders(111), Langfuse(115), ArchiveBox(506), Pagodo(105) |-------|-----|
- Key running: Jellyfin(538), qdrant(509/NOW), qbittorrent(106), SUPERkali(109), browserless(154) | Qdrant crash-looped 1,393x (GLIBC_2.38 mismatch) | Downgraded v1.12.5, cleared corrupt 6.7MB data |
| Proxmox API tokens all stale | Created `root@pam!hermes-admin` (privsep=0) ✅ |
| Langfuse at wrong IP in memory (.27 → .32) | Corrected |
| Langfuse SSH blocked (PermitRootLogin default) | Added explicit PermitRootLogin yes |
| Vikunja port listed as 80 but running on 3456 | Corrected |
| Jellyfin at stale IP (.86 → .88 DHCP) | Corrected |
### Current Fleet State ### Fleet Health
- **22 hosts** on the network - **SSH BatchMode**: 12/12 hosts ✅
- **10 SSH-accessible** (all BatchMode) - **Proxmox API**: Working with admin token ✅
- **3 dark** (150, 166, 27 - likely powered off) - **Langfuse observability**: Live ✅
- **The Hex**: 835 indexed, 821 magnets, 1119 qBittorrent torrents ✅
- **C2builders**: Back online ✅
### Still Stopped/Dark
- ArchiveBox (CT 506), Ollama (.110), Supabase (.179) — powered off VMs
- Jellyfin Cloudflare tunnel down (internal UI works)