Merge HyperSwap next-level work: measured tuning, persistence, closed-loop autotune
Six commits reworking the arbitrator, memory accounting, telemetry and overclock handling. The through-line is replacing assumptions with measurements, several of which turned out to be wrong: - VRAM yields are confirmed against NVML rather than fire-and-forget. The HTTP call returns in ~63ms; the driver needs a further ~77ms to release 14.9GB. That window is where ComfyUI could allocate into occupied VRAM. - Page-cache residency is measured with cachestat(2), with a randomised read-rate probe where the kernel refuses it. mincore(2) had been claiming 128GB resident on a box with 46GB of page cache. - Cache-hit classification is calibrated against real cold and warm loads (0.38 vs 2.63 GB/s for the same 12.87GB model), not PCIe bus bandwidth. - Overclock profiles are rebuilt from sweeps. Clock offsets turned out to be silently ignored by this driver, clock locks changed nothing for either workload, and LLM decode is not power-bound at all. Only ComfyUI's 370W limit earns its keep (+2.8%). - Fans are automatic everywhere. 48k samples show 81C all-time max and zero thermal throttle events, while the ollama profile had been holding 49.6C at 87% fan. - Telemetry and events persist to SQLite so profile performance can be compared at all. - A thermal governor de-escalates on sustained heat, and stock state is restored on shutdown and via systemd ExecStopPost. ComfyUI VRAM is no longer purged between workflow iterations, the 1Hz telemetry sample is taken once and fanned out rather than recomputed per client, and the MCP surface is back in parity at 23 tools and 6 resources.
This commit is contained in:
drjones
5
.gitignore
vendored
5
.gitignore
vendored
@@ -5,3 +5,8 @@ __pycache__/
|
||||
.venv/
|
||||
venv/
|
||||
.DS_Store
|
||||
|
||||
# persistent telemetry store
|
||||
hyperswap.db
|
||||
hyperswap.db-wal
|
||||
hyperswap.db-shm
|
||||
|
||||
167
README.md
167
README.md
@@ -20,39 +20,95 @@
|
||||
## 1. Feature Matrix
|
||||
|
||||
### ⚡ Bidirectional VRAM Hot-Swapping & Arbitration
|
||||
* **Sub-25ms Soft-Yield**: Instantly releases Ollama VRAM allocations (`keep_alive: 0`) down to 0 MB when ComfyUI needs to run diffusion workloads without evicting weights from system RAM.
|
||||
* **Auto-Purge for ComfyUI**: Automatically purges diffusion pipeline checkpoints and VRAM buffers when an image/video generation job finishes, releasing 100% of VRAM back to Ollama.
|
||||
* **Real-Time ComfyUI WebSocket & Watchdog Listener**: Subscribes directly to `ws://127.0.0.1:8188/ws` and runs a 300ms watchdog loop to detect prompt queueing and node execution in real time.
|
||||
* **Confirmed Soft-Yield (barrier, not fire-and-forget)**: Releases Ollama VRAM allocations (`keep_alive: 0`) down to 0 MB, then **waits on NVML until the driver has actually freed the allocation** before letting ComfyUI proceed. Posting `keep_alive: 0` only *asks* Ollama to unload; on this box the HTTP call returns in ~63 ms while the driver takes a further ~77 ms to release 14.9 GB. Returning during that window is how diffusion ends up allocating into VRAM that is still occupied.
|
||||
* **Idle-Aware ComfyUI Purge**: Diffusion checkpoints are held for `COMFY_IDLE_PURGE_S` (30 s) of genuinely empty queue rather than purged 1.5 s after every prompt — iterating on a workflow no longer pays a full checkpoint reload per run. An immediate purge still happens the moment Ollama actually asks for VRAM (`POST /api/request-vram`).
|
||||
* **Real-Time ComfyUI WebSocket & Watchdog Listener**: Subscribes directly to `ws://127.0.0.1:8188/ws`. The WebSocket is the primary signal; a connection-pooled watchdog polls `/queue` at 1 Hz purely as a fallback, backing off to 3 s while the socket is healthy.
|
||||
* **Process-Level VRAM Attribution**: Live NVML process inspection attributes exact GPU memory usage across Ollama (`llama-server`), ComfyUI (`python`), and Desktop display servers (`gnome-shell`, `Xorg`).
|
||||
* **Hot-Swap Transition History**: Circular buffer logs all model switch events, swap durations (in ms), tokens/sec throughput, and RAM cache hit status (`RAM Cache Hit ⚡` vs `Cold Disk Load 💾`).
|
||||
* **Bandwidth-Classified Transition History**: Every switch is classified by the bandwidth it actually achieved (`model size ÷ load duration`) rather than a fixed duration threshold: `RAM Cache Hit ⚡` (≥5 GB/s), `Partial Cache 🌤` (≥1.5 GB/s), `Cold Disk Load 💾` (below that). The previous `load_duration < 2500 ms` rule called a 12.9 GB model read at 2.9 GB/s a "cold disk load" and a 0.5 GB model read from NVMe a "cache hit".
|
||||
|
||||
### 🧠 64GB Host RAM Cache & Page Pre-warmer
|
||||
* **Zero-Latency Model Discovery**: Automatic cataloging of all local Ollama models (`/usr/share/ollama/.ollama/models`, `~/.ollama/models`) and ComfyUI model directories (`checkpoints`, `diffusion_models`, `unet`, `vae`, `clip`, `loras`, `controlnet`).
|
||||
* **POSIX `fadvise` & Pinned Pre-warmer**: Pre-faults multi-gigabyte GGUFs and Safetensors into the Linux OS Page Cache so that reloading models across PCIe 4.0 x16 runs at ~31.5 GB/s (sub-second VRAM loads).
|
||||
* **Granular Pre-warming Controls**: Pre-warm all discovered models in bulk or target individual models/safetensors on demand.
|
||||
* **Memory Telemetry**: Real-time breakdown of Total Host RAM, Applications Memory, Active Model Page Cache, Free Memory, and Cache Residency Ratio.
|
||||
* **POSIX `fadvise` & Pinned Pre-warmer**: Pre-faults multi-gigabyte GGUFs and Safetensors into the Linux OS Page Cache so that reloading models runs at page-cache speed rather than disk speed.
|
||||
* **Measured Residency via `cachestat(2)`**: Residency is measured, not assumed. `cachestat(2)` gives exact cached-page counts per file. Where the kernel refuses it — it only permits introspection of files you own, and Ollama's blobs are owned by uid `ollama` — HyperSwap falls back to a randomised read-rate probe and labels the result as such. Files it cannot measure are reported as unmeasurable rather than guessed at.
|
||||
* **Budgeted, Ranked Warming**: This box has 64 GB of RAM and >270 GB of model files; reading everything simply evicts whatever was warmed first. Files are ranked by recency/frequency (from the persisted event log) and warmed until a byte budget is spent, skipping anything already resident. `GET /api/warm-plan` previews the decision without executing it.
|
||||
* **Memory Telemetry**: Real-time breakdown of Total Host RAM, Applications Memory, Active Model Page Cache, Free Memory, and measured Cache Residency Ratio.
|
||||
|
||||
### 🎛️ Dynamic Overclocking & Thermal Management
|
||||
* **Workload-Aware Overclock Profiles**:
|
||||
* **`ollama` Profile (Memory-Bandwidth Bound)**: Max 370W power limit, +150 MHz Core Offset, +825 MHz Memory Offset, and 100% fan speed for maximum prompt eval / generation bandwidth.
|
||||
* **`comfy` Profile (Compute Bound)**: Max 370W power limit, +100 MHz Core Offset, +500 MHz Memory Offset, Core Clock locked to 2900–3105 MHz, and 75% fan speed for maximum diffusion compute.
|
||||
* **`balanced` Profile (Stock/General Purpose)**: Unlocked 370W power limit with stock dynamic boost curves and automatic fan control.
|
||||
* **Hardware Actuation Hierarchy**:
|
||||
* Level 1: Power Limit Control (`nvidia-smi -pl 370`).
|
||||
* Level 2: Core & Memory Clock Locking (`nvidia-smi -lgc` / `-lmc`).
|
||||
* Level 3: Clock Offsets via headless X display (`:8`) with Coolbits support (`nvidia-settings`).
|
||||
* **Hardware Fan Control**: Switch between `auto` and `manual` PWM control (30%–100%) with synchronized dual-fan actuation (`[fan:0]` and `[fan:1]`).
|
||||
* **Automated Lockstep Profile Switching**: AutoArbitrator automatically switches hardware profiles in lockstep with the active workload (`comfy` on generation start, `ollama` on completion).
|
||||
### 🎛️ Measured Overclock Profiles & Thermal Management
|
||||
|
||||
Every profile setting in this repo is now backed by a measurement from `autotune.py` on
|
||||
this specific card and driver. Several long-standing settings turned out to do nothing.
|
||||
|
||||
**What this driver actually honours** (NVIDIA 595.84, RTX 4080 SUPER):
|
||||
|
||||
| Lever | Mechanism | Works? |
|
||||
| :--- | :--- | :--- |
|
||||
| Power limit | `nvidia-smi -pl` | ✅ Yes — and it is the only lever that changes anything measurable |
|
||||
| Core / memory clock lock | `nvidia-smi -lgc` / `-lmc` | ✅ Applies correctly, but made no measurable difference to either workload |
|
||||
| Core / memory clock offsets | `nvidia-settings -a ...Offset` | ❌ **Silently ignored.** The driver reports `assigned value 0` and the attribute still reads back `250`. Detected automatically by `offsets_supported()`; `apply_profile` now skips them and says so rather than pretending. |
|
||||
| Fan control | `nvidia-settings GPUTargetFanSpeed` | ✅ Yes |
|
||||
|
||||
**Measured results** (`POST /api/autotune/sweep`):
|
||||
|
||||
*LLM decode is not power-bound.* Throughput is flat across the card's entire power range —
|
||||
the GPU never drew more than 224 W no matter what the limit allowed:
|
||||
|
||||
| Power limit | 222 W | 259 W | 296 W | 320 W | 333 W | 370 W |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| tok/s (`qwen3.8long`) | 73.17 | 73.13 | 73.43 | 73.51 | 73.10 | 73.04 |
|
||||
|
||||
*Diffusion is power-bound.* Here the watts genuinely buy throughput:
|
||||
|
||||
| Power limit | 222 W | 259 W | 296 W | 320 W | 333 W | 370 W |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| it/s (SDXL 1024, 20 steps) | 5.48 | 6.22 | 6.50 | 6.52 | 6.63 | **6.71** |
|
||||
|
||||
*Clock locks changed nothing for either workload.* Memory clock: 72.6 tok/s locked at
|
||||
11251 MHz vs 72.7 unlocked. Core clock: 6.73 it/s unlocked vs 6.77 locked at 3105 MHz —
|
||||
and 6.78 at 2400 MHz, so diffusion here is not core-clock-bound at all.
|
||||
|
||||
*Memory bandwidth is the decode bottleneck*, confirming the profile's original premise —
|
||||
dropping the memory clock to 5001 MHz halves throughput (35.9 tok/s vs 72.6). The card
|
||||
simply reaches its top memory clock on its own; pinning it there adds nothing.
|
||||
|
||||
**Resulting profiles**:
|
||||
* **`ollama`** — 320 W (stock), no locks, automatic fans. Decode draws ~224 W and is
|
||||
bandwidth-bound, so the previous 370 W limit and 100% fan pinning bought nothing.
|
||||
* **`comfy`** — 370 W, no locks, automatic fans. The extra power is worth a measured
|
||||
**+2.8%** over the 320 W stock default.
|
||||
* **`balanced`** — stock power and boost, automatic fans.
|
||||
|
||||
**On fans**: all three profiles previously pinned the fans to manual 100%. Across 48,435
|
||||
telemetry samples this card has never exceeded **81 °C** and has logged **zero** thermal
|
||||
throttle events; the `ollama` profile was holding 49.6 °C average by running the fans at
|
||||
87%. Fans are now automatic in every profile, with the thermal governor escalating them
|
||||
only if the card actually needs it.
|
||||
|
||||
### 🌡️ Thermal Governor (closed-loop de-escalation)
|
||||
* Every overclock lever here is sticky: a profile locks clocks and pins the fans to a manual PWM, and nothing used to undo that. The governor watches the telemetry the sampler already collects (so it costs no extra NVML calls) and walks the overclock back through a four-step derate ladder when the card runs hot or reports a hardware throttle.
|
||||
* **Hysteresis by design**: escalation needs 5 consecutive bad samples, recovery needs 30 consecutive good ones, with a 20 s cooldown between changes — a single spike during a diffusion step will not cause profile thrash.
|
||||
* **Guaranteed restore**: stock clocks, default power limit and automatic fans are restored by the server's shutdown hook *and* by a systemd `ExecStopPost=`, so a `SIGKILL` cannot leave the card with locked clocks and fans pinned at 100%.
|
||||
|
||||
### 🔬 Overclock Autotune (`autotune.py`)
|
||||
* Sweeps a knob (`power_limit_w`, `lock_mem_mhz`, `lock_core_max`, clock offsets) and reports the **fastest stable** value.
|
||||
* **Both workloads are measurable.** `workload=ollama` benchmarks decode throughput in tok/s; `workload=comfy` queues a fixed SDXL 1024/20-step graph through ComfyUI's API and measures it/s. Without the second one there was no way to tell whether the compute-oriented `comfy` profile was doing anything at all — and it was not.
|
||||
* **Refuses to sweep a knob the driver ignores.** A preflight applies a probe value and confirms the hardware moved; the probe is chosen as the candidate furthest from the current reading, since probing with the maximum proves nothing when the card already sits there. This is what caught the silently-discarded clock offsets.
|
||||
* **Instability detection**: kernel `Xid`/`NVRM` messages via `journalctl -k`, benchmark failure, degenerate output, and a temperature ceiling. The sweep stops climbing the moment a step looks unstable.
|
||||
* **Honest gain reporting**: gain against the profile's *current* setting is reported separately from the spread across values tried. Conflating them turns a flat result into a headline "+102%".
|
||||
* **Safety**: refuses to start while ComfyUI is executing, suspends the arbitrator's automatic profile switching for the duration (otherwise a diffusion benchmark trips `trigger_comfy_priority`, which reapplies the whole profile and overwrites the clock being measured), and restores the original profile in a `finally` block — including on exception or cancellation.
|
||||
|
||||
### 🗄️ Persistent Telemetry Store (`telemetry_store.py`)
|
||||
* Swap history used to be an in-memory `deque(maxlen=50)` that evaporated on every restart. Telemetry and events now persist to SQLite (WAL, single writer thread, batched 1 Hz inserts, automatic retention pruning) at roughly **0.4 MB per hour**.
|
||||
* This is what makes the app's central question answerable: **`GET /api/analytics/profiles` compares decode throughput per overclock profile**, joined against the thermals recorded while that profile was active.
|
||||
|
||||
### 📊 Real-Time Web Telemetry Dashboard (`:9090`)
|
||||
* **Live Hardware Telemetry**: GPU utilization %, GPU temperature (°C), power draw (W), fan speeds (%), and graphics/memory clock frequencies (MHz).
|
||||
* **Live Dual-Axis Time-Series Chart**: Real-time graphical visualization of VRAM usage (GB) and Host RAM Cache (GB) with zero frontend polling overhead.
|
||||
* **Interactive Control Center**: Trigger model hot-swaps, soft-yields, cache pre-warms, fan adjustments, and benchmarks directly from the web interface.
|
||||
* **Server-Sent Events (SSE)**: Pushes unified 1Hz telemetry updates via `GET /api/stream`.
|
||||
* **Server-Sent Events (SSE)**: A single background sampler produces one 1 Hz snapshot and fans it out to every subscriber via `GET /api/stream`. Previously each connected client independently re-ran the whole snapshot — NVML, `/proc/meminfo`, an HTTP round-trip each to Ollama and ComfyUI, and a recursive walk of the ComfyUI models tree with a `stat()` per checkpoint — once per second, so opening the dashboard in three tabs tripled the load on the thing it was measuring. Slow clients drop stale frames instead of stalling the sampler.
|
||||
|
||||
### 🤖 Model Context Protocol (MCP 2.0) Server
|
||||
* **12 Native Agentic Tools**: Allows AI agents (Antigravity CLI, Claude Desktop, Cursor) to manage GPU resources, trigger model hot-swaps, tune fan curves, and inspect telemetry.
|
||||
* **3 Live MCP Resources**: Exposes live metrics, model catalogs, and switch logs as streamable resources (`gpu://metrics/live`, `gpu://models/catalog`, `gpu://history/switches`).
|
||||
* **23 Native Agentic Tools**: Allows AI agents (Antigravity CLI, Claude Desktop, Cursor) to manage GPU resources, trigger model hot-swaps, measure page-cache residency, read persisted performance analytics, drive the thermal governor, and run overclock sweeps.
|
||||
* **6 Live MCP Resources**: Live metrics, model catalog, switch log, measured cache residency, per-profile analytics, and the overclock profiles with the evidence behind each setting.
|
||||
* **Dual Transport Support**: Run via standard input/output (`--stdio`) or network Server-Sent Events (`--sse --port 8001`).
|
||||
|
||||
### ⏱️ Automated Latency & Throughput Benchmark Engine
|
||||
@@ -79,20 +135,33 @@ flowchart TD
|
||||
REST["REST API & OpenAPI Docs"]
|
||||
MCP["Model Context Protocol (MCP 2.0)"]
|
||||
SSE["1Hz Real-Time SSE Stream"]
|
||||
Arbitrator["VRAM Arbitrator (15ms Soft-Yield)"]
|
||||
Arbitrator["VRAM Arbitrator (confirmed yield)"]
|
||||
Overclock["Overclock & Fan Manager"]
|
||||
Warmer["Page Cache Pre-Warmer"]
|
||||
end
|
||||
|
||||
HostRAM <== "PCIe 4.0 x16 Bus (~31.5 GB/s Hot-Swap)" ==> GPU
|
||||
HostRAM <== "PCIe 4.0 x16 Bus (measured 2.6 GB/s warm model load)" ==> GPU
|
||||
Orchestrator --> GPU
|
||||
Orchestrator --> HostRAM
|
||||
```
|
||||
|
||||
### The Physics of Sub-Second Switching
|
||||
* **Host RAM as Staging**: Active LLMs and diffusion checkpoints remain resident in the 64GB Linux Page Cache.
|
||||
* **PCIe 4.0 x16 Hot-Swapping**: Transferring weights across PCIe 4.0 x16 achieves **~31.5 GB/s** bandwidth, reducing model loads from 30+ seconds (disk) to **under 1.5 seconds**.
|
||||
* **Soft-Yielding**: Dropping Ollama's VRAM allocation via `keep_alive: 0` takes **~15ms** while preserving the weights in host RAM.
|
||||
* **Warm vs cold model loads, measured.** The same 12.87 GB model, loaded through Ollama on this box:
|
||||
|
||||
| Page-cache residency | Load time | Effective rate |
|
||||
| :--- | :--- | :--- |
|
||||
| 3.1% (dropped with `FADV_DONTNEED`) | 34.3 s | 0.38 GB/s |
|
||||
| 100% (force-warmed) | 4.9 s | 2.63 GB/s |
|
||||
|
||||
A **6.9× speedup**, and the reason the page cache matters. Note the effective rate is
|
||||
well below the PCIe 4.0 x16 bus rate and below the 6.4 GB/s the page cache itself
|
||||
reads at: Ollama's `load_duration` also covers host-to-device transfer and model
|
||||
initialisation, not just the file read. Classification thresholds are calibrated
|
||||
against these measured numbers rather than the theoretical bus bandwidth — an earlier
|
||||
5 GB/s cache-hit bar sat above what a fully warm load can even achieve, so every warm
|
||||
load was misreported as a partial hit.
|
||||
* **Soft-Yielding**: Dropping Ollama's VRAM allocation via `keep_alive: 0` preserves the weights in host RAM. Measured on this box: the HTTP request returns in **~63 ms**, and the driver finishes releasing 14.9 GB **~77 ms after that**. HyperSwap waits for the second number before handing VRAM to ComfyUI — the earlier "~15 ms" figure timed the request, not the release.
|
||||
|
||||
---
|
||||
|
||||
@@ -119,12 +188,36 @@ The HyperSwap server runs on port `9090` by default. Interactive OpenAPI/Swagger
|
||||
| Endpoint | Method | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `/api/switch-model` | `POST` | Hot-swaps the active Ollama LLM in VRAM and tracks transition timing. |
|
||||
| `/api/free-vram` | `POST` | Instructs Ollama to soft-yield VRAM down to 0 MB in ~15ms while retaining RAM cache. |
|
||||
| `/api/free-vram` | `POST` | Soft-yields Ollama VRAM to 0 MB and **waits for NVML to confirm the release** (`?confirm=false` to skip). Returns `request_ms`, `confirm_ms` and the GB actually freed. |
|
||||
| `/api/comfy-free` | `POST` | Instructs ComfyUI to purge loaded diffusion weights and VRAM cache. |
|
||||
| `/api/warm-all` | `POST` | Pre-faults all installed Ollama models and ComfyUI Safetensors into the Linux page cache. |
|
||||
| `/api/warm-model` | `POST` | Pre-warms a specific model or file into RAM. |
|
||||
| `/api/request-vram` | `POST` | Ollama-priority path: purges ComfyUI immediately if there is not enough free VRAM. |
|
||||
| `/api/warm-all` | `POST` | Warms the highest-value models into page cache within a byte budget (`budget_gb`). |
|
||||
| `/api/warm-plan` | `GET` | Previews what warming would read, in what order, and what it would skip — without doing it. |
|
||||
| `/api/warm-model` | `POST` | Pre-warms a specific model or file into RAM (`blob_only` warms weights without touching VRAM). |
|
||||
| `/api/cache/report` | `GET` | Measured page-cache residency per model file, with the measurement method used for each. |
|
||||
| `/api/benchmark` | `POST` | Runs an automated back-and-forth model swap benchmark and calculates average latency. |
|
||||
|
||||
### Analytics Endpoints (persisted)
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `/api/analytics/profiles` | `GET` | **Decode throughput per overclock profile**, joined with the thermals recorded under it. |
|
||||
| `/api/analytics/swaps` | `GET` | Aggregated swap/yield/purge latencies, cache-hit split, and per-model throughput. |
|
||||
| `/api/analytics/timeseries` | `GET` | Downsampled telemetry history for charts that outlive a page refresh. |
|
||||
| `/api/analytics/models` | `GET` | Recency/frequency model ranking used to prioritise the warm budget. |
|
||||
| `/api/history?durable=true` | `GET` | Swap history from the persistent store rather than the in-memory ring. |
|
||||
| `/api/db` | `GET` | Store location, row counts and how many hours of history are held. |
|
||||
|
||||
### Governor & Autotune Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `/api/governor` | `GET` / `POST` | Current derate level and why; enable/disable, or clear an active derate. |
|
||||
| `/api/overclock/restore` | `POST` | Drop all clock locks and offsets, restore default power limit and automatic fans. |
|
||||
| `/api/autotune` | `GET` | Sweep progress, last result, and every recorded autotune step. |
|
||||
| `/api/autotune/sweep` | `POST` | Sweep a knob against a real workload (`workload`: `ollama` decode tok/s, `comfy` SDXL it/s), verifying the knob moves the hardware first. |
|
||||
| `/api/autotune/cancel` | `POST` | Stop the current sweep after the step in flight; the profile is restored either way. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Model Context Protocol (MCP 2.0) Reference
|
||||
@@ -140,19 +233,33 @@ HyperSwap includes a native **MCP 2.0 server** (`mcp_server.py`) exposing orches
|
||||
| **`set_gpu_fan_speed`** | `mode` (str), `percent` (optional int) | Sets fan speed mode (`auto`\|`manual`) and target PWM % (30–100%). |
|
||||
| **`get_host_memory_status`** | *None* | 64GB host RAM breakdown, active page cache size, and cache ratio. |
|
||||
| **`switch_ollama_model`** | `model_name` (str), `keep_alive` (str) | Hot-swaps active LLM in VRAM, measures latency (ms) and tokens/sec. |
|
||||
| **`soft_yield_ollama_vram`** | `model_name` (optional str) | Yields Ollama VRAM to 0 MB in ~15ms while keeping model weights in RAM cache. |
|
||||
| **`soft_yield_ollama_vram`** | `model_name` (optional str) | Yields Ollama VRAM to 0 MB and waits for NVML to confirm the driver actually released it. Returns the request/confirm split. |
|
||||
| **`purge_comfyui_vram`** | *None* | Purges loaded diffusion models from ComfyUI pipeline VRAM. |
|
||||
| **`prewarm_all_models_to_ram`** | *None* | Faults all local LLM and diffusion checkpoints into Linux OS page cache. |
|
||||
| **`prewarm_all_models_to_ram`** | *None* | Warms the highest-value models into page cache within a byte budget, skipping what is already resident. |
|
||||
| **`prewarm_single_model`** | `model_name` (optional str), `filepath` (optional str) | Pre-warms a single GGUF or Safetensors file into RAM. |
|
||||
| **`list_available_models`** | *None* | Lists all installed Ollama models and discovered ComfyUI Safetensors on disk. |
|
||||
| **`get_switch_history`** | `limit` (int, default 20) | Retrieves recent switch events, millisecond latencies, and RAM hit status. |
|
||||
| **`run_model_switch_benchmark`**| `iterations` (int, default 2) | Automated round-trip latency benchmark between installed models. |
|
||||
| **`get_page_cache_residency`** | `include_files` (bool) | Measured page-cache residency per model file, with the measurement method used for each. |
|
||||
| **`get_warm_plan`** | `budget_gb` (optional float) | Previews what warming would read and skip, ranked by recency/frequency. Does not warm. |
|
||||
| **`request_vram_for_ollama`** | `needed_gb` (float) | Purges ComfyUI's checkpoints immediately if VRAM headroom is short, bypassing the idle timer. |
|
||||
| **`get_profile_performance`** | `days` (float, default 7) | Measured tok/s and thermals per overclock profile, from persisted history. |
|
||||
| **`get_thermal_governor_status`** | *None* | Current derate level, the reason for it, and escalation history. |
|
||||
| **`set_thermal_governor`** | `enabled` (optional bool), `reset` (bool) | Enable/disable the governor, or clear an active derate. |
|
||||
| **`get_overclock_status`** | *None* | Active profile, all profiles with their evidence, and which levers this driver honours. |
|
||||
| **`apply_overclock_profile`** | `profile` (str) | Apply `ollama` \| `comfy` \| `balanced`. |
|
||||
| **`restore_stock_gpu_state`** | *None* | Drop clock locks and offsets, restore default power limit, return fans to automatic. |
|
||||
| **`run_overclock_sweep`** | `knob`, `profile`, `workload`, `start`, `stop`, `repeats`, `apply_best` | Sweep a knob against a real workload and report the fastest stable value. Verifies the knob moves the hardware first. Takes minutes. |
|
||||
| **`get_autotune_status`** | *None* | Sweep progress, the last result table, and all recorded autotune steps. |
|
||||
|
||||
### MCP Resources List
|
||||
|
||||
* `gpu://metrics/live`: Real-time snapshot of GPU sensors and RAM page cache.
|
||||
* `gpu://models/catalog`: Catalog of all discovered GGUF and Safetensors models.
|
||||
* `gpu://history/switches`: Event log of recent model transitions and swap speeds.
|
||||
* `gpu://cache/residency`: Measured page-cache residency across every model on disk.
|
||||
* `gpu://analytics/profiles`: Measured throughput and thermals per overclock profile.
|
||||
* `gpu://overclock/profiles`: Overclock profiles including the measurement behind each setting.
|
||||
|
||||
---
|
||||
|
||||
|
||||
570
autotune.py
Normal file
570
autotune.py
Normal file
@@ -0,0 +1,570 @@
|
||||
"""Closed-loop overclock autotuner.
|
||||
|
||||
The profiles in this repo were hand-tuned and had already drifted apart from the defaults
|
||||
in overclock_manager.py, with no record of which numbers were actually faster. This module
|
||||
answers that empirically: it walks a clock offset upward, measures real decode throughput
|
||||
at each step, watches for instability, and reports the highest setting that was both
|
||||
stable and fastest.
|
||||
|
||||
Safety properties:
|
||||
* The original profile is always restored, including on exception or cancellation.
|
||||
* A sweep refuses to start while ComfyUI is executing, so it cannot corrupt someone's
|
||||
render by yanking clocks mid-graph.
|
||||
* Every step is bounded by a temperature ceiling and checked for kernel Xid messages,
|
||||
and the sweep stops climbing the moment a step looks unstable.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import random
|
||||
import subprocess
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import overclock_manager
|
||||
import telemetry_store
|
||||
import vram_arbitrator
|
||||
|
||||
logger = logging.getLogger("autotune")
|
||||
|
||||
BENCH_PROMPT = ("Write a detailed technical explanation of how virtual memory paging "
|
||||
"works in a modern operating system kernel.")
|
||||
BENCH_TOKENS = 160
|
||||
SETTLE_S = 2.5
|
||||
TEMP_CEILING_C = 84.0
|
||||
|
||||
KNOBS = {
|
||||
# Clock offsets go through nvidia-settings. On some drivers (595.84 here) the
|
||||
# attribute is accepted and then silently ignored -- assigning 0 reports success and
|
||||
# reads back 250 -- so a sweep of these can measure pure noise. _knob_effective()
|
||||
# checks before any sweep runs.
|
||||
"mem_offset_mhz": {"default_start": 0, "default_stop": 1000, "default_step": 100,
|
||||
"kind": "offset", "verify": "mem_offset"},
|
||||
"core_offset_mhz": {"default_start": 0, "default_stop": 300, "default_step": 25,
|
||||
"kind": "offset", "verify": "core_offset"},
|
||||
# Clock locks go through nvidia-smi and do work on the open/proprietary module alike.
|
||||
# Memory clock is the lever that matters for LLM decode, which is bandwidth bound.
|
||||
"lock_mem_mhz": {"kind": "discrete", "verify": "clock_mem",
|
||||
"values": None}, # filled from the card's supported clock list
|
||||
"lock_core_max": {"kind": "discrete", "verify": "clock_sm",
|
||||
"values": None},
|
||||
# The one lever this driver definitely honours. Worth knowing whether the extra
|
||||
# watts actually buy throughput, or just heat and fan noise.
|
||||
"power_limit_w": {"kind": "discrete", "verify": "power_limit",
|
||||
"values": None, "no_unlocked": True},
|
||||
}
|
||||
|
||||
|
||||
def _supported_clocks(which: str = "mem") -> List[int]:
|
||||
"""Discrete clock values the card will actually accept for -lmc / -lgc.
|
||||
|
||||
Always queried as the mem,gr pair: asking for a single field returns one column, and
|
||||
reading index 1 from it silently yields an empty list rather than an error.
|
||||
"""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["nvidia-smi", "--query-supported-clocks=mem,gr", "--format=csv,noheader,nounits"],
|
||||
capture_output=True, text=True, timeout=15)
|
||||
rows = []
|
||||
for line in proc.stdout.splitlines():
|
||||
parts = [p.strip() for p in line.split(",")]
|
||||
if len(parts) >= 2 and parts[0].isdigit() and parts[1].isdigit():
|
||||
rows.append((int(parts[0]), int(parts[1])))
|
||||
if not rows:
|
||||
return []
|
||||
if which == "mem":
|
||||
return sorted({m for m, _ in rows})
|
||||
# Graphics clocks are enumerated per memory clock; take the list for the highest
|
||||
# memory clock, which is the one any real workload runs at.
|
||||
top_mem = max(m for m, _ in rows)
|
||||
return sorted({g for m, g in rows if m == top_mem})
|
||||
except Exception as e:
|
||||
logger.debug(f"supported clock query failed: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def _supported_power_limits(steps: int = 5) -> List[int]:
|
||||
"""Power limits between the card's minimum and maximum, in even increments."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["nvidia-smi", "--query-gpu=power.min_limit,power.max_limit,power.default_limit",
|
||||
"--format=csv,noheader,nounits"],
|
||||
capture_output=True, text=True, timeout=10)
|
||||
parts = [p.strip() for p in proc.stdout.strip().split(",")]
|
||||
lo, hi, default = (int(float(parts[0])), int(float(parts[1])), int(float(parts[2])))
|
||||
except Exception as e:
|
||||
logger.debug(f"power limit query failed: {e}")
|
||||
return []
|
||||
# Start at 60% of max -- below that the card is not doing useful work for these
|
||||
# workloads -- and always include the stock default as a reference point.
|
||||
lo = max(lo, int(hi * 0.6))
|
||||
span = hi - lo
|
||||
vals = {lo + round(i * span / (steps - 1)) for i in range(steps)}
|
||||
vals.add(default)
|
||||
return sorted(v for v in vals if lo <= v <= hi)
|
||||
|
||||
|
||||
def _subsample(values: List[int], max_steps: int) -> List[int]:
|
||||
"""Evenly spaced subset, always keeping the endpoints.
|
||||
|
||||
The card enumerates ~194 graphics clocks in 15 MHz increments; benchmarking every one
|
||||
would take hours and tell us nothing that a handful of well-spread points does not.
|
||||
"""
|
||||
if len(values) <= max_steps:
|
||||
return values
|
||||
idx = [round(i * (len(values) - 1) / (max_steps - 1)) for i in range(max_steps)]
|
||||
return sorted({values[i] for i in idx})
|
||||
|
||||
|
||||
def _read_hw(field: str) -> Optional[float]:
|
||||
"""Read back the hardware state a knob is supposed to move."""
|
||||
gpu = vram_arbitrator.get_gpu_hardware_stats()
|
||||
if field == "clock_mem":
|
||||
return gpu.get("clock_mem_mhz")
|
||||
if field == "clock_sm":
|
||||
return gpu.get("clock_graphics_mhz")
|
||||
if field == "power_limit":
|
||||
return gpu.get("power_limit_w")
|
||||
if field in ("mem_offset", "core_offset"):
|
||||
r = overclock_manager._nvidia_settings(
|
||||
"-q", f"[gpu:0]/{'GPUMemoryTransferRateOffset' if field == 'mem_offset' else 'GPUGraphicsClockOffset'}[3]")
|
||||
for line in (r.get("out") or "").splitlines():
|
||||
if "Attribute" in line and "):" in line:
|
||||
try:
|
||||
return float(line.split("):")[-1].split(".")[0].strip())
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _knob_effective(knob: str, profile: str, candidates: List[int],
|
||||
baseline_value: int) -> Dict[str, Any]:
|
||||
"""Verify a knob actually moves the hardware before we sweep it.
|
||||
|
||||
Without this the tuner happily reports "best = the highest value tried" from run-to-run
|
||||
benchmark noise on a knob the driver is ignoring.
|
||||
|
||||
The probe value is chosen as the candidate *furthest* from where the hardware currently
|
||||
sits. Probing with the maximum is not good enough: if the card already happens to be at
|
||||
its top clock, setting it there again moves nothing and a perfectly good knob looks
|
||||
broken.
|
||||
"""
|
||||
field = KNOBS[knob].get("verify")
|
||||
before = _read_hw(field)
|
||||
if before is not None and candidates:
|
||||
probe_value = max(candidates, key=lambda v: abs(v - before))
|
||||
else:
|
||||
probe_value = candidates[-1] if candidates else 0
|
||||
overclock_manager.apply_profile(profile, overrides={knob: probe_value})
|
||||
time.sleep(2.0)
|
||||
after = _read_hw(field)
|
||||
overclock_manager.apply_profile(profile, overrides={knob: baseline_value})
|
||||
moved = (before is not None and after is not None and abs(after - before) > 1e-6)
|
||||
return {
|
||||
"effective": bool(moved),
|
||||
"field": field,
|
||||
"before": before,
|
||||
"after": after,
|
||||
"probe_value": probe_value,
|
||||
"detail": (f"{field} moved {before} -> {after}" if moved else
|
||||
f"{field} stayed at {after} after setting {knob}={probe_value}; "
|
||||
f"this driver accepts the setting and ignores it"),
|
||||
}
|
||||
|
||||
|
||||
def _xid_since(since_ts: float) -> List[str]:
|
||||
"""Look for NVIDIA Xid errors in the kernel log — the clearest instability signal."""
|
||||
try:
|
||||
since = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(since_ts))
|
||||
proc = subprocess.run(
|
||||
["journalctl", "-k", "--since", since, "--no-pager", "-q"],
|
||||
capture_output=True, text=True, timeout=10,
|
||||
)
|
||||
return [ln.strip() for ln in proc.stdout.splitlines()
|
||||
if "Xid" in ln or "NVRM:" in ln]
|
||||
except Exception as e:
|
||||
logger.debug(f"Xid check unavailable: {e}")
|
||||
return []
|
||||
|
||||
|
||||
async def _decode_benchmark(model: str) -> Dict[str, Any]:
|
||||
"""One fixed decode run. Throughput here is the thing being optimised."""
|
||||
client = vram_arbitrator._client(vram_arbitrator.OLLAMA_API_BASE, 300.0)
|
||||
t0 = time.perf_counter()
|
||||
resp = await client.post("/api/generate", json={
|
||||
"model": model,
|
||||
"prompt": BENCH_PROMPT,
|
||||
"stream": False,
|
||||
"keep_alive": "10m",
|
||||
"options": {"num_predict": BENCH_TOKENS, "temperature": 0.0, "seed": 42},
|
||||
})
|
||||
wall_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
if resp.status_code != 200:
|
||||
return {"ok": False, "error": f"HTTP {resp.status_code}: {resp.text[:200]}",
|
||||
"wall_ms": wall_ms}
|
||||
data = resp.json()
|
||||
eval_ms = data.get("eval_duration", 0) / 1e6
|
||||
eval_count = data.get("eval_count", 0)
|
||||
# Reasoning models put their output in `thinking` and leave `response` empty, so a
|
||||
# check that only looked at `response` flagged every one of them as degenerate.
|
||||
text = ((data.get("response") or "") + " " + (data.get("thinking") or "")).strip()
|
||||
return {
|
||||
"ok": True,
|
||||
"tokens_per_sec": round(eval_count / (eval_ms / 1000), 2) if eval_ms > 0 else 0.0,
|
||||
"eval_count": eval_count,
|
||||
"eval_ms": round(eval_ms, 2),
|
||||
"prompt_eval_ms": round(data.get("prompt_eval_duration", 0) / 1e6, 2),
|
||||
"wall_ms": wall_ms,
|
||||
"response_chars": len(text),
|
||||
# Corruption signals: the model stopped far short of the token budget, or it
|
||||
# produced text that is pure repetition. Token count is the primary signal --
|
||||
# empty text alone is not enough, since output can arrive in other fields.
|
||||
"degenerate": (eval_count < BENCH_TOKENS * 0.5
|
||||
or (len(text) > 0 and len(set(text.split())) < 8)),
|
||||
}
|
||||
|
||||
|
||||
# A fixed SDXL txt2img graph. Deterministic seed/steps/resolution so every step of a
|
||||
# sweep does identical work and the only variable is the clock. PreviewImage rather than
|
||||
# SaveImage keeps benchmark runs out of the user's output gallery.
|
||||
COMFY_BENCH_CKPT = "sd_xl_base_1.0.safetensors"
|
||||
COMFY_BENCH_STEPS = 20
|
||||
COMFY_BENCH_SIZE = 1024
|
||||
|
||||
|
||||
def _comfy_workflow(ckpt: str = COMFY_BENCH_CKPT, seed: Optional[int] = None) -> Dict[str, Any]:
|
||||
# The seed must vary per run. ComfyUI caches by node inputs, so a fixed seed makes the
|
||||
# second and later benchmarks return in ~1ms without executing anything at all. The
|
||||
# cost of the graph is identical regardless of seed, so this costs no comparability.
|
||||
seed = random.randint(1, 2**31) if seed is None else seed
|
||||
return {
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": ckpt}},
|
||||
"2": {"class_type": "CLIPTextEncode",
|
||||
"inputs": {"clip": ["1", 1],
|
||||
"text": "a detailed photograph of a mountain range at sunrise"}},
|
||||
"3": {"class_type": "CLIPTextEncode",
|
||||
"inputs": {"clip": ["1", 1], "text": "blurry, low quality"}},
|
||||
"4": {"class_type": "EmptyLatentImage",
|
||||
"inputs": {"width": COMFY_BENCH_SIZE, "height": COMFY_BENCH_SIZE, "batch_size": 1}},
|
||||
"5": {"class_type": "KSampler",
|
||||
"inputs": {"model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0],
|
||||
"latent_image": ["4", 0], "seed": seed, "steps": COMFY_BENCH_STEPS,
|
||||
"cfg": 7.0, "sampler_name": "euler", "scheduler": "normal",
|
||||
"denoise": 1.0}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}},
|
||||
}
|
||||
|
||||
|
||||
async def _diffusion_benchmark(ckpt: str = COMFY_BENCH_CKPT,
|
||||
timeout_s: float = 300.0) -> Dict[str, Any]:
|
||||
"""Queue one fixed SDXL graph and time it. This is the compute-bound counterpart to
|
||||
the decode benchmark, and the only way to tell whether the 'comfy' profile helps."""
|
||||
client = vram_arbitrator._client(vram_arbitrator.COMFY_API_BASE, 30.0)
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
resp = await client.post("/prompt", json={"prompt": _comfy_workflow(ckpt),
|
||||
"client_id": "hyperswap-autotune"})
|
||||
if resp.status_code != 200:
|
||||
return {"ok": False, "error": f"queue failed HTTP {resp.status_code}: {resp.text[:200]}"}
|
||||
prompt_id = resp.json().get("prompt_id")
|
||||
except Exception as e:
|
||||
return {"ok": False, "error": f"queue failed: {e}"}
|
||||
|
||||
while (time.perf_counter() - t0) < timeout_s:
|
||||
await asyncio.sleep(0.25)
|
||||
try:
|
||||
h = await client.get(f"/history/{prompt_id}")
|
||||
if h.status_code != 200:
|
||||
continue
|
||||
entry = (h.json() or {}).get(prompt_id)
|
||||
if not entry:
|
||||
continue
|
||||
status = entry.get("status", {})
|
||||
if status.get("status_str") == "error" or not status.get("completed", True):
|
||||
if status.get("status_str") == "error":
|
||||
return {"ok": False, "error": "ComfyUI reported an execution error",
|
||||
"wall_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
if status.get("completed"):
|
||||
wall = time.perf_counter() - t0
|
||||
# ComfyUI stamps execution_start/success in the status messages; the delta
|
||||
# between them excludes our polling overhead and the queue wait.
|
||||
stamps = {}
|
||||
for msg in status.get("messages", []):
|
||||
if isinstance(msg, list) and len(msg) >= 2 and isinstance(msg[1], dict):
|
||||
if "timestamp" in msg[1]:
|
||||
stamps[msg[0]] = msg[1]["timestamp"]
|
||||
exec_ms = None
|
||||
if "execution_start" in stamps and "execution_success" in stamps:
|
||||
exec_ms = round(stamps["execution_success"] - stamps["execution_start"], 2)
|
||||
effective_ms = exec_ms or wall * 1000
|
||||
# A graph that "finished" implausibly fast was served from ComfyUI's cache
|
||||
# rather than executed; treat it as an invalid sample, not a record score.
|
||||
cached = effective_ms < 250
|
||||
return {
|
||||
"ok": not cached,
|
||||
"error": "result served from ComfyUI cache, not executed" if cached else None,
|
||||
"wall_ms": round(wall * 1000, 2),
|
||||
"exec_ms": exec_ms,
|
||||
"steps": COMFY_BENCH_STEPS,
|
||||
"it_per_sec": round(COMFY_BENCH_STEPS / (effective_ms / 1000), 3),
|
||||
"degenerate": cached,
|
||||
}
|
||||
except Exception:
|
||||
continue
|
||||
return {"ok": False, "error": f"diffusion benchmark timed out after {timeout_s}s"}
|
||||
|
||||
|
||||
class SweepState:
|
||||
def __init__(self) -> None:
|
||||
self.running = False
|
||||
self.cancel = False
|
||||
self.current: Optional[Dict[str, Any]] = None
|
||||
self.last_result: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
state = SweepState()
|
||||
|
||||
|
||||
async def sweep(knob: str = "mem_offset_mhz",
|
||||
profile: str = "ollama",
|
||||
workload: str = "auto",
|
||||
model: Optional[str] = None,
|
||||
start: Optional[int] = None,
|
||||
stop: Optional[int] = None,
|
||||
step: Optional[int] = None,
|
||||
repeats: int = 1,
|
||||
max_steps: int = 6,
|
||||
include_unlocked: bool = True,
|
||||
apply_best: bool = False) -> Dict[str, Any]:
|
||||
"""Sweep one clock offset and return the fastest stable value."""
|
||||
if knob not in KNOBS:
|
||||
return {"success": False, "error": f"unknown knob '{knob}'; try {list(KNOBS)}"}
|
||||
if state.running:
|
||||
return {"success": False, "error": "a sweep is already running"}
|
||||
|
||||
comfy = await vram_arbitrator.get_comfyui_live_state()
|
||||
if comfy.get("executing") or comfy.get("queue_remaining"):
|
||||
return {"success": False, "error": "ComfyUI is busy; refusing to change clocks mid-render"}
|
||||
|
||||
# 'auto': tune the workload the profile is actually for.
|
||||
if workload == "auto":
|
||||
workload = "comfy" if profile == "comfy" else "ollama"
|
||||
if workload not in ("ollama", "comfy"):
|
||||
return {"success": False, "error": "workload must be 'ollama', 'comfy' or 'auto'"}
|
||||
if workload == "comfy" and not comfy.get("online"):
|
||||
return {"success": False, "error": "ComfyUI is not reachable; cannot run a diffusion sweep"}
|
||||
|
||||
if workload == "comfy":
|
||||
model = model or COMFY_BENCH_CKPT
|
||||
benchmark = lambda: _diffusion_benchmark(model)
|
||||
metric = "it_per_sec"
|
||||
else:
|
||||
benchmark = lambda: _decode_benchmark(model)
|
||||
metric = "tokens_per_sec"
|
||||
|
||||
if workload == "ollama" and not model:
|
||||
ollama = await vram_arbitrator.get_ollama_live_state()
|
||||
model = ollama.get("active_model_name")
|
||||
if not model:
|
||||
installed = ollama.get("installed_models") or []
|
||||
if not installed:
|
||||
return {"success": False, "error": "no Ollama model available to benchmark"}
|
||||
model = installed[0].get("name")
|
||||
benchmark = lambda: _decode_benchmark(model)
|
||||
|
||||
defaults = KNOBS[knob]
|
||||
if defaults.get("kind") == "discrete":
|
||||
if knob == "power_limit_w":
|
||||
supported = _supported_power_limits()
|
||||
else:
|
||||
supported = defaults.get("values") or _supported_clocks(
|
||||
"mem" if knob == "lock_mem_mhz" else "gr")
|
||||
if not supported:
|
||||
return {"success": False, "error": f"card reported no supported clocks for {knob}"}
|
||||
values = [v for v in supported
|
||||
if (start is None or v >= start) and (stop is None or v <= stop)]
|
||||
if not values:
|
||||
return {"success": False, "error": f"no supported values in range; card offers {supported}"}
|
||||
values = _subsample(values, max_steps or 6)
|
||||
# 0 means "no lock at all". That is the honest control for a profile whose whole
|
||||
# premise is that locking the clock beats letting the card boost on its own.
|
||||
if include_unlocked and not defaults.get("no_unlocked"):
|
||||
values = [0] + values
|
||||
start, stop, step = values[0], values[-1], None
|
||||
else:
|
||||
start = defaults["default_start"] if start is None else start
|
||||
stop = defaults["default_stop"] if stop is None else stop
|
||||
step = defaults["default_step"] if step is None else step
|
||||
if step <= 0 or stop < start:
|
||||
return {"success": False, "error": "invalid sweep range"}
|
||||
values = list(range(start, stop + 1, step))
|
||||
|
||||
baseline_cfg = overclock_manager.load_profiles().get(profile, {})
|
||||
baseline_value = int(baseline_cfg.get(knob, 0) or 0)
|
||||
|
||||
# Refuse to sweep a knob the driver is going to ignore.
|
||||
effectiveness = _knob_effective(knob, profile, [v for v in values if v] or values,
|
||||
baseline_value)
|
||||
if not effectiveness["effective"]:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"'{knob}' does not move this GPU: {effectiveness['detail']}",
|
||||
"effectiveness": effectiveness,
|
||||
}
|
||||
|
||||
state.running = True
|
||||
state.cancel = False
|
||||
results: List[Dict[str, Any]] = []
|
||||
t_start = time.time()
|
||||
|
||||
try:
|
||||
# Warm-up: load weights once up front so the first step does not pay the load cost.
|
||||
vram_arbitrator.arbitrator.suspend_oc("autotune sweep")
|
||||
await benchmark()
|
||||
|
||||
for value in values:
|
||||
if state.cancel:
|
||||
break
|
||||
overclock_manager.apply_profile(profile, overrides={knob: value})
|
||||
await asyncio.sleep(SETTLE_S)
|
||||
step_started = time.time()
|
||||
|
||||
samples = []
|
||||
for _ in range(max(repeats, 1)):
|
||||
samples.append(await benchmark())
|
||||
if state.cancel:
|
||||
break
|
||||
|
||||
gpu = vram_arbitrator.get_gpu_hardware_stats()
|
||||
xids = _xid_since(step_started)
|
||||
ok_samples = [s for s in samples if s.get("ok") and not s.get("degenerate")]
|
||||
temp = gpu.get("temperature_c", 0) or 0
|
||||
|
||||
instability = []
|
||||
if xids:
|
||||
instability.append(f"kernel Xid: {xids[0][:120]}")
|
||||
if len(ok_samples) < len(samples):
|
||||
instability.append("benchmark failed or produced degenerate output")
|
||||
if temp >= TEMP_CEILING_C:
|
||||
instability.append(f"temperature ceiling hit ({temp}°C)")
|
||||
|
||||
tok_s = round(max((s.get(metric, 0.0) for s in ok_samples), default=0.0), 2)
|
||||
row = {
|
||||
"knob": knob,
|
||||
"value": value,
|
||||
"profile": profile,
|
||||
"workload": workload,
|
||||
"metric": metric,
|
||||
"model": model,
|
||||
"tokens_per_sec": tok_s,
|
||||
"temp_c": temp,
|
||||
"power_w": gpu.get("power_w"),
|
||||
"clock_sm_mhz": gpu.get("clock_graphics_mhz"),
|
||||
"clock_mem_mhz": gpu.get("clock_mem_mhz"),
|
||||
"throttle_reasons": gpu.get("throttle_reasons"),
|
||||
"stable": not instability,
|
||||
"instability": "; ".join(instability) or None,
|
||||
"samples": samples,
|
||||
}
|
||||
results.append(row)
|
||||
telemetry_store.record_autotune({
|
||||
"profile": profile, "knob": knob,
|
||||
"core_offset_mhz": value if knob == "core_offset_mhz" else baseline_cfg.get("core_offset_mhz"),
|
||||
"mem_offset_mhz": value if knob == "mem_offset_mhz" else baseline_cfg.get("mem_offset_mhz"),
|
||||
"tokens_per_sec": tok_s, "temp_c": temp, "power_w": gpu.get("power_w"),
|
||||
"stable": row["stable"], "instability": row["instability"],
|
||||
"note": f"sweep {knob} {start}..{stop} step {step}",
|
||||
})
|
||||
state.current = {"knob": knob, "value": value, "stop": stop,
|
||||
"tokens_per_sec": tok_s, "stable": row["stable"]}
|
||||
logger.info(f"autotune {knob}={value}: {tok_s} tok/s, {temp}°C, "
|
||||
f"stable={row['stable']} {row['instability'] or ''}")
|
||||
|
||||
if not row["stable"]:
|
||||
logger.warning(f"autotune stopping climb at {knob}={value}: {row['instability']}")
|
||||
break
|
||||
|
||||
stable = [r for r in results if r["stable"] and r["tokens_per_sec"] > 0]
|
||||
best = max(stable, key=lambda r: r["tokens_per_sec"]) if stable else None
|
||||
|
||||
# Two different comparisons, and conflating them is how a sweep oversells itself.
|
||||
# The first step of the range is not "baseline" unless it happens to be what the
|
||||
# profile is already set to -- reporting "+102%" against the slowest value tried
|
||||
# implies a speedup the user would not actually observe.
|
||||
first_step = next((r for r in results if r["value"] == values[0]), None)
|
||||
current = next((r for r in results if r["value"] == baseline_value), None)
|
||||
gain_vs_first_step_pct = None
|
||||
if best and first_step and first_step["tokens_per_sec"] > 0:
|
||||
gain_vs_first_step_pct = round(
|
||||
(best["tokens_per_sec"] / first_step["tokens_per_sec"] - 1) * 100, 2)
|
||||
gain_vs_current_pct = None
|
||||
if best and current and current["tokens_per_sec"] > 0:
|
||||
gain_vs_current_pct = round(
|
||||
(best["tokens_per_sec"] / current["tokens_per_sec"] - 1) * 100, 2)
|
||||
|
||||
applied = None
|
||||
if apply_best and best:
|
||||
overclock_manager.set_profile(profile, {knob: best["value"]})
|
||||
applied = {knob: best["value"], "profile": profile}
|
||||
logger.info(f"autotune wrote {knob}={best['value']} into profile '{profile}'")
|
||||
|
||||
result = {
|
||||
"success": True,
|
||||
"knob": knob,
|
||||
"profile": profile,
|
||||
"workload": workload,
|
||||
"metric": metric,
|
||||
"model": model,
|
||||
"range": {"start": start, "stop": stop, "step": step, "values": values},
|
||||
"effectiveness": effectiveness,
|
||||
"steps_run": len(results),
|
||||
"duration_s": round(time.time() - t_start, 1),
|
||||
"cancelled": state.cancel,
|
||||
"best": {k: best[k] for k in ("value", "tokens_per_sec", "temp_c", "clock_mem_mhz",
|
||||
"clock_sm_mhz")} if best else None,
|
||||
"current_profile_value": baseline_value,
|
||||
"first_step_tokens_per_sec": first_step["tokens_per_sec"] if first_step else None,
|
||||
"current_tokens_per_sec": current["tokens_per_sec"] if current else None,
|
||||
"gain_vs_first_step_pct": gain_vs_first_step_pct,
|
||||
"gain_vs_current_pct": gain_vs_current_pct,
|
||||
"gain_note": ("compared against the profile's current setting"
|
||||
if current is not None else
|
||||
f"the profile's current value ({baseline_value}) was not in the "
|
||||
f"swept range, so only the spread across tried values is shown"),
|
||||
"applied_to_profile": applied,
|
||||
"first_unstable": next(({"value": r["value"], "why": r["instability"]}
|
||||
for r in results if not r["stable"]), None),
|
||||
"table": [{k: r[k] for k in ("value", "tokens_per_sec", "temp_c", "power_w",
|
||||
"clock_mem_mhz", "clock_sm_mhz", "stable",
|
||||
"instability")} for r in results],
|
||||
}
|
||||
state.last_result = result
|
||||
return result
|
||||
finally:
|
||||
# Always hand the card back exactly as we found it.
|
||||
state.running = False
|
||||
state.current = None
|
||||
vram_arbitrator.arbitrator.resume_oc(profile)
|
||||
try:
|
||||
overclock_manager.apply_profile(profile)
|
||||
logger.info(f"autotune restored profile '{profile}'")
|
||||
except Exception as e:
|
||||
logger.error(f"autotune failed to restore profile, forcing stock: {e}")
|
||||
overclock_manager.restore_safe("autotune restore failed")
|
||||
|
||||
|
||||
def get_status() -> Dict[str, Any]:
|
||||
return {
|
||||
"running": state.running,
|
||||
"current": state.current,
|
||||
"last_result": state.last_result,
|
||||
"knobs": KNOBS,
|
||||
"history": telemetry_store.autotune_history(100),
|
||||
}
|
||||
|
||||
|
||||
def cancel() -> Dict[str, Any]:
|
||||
if not state.running:
|
||||
return {"cancelled": False, "reason": "no sweep running"}
|
||||
state.cancel = True
|
||||
return {"cancelled": True}
|
||||
128
mcp_server.py
128
mcp_server.py
@@ -7,16 +7,19 @@ import logging
|
||||
from typing import Dict, List, Any, Optional
|
||||
|
||||
from mcp.server import MCPServer
|
||||
import ram_optimizer
|
||||
import vram_arbitrator
|
||||
import autotune
|
||||
import overclock_manager
|
||||
import ram_optimizer
|
||||
import telemetry_store
|
||||
import thermal_governor
|
||||
import vram_arbitrator
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s")
|
||||
logger = logging.getLogger("gpu_swapper_mcp")
|
||||
|
||||
mcp = MCPServer(
|
||||
name="gpu-program-swapper",
|
||||
version="1.0.0",
|
||||
version="2.0.0",
|
||||
description="Orchestrates high-speed GPU VRAM hot-swaps between Ollama LLMs and ComfyUI with 64GB RAM cache telemetry."
|
||||
)
|
||||
|
||||
@@ -133,6 +136,105 @@ def set_gpu_fan_speed(mode: str = "auto", percent: Optional[int] = None) -> str:
|
||||
res = overclock_manager.set_fan_auto()
|
||||
return json.dumps(res, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
def get_page_cache_residency(include_files: bool = True) -> str:
|
||||
"""Measure how much of each model on disk is genuinely resident in the Linux page cache.
|
||||
|
||||
Uses cachestat(2) where the kernel permits it and a read-rate probe where it does not
|
||||
(Ollama blobs are owned by another user). Reports which method was used per file, and
|
||||
marks anything it cannot measure rather than guessing."""
|
||||
report = ram_optimizer.get_cache_report(include_files=include_files)
|
||||
report["capability"] = ram_optimizer.residency_capability()
|
||||
return json.dumps(report, indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def get_warm_plan(budget_gb: Optional[float] = None) -> str:
|
||||
"""Preview which models pre-warming would load into RAM, in what order, and what it
|
||||
would skip — ranked by recency/frequency and capped by a byte budget. Does not warm."""
|
||||
return json.dumps(ram_optimizer.build_warm_plan(budget_gb), indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def request_vram_for_ollama(needed_gb: float = 0.0) -> str:
|
||||
"""Free VRAM for an LLM right now: purges ComfyUI's cached checkpoints immediately if
|
||||
there is not enough headroom, instead of waiting for the normal idle timer."""
|
||||
res = await vram_arbitrator.arbitrator.request_vram_for_ollama(needed_gb)
|
||||
return json.dumps(res, indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def get_profile_performance(days: float = 7.0) -> str:
|
||||
"""Compare measured decode throughput and thermals per overclock profile, from
|
||||
persisted history. Answers whether a given profile is actually delivering more tok/s."""
|
||||
return json.dumps({
|
||||
"window_days": days,
|
||||
"profiles": telemetry_store.profile_comparison(days),
|
||||
"swaps": telemetry_store.swap_stats(days),
|
||||
}, indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def get_thermal_governor_status() -> str:
|
||||
"""Current thermal derate level, why it was applied, and the escalation history."""
|
||||
return json.dumps(thermal_governor.governor.get_status(), indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def set_thermal_governor(enabled: Optional[bool] = None, reset: bool = False) -> str:
|
||||
"""Enable or disable the thermal governor, or clear an active derate and reapply the
|
||||
full profile."""
|
||||
if enabled is not None:
|
||||
thermal_governor.governor.set_enabled(enabled)
|
||||
if reset:
|
||||
thermal_governor.governor.reset()
|
||||
return json.dumps(thermal_governor.governor.get_status(), indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def get_overclock_status() -> str:
|
||||
"""Active overclock profile, all profiles with the evidence behind their settings, and
|
||||
which hardware levers this driver actually honours (clock offsets are ignored on some)."""
|
||||
return json.dumps(overclock_manager.get_status(), indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def apply_overclock_profile(profile: str) -> str:
|
||||
"""Apply an overclock profile by name: ollama | comfy | balanced."""
|
||||
return json.dumps(overclock_manager.apply_profile(profile), indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def restore_stock_gpu_state() -> str:
|
||||
"""Drop all clock locks and offsets, restore the default power limit, and return the
|
||||
fans to automatic control."""
|
||||
return json.dumps(overclock_manager.restore_safe("MCP request"), indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def run_overclock_sweep(knob: str = "power_limit_w", profile: str = "ollama",
|
||||
workload: str = "auto", start: Optional[int] = None,
|
||||
stop: Optional[int] = None, repeats: int = 1,
|
||||
apply_best: bool = False) -> str:
|
||||
"""Sweep one GPU knob against a real workload and report the fastest stable value.
|
||||
|
||||
knob: power_limit_w | lock_mem_mhz | lock_core_max | mem_offset_mhz | core_offset_mhz
|
||||
workload: 'ollama' (decode tok/s), 'comfy' (SDXL it/s), or 'auto' to match the profile.
|
||||
|
||||
Verifies the knob actually moves the hardware before sweeping, refuses to run while
|
||||
ComfyUI is busy, and always restores the original profile. Takes minutes."""
|
||||
res = await autotune.sweep(knob=knob, profile=profile, workload=workload,
|
||||
start=start, stop=stop, repeats=repeats,
|
||||
apply_best=apply_best)
|
||||
return json.dumps(res, indent=2, default=str)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def get_autotune_status() -> str:
|
||||
"""Sweep progress, the last sweep's full result table, and every recorded autotune step."""
|
||||
return json.dumps(autotune.get_status(), indent=2, default=str)
|
||||
|
||||
|
||||
# ==========================================
|
||||
# MCP RESOURCES
|
||||
# ==========================================
|
||||
@@ -154,6 +256,21 @@ def get_switch_history_resource() -> str:
|
||||
"""Recent model switch events and latencies."""
|
||||
return json.dumps(vram_arbitrator.get_switch_history(), indent=2)
|
||||
|
||||
@mcp.resource("gpu://cache/residency")
|
||||
def get_cache_residency_resource() -> str:
|
||||
"""Measured page-cache residency across every model on disk."""
|
||||
return json.dumps(ram_optimizer.get_cache_report(include_files=True), indent=2, default=str)
|
||||
|
||||
@mcp.resource("gpu://analytics/profiles")
|
||||
def get_profile_analytics_resource() -> str:
|
||||
"""Measured throughput and thermals per overclock profile, from persisted history."""
|
||||
return json.dumps(telemetry_store.profile_comparison(7.0), indent=2, default=str)
|
||||
|
||||
@mcp.resource("gpu://overclock/profiles")
|
||||
def get_overclock_profiles_resource() -> str:
|
||||
"""Overclock profiles, including the measurement recorded behind each setting."""
|
||||
return json.dumps(overclock_manager.get_status(), indent=2, default=str)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
@@ -163,6 +280,11 @@ if __name__ == "__main__":
|
||||
parser.add_argument("--port", type=int, default=8001, help="Port for SSE transport")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Only when run as a standalone server. server.py imports this module for the
|
||||
# benchmark tool, and starting the store at import scope would spin up a writer as a
|
||||
# side effect of that import.
|
||||
telemetry_store.start()
|
||||
|
||||
if args.sse:
|
||||
mcp.run(transport="sse", host="0.0.0.0", port=args.port)
|
||||
else:
|
||||
|
||||
@@ -67,6 +67,46 @@ DEFAULT_PROFILES: Dict[str, Dict[str, Any]] = {
|
||||
},
|
||||
}
|
||||
|
||||
_OFFSETS_SUPPORTED: Optional[bool] = None
|
||||
|
||||
|
||||
def offsets_supported(recheck: bool = False) -> bool:
|
||||
"""Whether nvidia-settings clock offsets actually take effect on this driver.
|
||||
|
||||
Driver 595.84 accepts GPUGraphicsClockOffset/GPUMemoryTransferRateOffset and silently
|
||||
discards them: assigning 0 returns success and the attribute still reads back its old
|
||||
value. Profiles carrying core_offset_mhz/mem_offset_mhz were therefore configuring
|
||||
nothing. Probed once and cached.
|
||||
"""
|
||||
global _OFFSETS_SUPPORTED
|
||||
if _OFFSETS_SUPPORTED is not None and not recheck:
|
||||
return _OFFSETS_SUPPORTED
|
||||
|
||||
def _read() -> Optional[int]:
|
||||
q = _nvidia_settings("-q", "[gpu:0]/GPUGraphicsClockOffset[3]")
|
||||
for line in (q.get("out") or "").splitlines():
|
||||
if "Attribute" in line and "):" in line:
|
||||
try:
|
||||
return int(line.split("):")[-1].split(".")[0].strip())
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
before = _read()
|
||||
if before is None:
|
||||
_OFFSETS_SUPPORTED = False
|
||||
return False
|
||||
probe = before + 25
|
||||
_nvidia_settings("-a", f"[gpu:0]/GPUGraphicsClockOffset[3]={probe}")
|
||||
after = _read()
|
||||
_nvidia_settings("-a", f"[gpu:0]/GPUGraphicsClockOffset[3]={before}")
|
||||
_OFFSETS_SUPPORTED = (after is not None and after != before)
|
||||
if not _OFFSETS_SUPPORTED:
|
||||
logger.warning("Clock offsets are not honoured by this driver "
|
||||
f"(set {probe}, read back {after}); profile offset fields are inert.")
|
||||
return _OFFSETS_SUPPORTED
|
||||
|
||||
|
||||
ACTIVE_PROFILE = "balanced"
|
||||
_LAST_RESULT: Dict[str, Any] = {}
|
||||
FAN_MANUAL = False
|
||||
@@ -207,14 +247,20 @@ def _apply_offsets(core_mhz: int, mem_mhz: int) -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
def apply_profile(name: str) -> Dict[str, Any]:
|
||||
"""Apply a named overclock profile to the GPU. Returns a full result report."""
|
||||
def apply_profile(name: str, overrides: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
|
||||
"""Apply a named overclock profile to the GPU. Returns a full result report.
|
||||
|
||||
`overrides` lets the thermal governor and the autotuner apply a modified version of a
|
||||
profile (a derated offset, a probe clock) without mutating what is stored on disk.
|
||||
"""
|
||||
global ACTIVE_PROFILE, _LAST_RESULT
|
||||
profiles = load_profiles()
|
||||
if name not in profiles:
|
||||
return {"success": False, "error": f"unknown profile '{name}'", "profile": name}
|
||||
|
||||
cfg = profiles[name]
|
||||
cfg = dict(profiles[name])
|
||||
if overrides:
|
||||
cfg.update(overrides)
|
||||
fan_mode = cfg.get("fan_mode", "auto")
|
||||
fan_speed = int(cfg.get("fan_speed_pct", 0))
|
||||
|
||||
@@ -225,20 +271,40 @@ def apply_profile(name: str) -> Dict[str, Any]:
|
||||
"power_limit": _apply_power_limit(int(cfg.get("power_limit_w", 370))),
|
||||
"clock_lock": _apply_clock_lock(int(cfg.get("lock_core_min", 0)), int(cfg.get("lock_core_max", 0))),
|
||||
"mem_lock": _apply_mem_lock(int(cfg.get("lock_mem_mhz", 0))),
|
||||
"offsets": _apply_offsets(int(cfg.get("core_offset_mhz", 0)), int(cfg.get("mem_offset_mhz", 0))),
|
||||
"offsets": (_apply_offsets(int(cfg.get("core_offset_mhz", 0)),
|
||||
int(cfg.get("mem_offset_mhz", 0)))
|
||||
if offsets_supported() else
|
||||
{"applied": False, "supported": False,
|
||||
"detail": "skipped: this driver accepts clock offsets and ignores them"}),
|
||||
"fan": apply_fan_control(fan_mode, fan_speed),
|
||||
}
|
||||
result["gpu"] = get_gpu_state()
|
||||
result["fan_status"] = get_fan_status()
|
||||
result["overrides"] = overrides or {}
|
||||
|
||||
_STATE_CACHE["value"] = None
|
||||
_FAN_CACHE["value"] = None
|
||||
ACTIVE_PROFILE = name
|
||||
_LAST_RESULT = result
|
||||
logger.info(f"Overclock profile applied: {name} -> {json.dumps(result, default=str)}")
|
||||
return result
|
||||
|
||||
|
||||
def get_gpu_state() -> Dict[str, Any]:
|
||||
"""Read back live GPU clocks/power/limits via nvidia-smi."""
|
||||
_STATE_CACHE: Dict[str, Any] = {"ts": 0.0, "value": None}
|
||||
_FAN_CACHE: Dict[str, Any] = {"ts": 0.0, "value": None}
|
||||
STATE_TTL_S = 2.0
|
||||
|
||||
|
||||
def get_gpu_state(force: bool = False) -> Dict[str, Any]:
|
||||
"""Read back live GPU clocks/power/limits via nvidia-smi.
|
||||
|
||||
Cached for STATE_TTL_S: this forks `sudo nvidia-smi`, and the dashboard polls the
|
||||
status endpoint every few seconds. NVML already covers the live 1 Hz telemetry.
|
||||
"""
|
||||
import time as _time
|
||||
if not force and _STATE_CACHE["value"] is not None and \
|
||||
(_time.time() - _STATE_CACHE["ts"]) < STATE_TTL_S:
|
||||
return _STATE_CACHE["value"]
|
||||
state: Dict[str, Any] = {}
|
||||
r = _smi(
|
||||
"--query-gpu=driver_version,name,memory.total,power.limit,power.max_limit,power.default_limit,"
|
||||
@@ -257,6 +323,7 @@ def get_gpu_state() -> Dict[str, Any]:
|
||||
state[k] = float(parts[i])
|
||||
except ValueError:
|
||||
state[k] = parts[i]
|
||||
_STATE_CACHE.update({"ts": __import__("time").time(), "value": state})
|
||||
return state
|
||||
|
||||
|
||||
@@ -298,11 +365,16 @@ def set_fan_auto() -> Dict[str, Any]:
|
||||
ok = r["rc"] == 0
|
||||
if ok:
|
||||
FAN_MANUAL = False
|
||||
_FAN_CACHE["value"] = None
|
||||
return {"success": ok, "manual": False, "fan_speed_pct": None, "detail": r.get("out") or r.get("err")}
|
||||
|
||||
|
||||
def get_fan_status() -> Dict[str, Any]:
|
||||
"""Read current fan control mode + target speed."""
|
||||
def get_fan_status(force: bool = False) -> Dict[str, Any]:
|
||||
"""Read current fan control mode + target speed (cached; forks nvidia-settings)."""
|
||||
import time as _time
|
||||
if not force and _FAN_CACHE["value"] is not None and \
|
||||
(_time.time() - _FAN_CACHE["ts"]) < STATE_TTL_S:
|
||||
return _FAN_CACHE["value"]
|
||||
global FAN_MANUAL
|
||||
target = None
|
||||
manual = FAN_MANUAL
|
||||
@@ -320,13 +392,43 @@ def get_fan_status() -> Dict[str, Any]:
|
||||
target = int(line.split("):")[-1].split(".")[0].strip())
|
||||
except Exception:
|
||||
pass
|
||||
return {"manual": manual, "mode": "manual" if manual else "auto", "target_speed_pct": target}
|
||||
result = {"manual": manual, "mode": "manual" if manual else "auto", "target_speed_pct": target}
|
||||
_FAN_CACHE.update({"ts": __import__("time").time(), "value": result})
|
||||
return result
|
||||
|
||||
|
||||
def restore_safe(reason: str = "shutdown") -> Dict[str, Any]:
|
||||
"""Return the card to stock: no clock locks, no offsets, default power, automatic fans.
|
||||
|
||||
This matters because every lever here is sticky. If the service dies while a profile is
|
||||
applied, the GPU keeps the locked clocks and, worse, keeps the fans pinned at whatever
|
||||
manual PWM was last set. Nothing was undoing that.
|
||||
"""
|
||||
logger.warning(f"Restoring GPU to safe stock state ({reason})")
|
||||
result = {
|
||||
"reason": reason,
|
||||
"clock_lock": _apply_clock_lock(0, 0),
|
||||
"mem_lock": _apply_mem_lock(0),
|
||||
"offsets": _apply_offsets(0, 0),
|
||||
"fan": set_fan_auto(),
|
||||
}
|
||||
# Hand the power limit back to the card's own default rather than assuming 370 W.
|
||||
state = get_gpu_state()
|
||||
default_w = state.get("power_default_w")
|
||||
if isinstance(default_w, (int, float)) and default_w > 0:
|
||||
result["power_limit"] = _apply_power_limit(int(default_w))
|
||||
global ACTIVE_PROFILE
|
||||
ACTIVE_PROFILE = "stock"
|
||||
return result
|
||||
|
||||
|
||||
def get_status() -> Dict[str, Any]:
|
||||
"""Full overclock status for the dashboard."""
|
||||
return {
|
||||
"active_profile": ACTIVE_PROFILE,
|
||||
"offsets_supported": offsets_supported(),
|
||||
"effective_levers": (["power_limit", "clock_lock", "mem_lock", "fan"]
|
||||
+ (["offsets"] if offsets_supported() else [])),
|
||||
"profiles": load_profiles(),
|
||||
"gpu": get_gpu_state(),
|
||||
"fan": get_fan_status(),
|
||||
|
||||
@@ -1,29 +1,32 @@
|
||||
{
|
||||
"ollama": {
|
||||
"label": "Ollama \u2014 LLM decode (memory-bandwidth bound)",
|
||||
"power_limit_w": 370,
|
||||
"core_offset_mhz": 150,
|
||||
"mem_offset_mhz": 825,
|
||||
"label": "Ollama — LLM decode (memory-bandwidth bound; measured insensitive to power and clocks)",
|
||||
"measured": "73.0-73.5 tok/s flat from 222W to 370W (qwen3.8long, 2026-08-28). Actual draw never exceeded 224W at any limit. Memory clock lock made no difference (72.6 locked vs 72.7 unlocked).",
|
||||
"power_limit_w": 320,
|
||||
"core_offset_mhz": 0,
|
||||
"mem_offset_mhz": 0,
|
||||
"lock_core_min": 0,
|
||||
"lock_core_max": 0,
|
||||
"lock_mem_mhz": 0,
|
||||
"fan_mode": "manual",
|
||||
"fan_speed_pct": 100
|
||||
"fan_mode": "auto",
|
||||
"fan_speed_pct": 0
|
||||
},
|
||||
"comfy": {
|
||||
"label": "ComfyUI \u2014 diffusion (core-compute bound)",
|
||||
"label": "ComfyUI — diffusion (compute bound; genuinely power-scaling)",
|
||||
"measured": "SDXL 1024/20-step: 5.48 it/s @222W, 6.22 @259W, 6.50 @296W, 6.52 @320W, 6.63 @333W, 6.71 @370W (2026-08-28). Worth +2.8% over the 320W stock default. Core clock lock made no difference across 2400-3105 MHz.",
|
||||
"power_limit_w": 370,
|
||||
"core_offset_mhz": 100,
|
||||
"mem_offset_mhz": 500,
|
||||
"lock_core_min": 2900,
|
||||
"lock_core_max": 3105,
|
||||
"core_offset_mhz": 0,
|
||||
"mem_offset_mhz": 0,
|
||||
"lock_core_min": 0,
|
||||
"lock_core_max": 0,
|
||||
"lock_mem_mhz": 0,
|
||||
"fan_mode": "manual",
|
||||
"fan_speed_pct": 75
|
||||
"fan_mode": "auto",
|
||||
"fan_speed_pct": 0
|
||||
},
|
||||
"balanced": {
|
||||
"label": "Balanced \u2014 stock boost, power unlocked",
|
||||
"power_limit_w": 370,
|
||||
"label": "Balanced — stock power and boost, automatic fans",
|
||||
"measured": "Card's own design point. 48k telemetry samples show 67.8C average under load at 39.5% auto fan, 81C all-time max, zero thermal throttle events.",
|
||||
"power_limit_w": 320,
|
||||
"core_offset_mhz": 0,
|
||||
"mem_offset_mhz": 0,
|
||||
"lock_core_min": 0,
|
||||
|
||||
643
ram_optimizer.py
643
ram_optimizer.py
@@ -1,16 +1,48 @@
|
||||
"""RAM Optimizer and Model Pre-warmer for High-Speed Switching."""
|
||||
import os
|
||||
import glob
|
||||
import time
|
||||
import httpx
|
||||
"""RAM Optimizer and Model Pre-warmer for High-Speed Switching.
|
||||
|
||||
Two things changed here versus the naive version:
|
||||
|
||||
1. Residency is *measured*, not assumed. mincore(2) tells us exactly what fraction of
|
||||
each model file is resident in the Linux page cache, so "RAM Cache Hit" stops being
|
||||
a guess based on how long a load took.
|
||||
2. Warming is *budgeted*. This box has 64 GB of RAM and >33 GB of models; reading every
|
||||
file top-to-bottom simply evicts whatever was warmed first. Files are now scored by
|
||||
recency/frequency (from the telemetry store) and warmed until a byte budget is hit,
|
||||
skipping anything already resident.
|
||||
"""
|
||||
import ctypes
|
||||
import ctypes.util
|
||||
import json
|
||||
import logging
|
||||
from typing import Dict, List, Any
|
||||
import os
|
||||
import random
|
||||
import time
|
||||
from typing import Dict, List, Any, Optional, Tuple
|
||||
|
||||
import httpx
|
||||
|
||||
import telemetry_store
|
||||
|
||||
logger = logging.getLogger("ram_optimizer")
|
||||
|
||||
OLLAMA_API_BASE = "http://localhost:11434"
|
||||
COMFY_API_BASE = "http://127.0.0.1:8188"
|
||||
COMFY_MODELS_DIR = "/home/drjones/ComfyUI/models"
|
||||
COMFY_MODELS_DIR = os.environ.get("HYPERSWAP_COMFY_MODELS", "/home/drjones/ComfyUI/models")
|
||||
OLLAMA_MODEL_DIRS = [
|
||||
"/usr/share/ollama/.ollama/models",
|
||||
os.path.expanduser("~/.ollama/models"),
|
||||
]
|
||||
|
||||
PAGE_SIZE = os.sysconf("SC_PAGE_SIZE")
|
||||
# Files bigger than this are sampled rather than fully mapped for residency.
|
||||
RESIDENCY_FULL_MAP_LIMIT = 2 * 1024 ** 3
|
||||
RESIDENCY_SAMPLE_WINDOWS = 64
|
||||
RESIDENCY_WINDOW_BYTES = 16 * 1024 * 1024
|
||||
# A file at/above this residency is considered warm and is skipped by the warmer.
|
||||
WARM_SKIP_THRESHOLD_PCT = 90.0
|
||||
|
||||
CATALOG_TTL_S = 30.0
|
||||
|
||||
|
||||
def get_detailed_meminfo() -> Dict[str, Any]:
|
||||
"""Parse /proc/meminfo for precise page cache and RAM stats."""
|
||||
@@ -51,22 +83,423 @@ def get_detailed_meminfo() -> Dict[str, Any]:
|
||||
"cache_ratio_pct": round((cached / total * 100) if total > 0 else 0, 1),
|
||||
}
|
||||
|
||||
def warm_file_to_ram(filepath: str, chunk_size: int = 16 * 1024 * 1024) -> Dict[str, Any]:
|
||||
"""Pre-fault/read file into Linux OS Page Cache at maximum disk read speed."""
|
||||
|
||||
# ---------------------------------------------------------------- page residency
|
||||
#
|
||||
# Measuring page-cache residency turned out to be the subtle part.
|
||||
#
|
||||
# * cachestat(2) (Linux 6.5+) is the right tool: exact cached-page counts for an fd,
|
||||
# no mmap, microseconds per call. But the kernel only permits it on files you own
|
||||
# or can write -- the Ollama blobs are owned by uid `ollama`, so it returns EPERM.
|
||||
# * mincore(2) does NOT fail closed for those files on this kernel: it reports every
|
||||
# page as resident, which produced 128 GB of "resident" model weights on a box with
|
||||
# 46 GB of page cache. It is therefore not used at all.
|
||||
#
|
||||
# So: cachestat where permitted, and an explicit read-throughput probe where it is not.
|
||||
# Anything we cannot measure is reported as unmeasurable rather than guessed at.
|
||||
|
||||
_libc = None
|
||||
_SYS_cachestat = 451 # x86_64
|
||||
|
||||
|
||||
class _CachestatRange(ctypes.Structure):
|
||||
_fields_ = [("off", ctypes.c_uint64), ("len", ctypes.c_uint64)]
|
||||
|
||||
|
||||
class _Cachestat(ctypes.Structure):
|
||||
_fields_ = [
|
||||
("nr_cache", ctypes.c_uint64),
|
||||
("nr_dirty", ctypes.c_uint64),
|
||||
("nr_writeback", ctypes.c_uint64),
|
||||
("nr_evicted", ctypes.c_uint64),
|
||||
("nr_recently_evicted", ctypes.c_uint64),
|
||||
]
|
||||
|
||||
|
||||
def _get_libc():
|
||||
global _libc
|
||||
if _libc is None:
|
||||
_libc = ctypes.CDLL(ctypes.util.find_library("c") or "libc.so.6", use_errno=True)
|
||||
return _libc
|
||||
|
||||
|
||||
def _cachestat(fd: int, offset: int, length: int) -> Optional[_Cachestat]:
|
||||
"""Raw cachestat(2). Returns None if the kernel refuses (EPERM/ENOSYS)."""
|
||||
libc = _get_libc()
|
||||
rng = _CachestatRange(offset, length)
|
||||
cs = _Cachestat()
|
||||
ctypes.set_errno(0)
|
||||
rc = libc.syscall(ctypes.c_long(_SYS_cachestat), ctypes.c_int(fd),
|
||||
ctypes.byref(rng), ctypes.byref(cs), ctypes.c_uint(0))
|
||||
if rc != 0:
|
||||
return None
|
||||
return cs
|
||||
|
||||
|
||||
PROBE_WINDOWS = 12
|
||||
PROBE_WINDOW_BYTES = 2 * 1024 * 1024
|
||||
# Measured on this box: cold NVMe reads land around 0.35-0.5 GB/s, page-cache reads at
|
||||
# 3.2-13 GB/s. 1.5 GB/s sits in the empty middle of that gap.
|
||||
PROBE_CACHED_GBPS = 1.5
|
||||
|
||||
|
||||
def _throughput_probe(fd: int, size: int, windows_override: Optional[int] = None) -> Dict[str, Any]:
|
||||
"""Infer residency by timing reads of small windows spread across the file.
|
||||
|
||||
Used only where cachestat is not permitted (Ollama's blobs are owned by uid `ollama`).
|
||||
|
||||
Two details matter for correctness:
|
||||
|
||||
* Offsets are random per call. A fixed stride made the probe self-fulfilling: the
|
||||
first pass faulted its 24 MB of sample windows into the page cache, and every pass
|
||||
after that re-read exactly those windows and reported 100% resident for a file that
|
||||
was almost entirely cold.
|
||||
* Windows that read cold are handed straight back with FADV_DONTNEED. Those pages are
|
||||
pollution the probe itself created, and leaving them behind would slowly warm the
|
||||
cache with data nobody asked for.
|
||||
"""
|
||||
windows = min(windows_override or PROBE_WINDOWS, max(int(size // PROBE_WINDOW_BYTES), 1))
|
||||
if windows <= 0:
|
||||
return {"resident_pct": 0.0, "windows": 0}
|
||||
|
||||
max_off = max(size - PROBE_WINDOW_BYTES, 0)
|
||||
offsets = sorted(random.randint(0, max_off) for _ in range(windows)) if max_off else [0]
|
||||
buf = bytearray(PROBE_WINDOW_BYTES)
|
||||
cached = 0
|
||||
rates = []
|
||||
for off in offsets:
|
||||
length = min(PROBE_WINDOW_BYTES, size - off)
|
||||
if length <= 0:
|
||||
continue
|
||||
view = memoryview(buf)[:length]
|
||||
t0 = time.perf_counter()
|
||||
os.preadv(fd, [view], off)
|
||||
dt = time.perf_counter() - t0
|
||||
gbps = (length / (1024 ** 3)) / dt if dt > 0 else 0.0
|
||||
rates.append(gbps)
|
||||
if gbps >= PROBE_CACHED_GBPS:
|
||||
cached += 1
|
||||
else:
|
||||
# We just pulled this off disk; put it back the way we found it.
|
||||
try:
|
||||
os.posix_fadvise(fd, off, length, os.POSIX_FADV_DONTNEED)
|
||||
except Exception:
|
||||
pass
|
||||
n = len(rates)
|
||||
return {
|
||||
"resident_pct": round((cached / n * 100) if n else 0.0, 1),
|
||||
"windows": n,
|
||||
"median_gbps": round(sorted(rates)[n // 2], 2) if n else 0.0,
|
||||
"sampled_gb": round(n * PROBE_WINDOW_BYTES / (1024 ** 3), 3),
|
||||
}
|
||||
|
||||
|
||||
def page_residency(filepath: str, allow_probe: bool = True,
|
||||
probe_windows: Optional[int] = None) -> Dict[str, Any]:
|
||||
"""Measure what fraction of a file is resident in the Linux page cache."""
|
||||
try:
|
||||
size = os.path.getsize(filepath)
|
||||
except OSError as e:
|
||||
return {"success": False, "error": str(e), "resident_pct": 0.0, "measurable": False}
|
||||
if size == 0:
|
||||
return {"success": True, "resident_pct": 0.0, "size_bytes": 0, "measurable": True,
|
||||
"method": "empty"}
|
||||
|
||||
try:
|
||||
fd = os.open(filepath, os.O_RDONLY)
|
||||
except OSError as e:
|
||||
return {"success": False, "error": str(e), "resident_pct": 0.0, "measurable": False}
|
||||
try:
|
||||
cs = _cachestat(fd, 0, size)
|
||||
if cs is not None:
|
||||
total_pages = (size + PAGE_SIZE - 1) // PAGE_SIZE
|
||||
pct = round((cs.nr_cache / total_pages * 100) if total_pages else 0.0, 1)
|
||||
method, measurable = "cachestat", True
|
||||
extra = {"dirty_pages": cs.nr_dirty, "evicted_pages": cs.nr_evicted}
|
||||
elif allow_probe:
|
||||
probe = _throughput_probe(fd, size, probe_windows)
|
||||
pct = probe["resident_pct"]
|
||||
method, measurable = "probe", True
|
||||
extra = {"probe_windows": probe["windows"], "probe_median_gbps": probe.get("median_gbps")}
|
||||
else:
|
||||
return {"success": True, "filepath": filepath, "size_bytes": size,
|
||||
"size_gb": round(size / (1024**3), 3), "resident_pct": None,
|
||||
"measurable": False, "method": "unavailable", "warm": None,
|
||||
"reason": "cachestat not permitted for this file (not owned by us)"}
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"filepath": filepath,
|
||||
"size_bytes": size,
|
||||
"size_gb": round(size / (1024**3), 3),
|
||||
"resident_pct": pct,
|
||||
"resident_bytes": int(size * pct / 100.0),
|
||||
"method": method,
|
||||
"measurable": measurable,
|
||||
"warm": pct >= WARM_SKIP_THRESHOLD_PCT,
|
||||
# Only an exact measurement is trustworthy enough to skip work on. A probe of a
|
||||
# dozen 2 MB windows can clear 90% on a file that is mostly cold -- observed
|
||||
# here as a 12.87 GB "already resident" blob that then loaded at 2.44 GB/s.
|
||||
"warm_confident": (method == "cachestat" and pct >= WARM_SKIP_THRESHOLD_PCT)
|
||||
or (method == "probe" and pct >= 100.0),
|
||||
**extra,
|
||||
}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "resident_pct": 0.0,
|
||||
"size_bytes": size, "measurable": False}
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def residency_capability() -> Dict[str, Any]:
|
||||
"""Report whether exact residency is available, and how to enable it if not."""
|
||||
catalog = get_model_catalog()
|
||||
blocked = []
|
||||
for f in catalog["ollama"]:
|
||||
try:
|
||||
fd = os.open(f["full_path"], os.O_RDONLY)
|
||||
except OSError:
|
||||
continue
|
||||
try:
|
||||
if _cachestat(fd, 0, 4096) is None:
|
||||
blocked.append(f["full_path"])
|
||||
finally:
|
||||
os.close(fd)
|
||||
break # one probe is enough; blobs share a directory and owner
|
||||
if not blocked:
|
||||
return {"exact_everywhere": True}
|
||||
owner = ""
|
||||
try:
|
||||
import pwd
|
||||
owner = pwd.getpwuid(os.stat(blocked[0]).st_uid).pw_name
|
||||
except Exception:
|
||||
owner = str(os.stat(blocked[0]).st_uid)
|
||||
return {
|
||||
"exact_everywhere": False,
|
||||
"method_for_blocked": "probe",
|
||||
"reason": f"cachestat(2) is only permitted on files you own or can write; "
|
||||
f"Ollama blobs are owned by '{owner}'",
|
||||
"hint": f"exact numbers for Ollama weights need read/write access, e.g. "
|
||||
f"'sudo usermod -aG {owner} $USER' plus group-write on the blobs directory",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- catalogs
|
||||
|
||||
_catalog_cache: Dict[str, Any] = {"ts": 0.0, "sig": None, "comfy": [], "ollama": []}
|
||||
|
||||
|
||||
def _dir_signature(root: str) -> Tuple:
|
||||
"""Cheap fingerprint of a model tree: (mtime, entry count) per subdirectory."""
|
||||
sig = []
|
||||
if not os.path.isdir(root):
|
||||
return tuple(sig)
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
try:
|
||||
sig.append((dirpath, os.stat(dirpath).st_mtime_ns, len(filenames)))
|
||||
except OSError:
|
||||
continue
|
||||
return tuple(sig)
|
||||
|
||||
|
||||
def find_ollama_model_files() -> List[Dict[str, Any]]:
|
||||
"""Map installed Ollama models to their on-disk GGUF blobs via the manifest tree.
|
||||
|
||||
Knowing the blob path is what lets us warm (or measure) a specific model's weights
|
||||
without pulling them into VRAM.
|
||||
"""
|
||||
results: List[Dict[str, Any]] = []
|
||||
seen = set()
|
||||
for root in OLLAMA_MODEL_DIRS:
|
||||
manifests = os.path.join(root, "manifests")
|
||||
blobs = os.path.join(root, "blobs")
|
||||
if not os.path.isdir(manifests):
|
||||
continue
|
||||
for dirpath, _, filenames in os.walk(manifests):
|
||||
for tag in filenames:
|
||||
manifest_path = os.path.join(dirpath, tag)
|
||||
try:
|
||||
with open(manifest_path) as f:
|
||||
manifest = json.load(f)
|
||||
except Exception:
|
||||
continue
|
||||
rel = os.path.relpath(dirpath, manifests)
|
||||
parts = rel.split(os.sep)
|
||||
# registry/namespace/name -> "name:tag", keeping non-library namespaces
|
||||
name = parts[-1] if parts else rel
|
||||
namespace = parts[-2] if len(parts) >= 2 else "library"
|
||||
model_name = f"{name}:{tag}" if namespace == "library" else f"{namespace}/{name}:{tag}"
|
||||
for layer in manifest.get("layers", []):
|
||||
if layer.get("mediaType") != "application/vnd.ollama.image.model":
|
||||
continue
|
||||
digest = (layer.get("digest") or "").replace(":", "-")
|
||||
blob_path = os.path.join(blobs, digest)
|
||||
if not os.path.exists(blob_path):
|
||||
continue
|
||||
key = (model_name, blob_path)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
size = layer.get("size") or os.path.getsize(blob_path)
|
||||
results.append({
|
||||
"model": model_name,
|
||||
"filename": digest,
|
||||
"full_path": blob_path,
|
||||
"size_bytes": size,
|
||||
"size_gb": round(size / (1024**3), 3),
|
||||
"kind": "ollama",
|
||||
})
|
||||
return results
|
||||
|
||||
|
||||
def find_comfy_model_files(force_refresh: bool = False) -> List[Dict[str, Any]]:
|
||||
"""Discover all model files under ComfyUI models (cached).
|
||||
|
||||
This used to run inside the 1Hz telemetry snapshot, meaning a full recursive walk plus
|
||||
a stat() of every checkpoint once per second per connected dashboard. It is now cached
|
||||
behind a directory-mtime fingerprint.
|
||||
"""
|
||||
_refresh_catalog(force_refresh)
|
||||
return _catalog_cache["comfy"]
|
||||
|
||||
|
||||
def get_model_catalog(force_refresh: bool = False) -> Dict[str, Any]:
|
||||
_refresh_catalog(force_refresh)
|
||||
return {
|
||||
"comfy": _catalog_cache["comfy"],
|
||||
"ollama": _catalog_cache["ollama"],
|
||||
"cached_at": _catalog_cache["ts"],
|
||||
}
|
||||
|
||||
|
||||
def _refresh_catalog(force: bool = False) -> None:
|
||||
now = time.time()
|
||||
if not force and (now - _catalog_cache["ts"]) < CATALOG_TTL_S:
|
||||
return
|
||||
sig = _dir_signature(COMFY_MODELS_DIR)
|
||||
if not force and sig == _catalog_cache["sig"] and _catalog_cache["comfy"]:
|
||||
_catalog_cache["ts"] = now
|
||||
return
|
||||
|
||||
extensions = (".safetensors", ".ckpt", ".pt", ".bin", ".gguf", ".sft")
|
||||
results = []
|
||||
if os.path.exists(COMFY_MODELS_DIR):
|
||||
for root, _, files in os.walk(COMFY_MODELS_DIR):
|
||||
for file in files:
|
||||
if not file.endswith(extensions):
|
||||
continue
|
||||
full_path = os.path.join(root, file)
|
||||
try:
|
||||
st = os.stat(full_path)
|
||||
except OSError:
|
||||
continue
|
||||
results.append({
|
||||
"filename": file,
|
||||
"rel_path": os.path.relpath(full_path, COMFY_MODELS_DIR),
|
||||
"full_path": full_path,
|
||||
"category": os.path.relpath(root, COMFY_MODELS_DIR).split(os.sep)[0],
|
||||
"size_bytes": st.st_size,
|
||||
"size_mb": round(st.st_size / (1024**2), 2),
|
||||
"size_gb": round(st.st_size / (1024**3), 3),
|
||||
"mtime": st.st_mtime,
|
||||
"kind": "comfy",
|
||||
})
|
||||
_catalog_cache.update({"ts": now, "sig": sig, "comfy": results,
|
||||
"ollama": find_ollama_model_files()})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- residency report
|
||||
|
||||
_report_cache: Dict[str, Any] = {"ts": 0.0, "report": None}
|
||||
REPORT_TTL_S = 15.0
|
||||
|
||||
|
||||
def get_cache_report(include_files: bool = True, force_refresh: bool = False) -> Dict[str, Any]:
|
||||
"""Measured page-cache residency across the whole model catalog.
|
||||
|
||||
Deduplicated by blob path: several Ollama tags routinely point at the same GGUF, and
|
||||
counting each tag separately produced more "resident" bytes than the box has RAM.
|
||||
"""
|
||||
now = time.time()
|
||||
cached = _report_cache["report"]
|
||||
if cached and not force_refresh and (now - _report_cache["ts"]) < REPORT_TTL_S:
|
||||
return cached if include_files else {**cached, "files": []}
|
||||
|
||||
t0 = time.perf_counter()
|
||||
catalog = get_model_catalog()
|
||||
by_path: Dict[str, Dict[str, Any]] = {}
|
||||
for f in list(catalog["ollama"]) + list(catalog["comfy"]):
|
||||
path = f["full_path"]
|
||||
name = f.get("model") or f.get("rel_path") or f.get("filename")
|
||||
if path in by_path:
|
||||
by_path[path]["aliases"].append(name)
|
||||
continue
|
||||
by_path[path] = {"entry": f, "name": name, "aliases": []}
|
||||
|
||||
entries = []
|
||||
total_bytes = resident_bytes = 0
|
||||
for path, meta in by_path.items():
|
||||
f = meta["entry"]
|
||||
res = page_residency(path)
|
||||
size = f.get("size_bytes") or res.get("size_bytes") or 0
|
||||
rb = res.get("resident_bytes", 0)
|
||||
total_bytes += size
|
||||
resident_bytes += rb
|
||||
entries.append({
|
||||
"name": meta["name"],
|
||||
"aliases": meta["aliases"],
|
||||
"kind": f.get("kind"),
|
||||
"full_path": path,
|
||||
"size_gb": round(size / (1024**3), 3),
|
||||
"resident_pct": res.get("resident_pct", 0.0),
|
||||
"resident_gb": round(rb / (1024**3), 3),
|
||||
"warm": res.get("warm", False),
|
||||
})
|
||||
entries.sort(key=lambda e: e["resident_gb"], reverse=True)
|
||||
|
||||
report = {
|
||||
"scan_ms": round((time.perf_counter() - t0) * 1000, 1),
|
||||
"files_scanned": len(entries),
|
||||
"unique_blobs": len(by_path),
|
||||
"catalog_total_gb": round(total_bytes / (1024**3), 2),
|
||||
"resident_total_gb": round(resident_bytes / (1024**3), 2),
|
||||
"residency_pct": round((resident_bytes / total_bytes * 100) if total_bytes else 0, 1),
|
||||
"warm_files": sum(1 for e in entries if e["warm"]),
|
||||
"files": entries,
|
||||
}
|
||||
_report_cache.update({"ts": now, "report": report})
|
||||
return report if include_files else {**report, "files": []}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- warming
|
||||
|
||||
def warm_file_to_ram(filepath: str, chunk_size: int = 16 * 1024 * 1024,
|
||||
skip_if_warm: bool = True, force: bool = False) -> Dict[str, Any]:
|
||||
"""Pre-fault a file into the Linux page cache, skipping it only if confidently resident."""
|
||||
if not os.path.exists(filepath):
|
||||
return {"success": False, "error": f"File not found: {filepath}", "duration_ms": 0}
|
||||
|
||||
# Probe densely here: this decision skips real work, so it is worth 32 samples
|
||||
# rather than 12.
|
||||
before = page_residency(filepath, probe_windows=32)
|
||||
if skip_if_warm and not force and before.get("warm_confident"):
|
||||
return {
|
||||
"success": True, "filepath": filepath, "skipped": True,
|
||||
"reason": "already resident", "resident_pct": before.get("resident_pct"),
|
||||
"method": before.get("method"),
|
||||
"size_mb": round(before.get("size_bytes", 0) / (1024**2), 2),
|
||||
"duration_ms": 0.0, "bytes_read": 0,
|
||||
}
|
||||
|
||||
t0 = time.perf_counter()
|
||||
file_size = os.path.getsize(filepath)
|
||||
bytes_read = 0
|
||||
try:
|
||||
with open(filepath, "rb") as f:
|
||||
# Hint kernel that we will read this sequentially
|
||||
try:
|
||||
os.posix_fadvise(f.fileno(), 0, file_size, os.POSIX_FADV_WILLNEED)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
buf = bytearray(chunk_size)
|
||||
while True:
|
||||
n = f.readinto(buf)
|
||||
@@ -75,19 +508,23 @@ def warm_file_to_ram(filepath: str, chunk_size: int = 16 * 1024 * 1024) -> Dict[
|
||||
bytes_read += n
|
||||
|
||||
duration = time.perf_counter() - t0
|
||||
duration_ms = round(duration * 1000, 2)
|
||||
speed_mb_s = round((bytes_read / (1024**2)) / duration if duration > 0 else 0, 2)
|
||||
after = page_residency(filepath, probe_windows=32)
|
||||
return {
|
||||
"success": True,
|
||||
"filepath": filepath,
|
||||
"skipped": False,
|
||||
"size_bytes": file_size,
|
||||
"size_mb": round(file_size / (1024**2), 2),
|
||||
"bytes_read": bytes_read,
|
||||
"duration_ms": duration_ms,
|
||||
"speed_mb_s": speed_mb_s,
|
||||
"duration_ms": round(duration * 1000, 2),
|
||||
"speed_mb_s": round((bytes_read / (1024**2)) / duration if duration > 0 else 0, 2),
|
||||
"resident_pct_before": before.get("resident_pct", 0.0),
|
||||
"resident_pct_after": after.get("resident_pct", 0.0),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
return {"success": False, "error": str(e),
|
||||
"duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
|
||||
async def warm_ollama_model(model_name: str, keep_alive: str = "5m") -> Dict[str, Any]:
|
||||
"""Warm an Ollama model into memory and measure time."""
|
||||
@@ -101,76 +538,148 @@ async def warm_ollama_model(model_name: str, keep_alive: str = "5m") -> Dict[str
|
||||
duration = time.perf_counter() - t0
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
return {
|
||||
res = {
|
||||
"success": True,
|
||||
"model": model_name,
|
||||
"duration_ms": round(duration * 1000, 2),
|
||||
"load_duration_ms": round(data.get("load_duration", 0) / 1e6, 2),
|
||||
"total_duration_ms": round(data.get("total_duration", 0) / 1e6, 2),
|
||||
}
|
||||
else:
|
||||
telemetry_store.record_event({
|
||||
"event_type": "Model Warm", "source": "warmer", "target": model_name,
|
||||
"duration_ms": res["duration_ms"], "load_duration_ms": res["load_duration_ms"],
|
||||
})
|
||||
return res
|
||||
return {
|
||||
"success": False,
|
||||
"model": model_name,
|
||||
"success": False, "model": model_name,
|
||||
"error": f"HTTP {resp.status_code}: {resp.text}",
|
||||
"duration_ms": round(duration * 1000, 2),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"success": False, "model": model_name, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
return {"success": False, "model": model_name, "error": str(e),
|
||||
"duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
def find_comfy_model_files() -> List[Dict[str, Any]]:
|
||||
"""Discover all model files under ComfyUI models."""
|
||||
results = []
|
||||
extensions = ("*.safetensors", "*.ckpt", "*.pt", "*.bin")
|
||||
if os.path.exists(COMFY_MODELS_DIR):
|
||||
for root, _, files in os.walk(COMFY_MODELS_DIR):
|
||||
for file in files:
|
||||
if any(file.endswith(ext.replace("*", "")) for ext in extensions):
|
||||
full_path = os.path.join(root, file)
|
||||
rel_path = os.path.relpath(full_path, COMFY_MODELS_DIR)
|
||||
size = os.path.getsize(full_path)
|
||||
results.append({
|
||||
"filename": file,
|
||||
"rel_path": rel_path,
|
||||
"full_path": full_path,
|
||||
"size_bytes": size,
|
||||
"size_mb": round(size / (1024**2), 2),
|
||||
"size_gb": round(size / (1024**3), 3),
|
||||
})
|
||||
return results
|
||||
|
||||
async def warm_all_models() -> Dict[str, Any]:
|
||||
"""Warm all available Ollama and ComfyUI models into Linux RAM Cache."""
|
||||
def warm_ollama_blob(model_name: str, force: bool = False) -> Dict[str, Any]:
|
||||
"""Warm a specific Ollama model's GGUF into page cache without touching VRAM."""
|
||||
for f in find_ollama_model_files():
|
||||
if f["model"] == model_name:
|
||||
res = warm_file_to_ram(f["full_path"], force=force)
|
||||
res["model"] = model_name
|
||||
return res
|
||||
return {"success": False, "error": f"no blob found for model '{model_name}'"}
|
||||
|
||||
|
||||
def _warm_priority(days: float = 30.0) -> Dict[str, float]:
|
||||
"""Recency/frequency score per model name, from the persisted event log."""
|
||||
try:
|
||||
return {r["model"]: r["score"] for r in telemetry_store.model_usage_ranking(days)}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def build_warm_plan(budget_gb: Optional[float] = None) -> Dict[str, Any]:
|
||||
"""Decide *what* to warm, in what order, within a byte budget.
|
||||
|
||||
Warming everything on a 64 GB box with 33+ GB of models just evicts the earliest
|
||||
files, so we rank by usage (Ollama, from history) and recency (ComfyUI, by mtime),
|
||||
then fill until the budget is spent. Already-resident files cost nothing.
|
||||
"""
|
||||
mem = get_detailed_meminfo()
|
||||
if budget_gb is None:
|
||||
# Leave headroom so warming never pushes the box into reclaim.
|
||||
budget_gb = max((mem["available_bytes"] * 0.7) / (1024**3), 1.0)
|
||||
budget_bytes = int(budget_gb * (1024**3))
|
||||
|
||||
catalog = get_model_catalog()
|
||||
scores = _warm_priority()
|
||||
now = time.time()
|
||||
|
||||
candidates = []
|
||||
for f in catalog["ollama"]:
|
||||
candidates.append({**f, "score": scores.get(f["model"], 0.0) + 0.5,
|
||||
"name": f["model"]})
|
||||
for f in catalog["comfy"]:
|
||||
age_days = max((now - f.get("mtime", now)) / 86400.0, 0.01)
|
||||
candidates.append({**f, "score": scores.get(f["rel_path"], 0.0) + 1.0 / (1.0 + age_days),
|
||||
"name": f["rel_path"]})
|
||||
|
||||
candidates.sort(key=lambda c: c["score"], reverse=True)
|
||||
|
||||
plan, spent, skipped = [], 0, []
|
||||
seen_paths = set()
|
||||
for c in candidates:
|
||||
if c["full_path"] in seen_paths:
|
||||
continue
|
||||
seen_paths.add(c["full_path"])
|
||||
res = page_residency(c["full_path"], probe_windows=32)
|
||||
entry = {
|
||||
"name": c["name"], "kind": c["kind"], "full_path": c["full_path"],
|
||||
"size_gb": c.get("size_gb", 0), "score": round(c["score"], 4),
|
||||
"resident_pct": res.get("resident_pct", 0.0),
|
||||
}
|
||||
if res.get("warm_confident"):
|
||||
entry["action"] = "already-warm"
|
||||
skipped.append(entry)
|
||||
continue
|
||||
need = int(c.get("size_bytes", 0) * (1 - res.get("resident_pct", 0) / 100.0))
|
||||
if spent + need > budget_bytes:
|
||||
entry["action"] = "over-budget"
|
||||
skipped.append(entry)
|
||||
continue
|
||||
spent += need
|
||||
entry["action"] = "warm"
|
||||
entry["bytes_to_read"] = need
|
||||
plan.append(entry)
|
||||
|
||||
return {
|
||||
"budget_gb": round(budget_gb, 2),
|
||||
"planned_gb": round(spent / (1024**3), 2),
|
||||
"warm_count": len(plan),
|
||||
"skipped_count": len(skipped),
|
||||
"plan": plan,
|
||||
"skipped": skipped,
|
||||
"meminfo": mem,
|
||||
}
|
||||
|
||||
|
||||
async def warm_all_models(budget_gb: Optional[float] = None,
|
||||
include_vram_load: bool = False) -> Dict[str, Any]:
|
||||
"""Warm the highest-value models into the page cache within a byte budget."""
|
||||
t0 = time.perf_counter()
|
||||
warmed_ollama = []
|
||||
warmed_comfy = []
|
||||
plan = build_warm_plan(budget_gb)
|
||||
warmed = []
|
||||
for entry in plan["plan"]:
|
||||
res = warm_file_to_ram(entry["full_path"])
|
||||
res["name"] = entry["name"]
|
||||
res["kind"] = entry["kind"]
|
||||
warmed.append(res)
|
||||
# Budgets are computed up front, but the page cache is shared with the rest of
|
||||
# the box; bail out if we start pushing the system into reclaim.
|
||||
if get_detailed_meminfo()["available_gb"] < 4.0:
|
||||
logger.warning("warm_all_models: stopping early, MemAvailable below 4 GB")
|
||||
break
|
||||
|
||||
# 1. Ollama models
|
||||
if include_vram_load:
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
tags_resp = await client.get(f"{OLLAMA_API_BASE}/api/tags")
|
||||
if tags_resp.status_code == 200:
|
||||
models = tags_resp.json().get("models", [])
|
||||
for m in models:
|
||||
name = m.get("name")
|
||||
res = await warm_ollama_model(name, keep_alive="1m")
|
||||
warmed_ollama.append(res)
|
||||
tags = await client.get(f"{OLLAMA_API_BASE}/api/tags")
|
||||
if tags.status_code == 200:
|
||||
top = sorted(tags.json().get("models", []),
|
||||
key=lambda m: _warm_priority().get(m.get("name"), 0),
|
||||
reverse=True)[:1]
|
||||
for m in top:
|
||||
await warm_ollama_model(m.get("name"), keep_alive="1m")
|
||||
except Exception as e:
|
||||
logger.error(f"Error discovering Ollama models: {e}")
|
||||
|
||||
# 2. ComfyUI models
|
||||
comfy_files = find_comfy_model_files()
|
||||
for f in comfy_files:
|
||||
res = warm_file_to_ram(f["full_path"])
|
||||
warmed_comfy.append(res)
|
||||
|
||||
total_duration_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
meminfo = get_detailed_meminfo()
|
||||
logger.debug(f"optional VRAM preload skipped: {e}")
|
||||
|
||||
return {
|
||||
"status": "completed",
|
||||
"total_duration_ms": total_duration_ms,
|
||||
"ollama_models_warmed": warmed_ollama,
|
||||
"comfy_files_warmed": warmed_comfy,
|
||||
"meminfo_after": meminfo,
|
||||
"total_duration_ms": round((time.perf_counter() - t0) * 1000, 2),
|
||||
"budget_gb": plan["budget_gb"],
|
||||
"planned_gb": plan["planned_gb"],
|
||||
"files_warmed": warmed,
|
||||
"bytes_read": sum(w.get("bytes_read", 0) for w in warmed),
|
||||
"skipped": plan["skipped"],
|
||||
"meminfo_after": get_detailed_meminfo(),
|
||||
}
|
||||
|
||||
446
server.py
446
server.py
@@ -1,27 +1,211 @@
|
||||
"""FastAPI Backend Server with SSE Real-Time Telemetry and Model Orchestration API."""
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import logging
|
||||
from typing import Dict, Any, Optional, List
|
||||
import signal
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import Dict, Any, Optional, List, Set
|
||||
|
||||
from fastapi import FastAPI, Request, HTTPException, Query
|
||||
from fastapi.responses import HTMLResponse, StreamingResponse, JSONResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
import ram_optimizer
|
||||
import vram_arbitrator
|
||||
import autotune
|
||||
import overclock_manager
|
||||
import ram_optimizer
|
||||
import telemetry_store
|
||||
import thermal_governor
|
||||
import vram_arbitrator
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s")
|
||||
logger = logging.getLogger("model_manager_server")
|
||||
|
||||
# The sampler makes four HTTP calls a second; at INFO, httpx narrates every one of them.
|
||||
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||
logging.getLogger("httpcore").setLevel(logging.WARNING)
|
||||
|
||||
BASE_DIR = "/home/drjones/unified-model-manager"
|
||||
|
||||
|
||||
# ==========================================
|
||||
# TELEMETRY BROKER
|
||||
# ==========================================
|
||||
|
||||
class TelemetryBroker:
|
||||
"""One sampler, many subscribers.
|
||||
|
||||
Every SSE client used to run its own copy of the full snapshot once per second:
|
||||
NVML queries, /proc/meminfo, an HTTP round-trip each to Ollama and ComfyUI, and — the
|
||||
expensive one — a recursive walk of the ComfyUI models tree with a stat() per
|
||||
checkpoint. Opening the dashboard in three tabs tripled the load on the very thing it
|
||||
was measuring. Now a single background task samples at 1 Hz and fans the snapshot out.
|
||||
|
||||
The sampler is also the natural feed for the thermal governor and the persistence
|
||||
layer, so neither needs to poll the GPU on its own.
|
||||
"""
|
||||
|
||||
def __init__(self, interval_s: float = 1.0) -> None:
|
||||
self.interval_s = interval_s
|
||||
self.snapshot: Dict[str, Any] = {}
|
||||
self.subscribers: Set[asyncio.Queue] = set()
|
||||
self.task: Optional[asyncio.Task] = None
|
||||
self.running = False
|
||||
self.samples = 0
|
||||
self.last_sample_ms = 0.0
|
||||
# Set on shutdown so open SSE generators finish instead of holding the server up.
|
||||
self.closing = False
|
||||
|
||||
async def start(self) -> None:
|
||||
if self.running:
|
||||
return
|
||||
self.running = True
|
||||
self.task = asyncio.create_task(self._loop())
|
||||
|
||||
def begin_shutdown(self) -> None:
|
||||
"""Release every SSE subscriber. Safe to call from a signal handler."""
|
||||
self.closing = True
|
||||
for q in list(self.subscribers):
|
||||
with contextlib.suppress(asyncio.QueueFull):
|
||||
q.put_nowait(None)
|
||||
|
||||
async def stop(self) -> None:
|
||||
self.running = False
|
||||
self.closing = True
|
||||
# Wake every subscriber so their generator can return. Without this, uvicorn waits
|
||||
# on the open SSE responses during graceful shutdown and systemd eventually
|
||||
# SIGKILLs the unit -- which skips the in-process GPU restore hook entirely.
|
||||
for q in list(self.subscribers):
|
||||
with contextlib.suppress(asyncio.QueueFull):
|
||||
q.put_nowait(None)
|
||||
if self.task:
|
||||
self.task.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await self.task
|
||||
|
||||
def subscribe(self) -> asyncio.Queue:
|
||||
q: asyncio.Queue = asyncio.Queue(maxsize=2)
|
||||
self.subscribers.add(q)
|
||||
return q
|
||||
|
||||
def unsubscribe(self, q: asyncio.Queue) -> None:
|
||||
self.subscribers.discard(q)
|
||||
|
||||
async def _loop(self) -> None:
|
||||
while self.running:
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
snap = await self._sample()
|
||||
self.snapshot = snap
|
||||
self.samples += 1
|
||||
self.last_sample_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
|
||||
# Feed the governor and the durable store from the sample we already have.
|
||||
thermal_governor.governor.observe(snap.get("gpu", {}),
|
||||
overclock_manager.ACTIVE_PROFILE)
|
||||
telemetry_store.record_telemetry(
|
||||
snap.get("gpu", {}), snap.get("ram", {}),
|
||||
profile=overclock_manager.ACTIVE_PROFILE,
|
||||
throttle_reasons=",".join(snap.get("gpu", {}).get("throttle_reasons") or []),
|
||||
)
|
||||
|
||||
for q in list(self.subscribers):
|
||||
if q.full():
|
||||
# Slow client: drop the stale frame rather than stalling the sampler.
|
||||
with contextlib.suppress(asyncio.QueueEmpty):
|
||||
q.get_nowait()
|
||||
with contextlib.suppress(asyncio.QueueFull):
|
||||
q.put_nowait(snap)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"telemetry sampler error: {e}")
|
||||
await asyncio.sleep(max(self.interval_s - (time.perf_counter() - t0), 0.05))
|
||||
|
||||
async def _sample(self) -> Dict[str, Any]:
|
||||
gpu_stats = vram_arbitrator.get_gpu_hardware_stats()
|
||||
mem_stats = ram_optimizer.get_detailed_meminfo()
|
||||
ollama_state, comfy_state = await asyncio.gather(
|
||||
vram_arbitrator.get_ollama_live_state(),
|
||||
vram_arbitrator.get_comfyui_live_state(),
|
||||
)
|
||||
return {
|
||||
"timestamp": time.time(), # wall clock, not the event loop's monotonic clock
|
||||
"monotonic": asyncio.get_running_loop().time(),
|
||||
"gpu": gpu_stats,
|
||||
"ram": mem_stats,
|
||||
"ollama": ollama_state,
|
||||
"comfyui": comfy_state,
|
||||
"arbitrator": vram_arbitrator.arbitrator.get_status(),
|
||||
"governor": thermal_governor.governor.get_status(),
|
||||
"overclock": {"active_profile": overclock_manager.ACTIVE_PROFILE},
|
||||
"history": vram_arbitrator.get_switch_history(),
|
||||
"comfy_models_count": len(ram_optimizer.find_comfy_model_files()),
|
||||
"sampler": {"samples": self.samples, "last_sample_ms": self.last_sample_ms,
|
||||
"subscribers": len(self.subscribers)},
|
||||
}
|
||||
|
||||
async def get(self) -> Dict[str, Any]:
|
||||
"""Latest snapshot, sampling on demand if the loop has not produced one yet."""
|
||||
if not self.snapshot:
|
||||
self.snapshot = await self._sample()
|
||||
return self.snapshot
|
||||
|
||||
|
||||
broker = TelemetryBroker()
|
||||
|
||||
|
||||
def _install_shutdown_hook() -> None:
|
||||
"""Close SSE streams the moment a shutdown signal arrives.
|
||||
|
||||
uvicorn runs the lifespan shutdown only after it has finished waiting on open
|
||||
connections, so releasing subscribers from there is too late: the streams keep the
|
||||
server busy until the graceful timeout expires and every one of them is force
|
||||
cancelled, which logs a CancelledError traceback apiece. Chaining onto the existing
|
||||
signal handler lets us drain them first and leaves uvicorn's own shutdown intact.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
previous = signal.getsignal(sig)
|
||||
|
||||
def handler(signum, frame, _prev=previous):
|
||||
broker.begin_shutdown()
|
||||
if callable(_prev):
|
||||
_prev(signum, frame)
|
||||
|
||||
try:
|
||||
signal.signal(sig, handler)
|
||||
except (ValueError, OSError):
|
||||
pass # not on the main thread; the lifespan path still cleans up
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
telemetry_store.start()
|
||||
await broker.start()
|
||||
await vram_arbitrator.arbitrator.start()
|
||||
_install_shutdown_hook()
|
||||
yield
|
||||
await vram_arbitrator.arbitrator.stop()
|
||||
await broker.stop()
|
||||
# Never leave the card with locked clocks and pinned fans after we exit.
|
||||
try:
|
||||
overclock_manager.restore_safe("server shutdown")
|
||||
except Exception as e:
|
||||
logger.error(f"restore_safe on shutdown failed: {e}")
|
||||
telemetry_store.stop()
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="HyperSwap // GPU Program Swapper & Telemetry API",
|
||||
version="1.0.0",
|
||||
version="2.0.0",
|
||||
description="High-performance VRAM arbitration and 64GB RAM cache orchestrator for simultaneous Ollama and ComfyUI workloads on Linux.",
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
app.add_middleware(
|
||||
@@ -32,26 +216,25 @@ app.add_middleware(
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
@app.on_event("startup")
|
||||
async def on_startup():
|
||||
await vram_arbitrator.arbitrator.start()
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def on_shutdown():
|
||||
await vram_arbitrator.arbitrator.stop()
|
||||
|
||||
# Pydantic Request Models
|
||||
class SwitchRequest(BaseModel):
|
||||
model: str = Field(..., description="Name of the Ollama model to hot-swap to in VRAM", example="qwen3.8fast:latest")
|
||||
keep_alive: Optional[str] = Field("30m", description="Keep-alive duration in VRAM (e.g. 5m, 30m, 0)", example="30m")
|
||||
free_comfy_first: bool = Field(False, description="Purge ComfyUI VRAM first if it is holding memory")
|
||||
|
||||
class WarmRequest(BaseModel):
|
||||
model_name: Optional[str] = Field(None, description="Ollama model name to warm into OS page cache", example="gemma4:26b")
|
||||
filepath: Optional[str] = Field(None, description="Absolute file path of Safetensors/GGUF to warm into RAM", example="/home/drjones/ComfyUI/models/checkpoints/v1-5-pruned-emaonly-fp16.safetensors")
|
||||
model_name: Optional[str] = Field(None, description="Ollama model name to warm", example="gemma4:26b")
|
||||
filepath: Optional[str] = Field(None, description="Absolute file path of Safetensors/GGUF to warm into RAM")
|
||||
blob_only: bool = Field(False, description="Warm the model's weights into page cache without loading VRAM")
|
||||
force: bool = Field(False, description="Warm even if residency sampling thinks it is already resident")
|
||||
|
||||
class WarmAllRequest(BaseModel):
|
||||
budget_gb: Optional[float] = Field(None, description="Byte budget for warming; defaults to 70% of MemAvailable", example=24.0)
|
||||
|
||||
class BenchmarkRequest(BaseModel):
|
||||
iterations: Optional[int] = Field(2, description="Number of back-and-forth switch iterations to measure", example=2)
|
||||
models: Optional[List[str]] = Field(None, description="Optional pair of models to benchmark between", example=["qwen3.8fast:latest", "smtek/Qwen3.8-27B:Q2_K_XL"])
|
||||
models: Optional[List[str]] = Field(None, description="Optional pair of models to benchmark between")
|
||||
|
||||
class OverclockApplyRequest(BaseModel):
|
||||
profile: str = Field(..., description="Profile name: ollama | comfy | balanced", example="ollama")
|
||||
@@ -64,6 +247,26 @@ class FanRequest(BaseModel):
|
||||
percent: Optional[int] = Field(None, description="Fan speed 30-100 when mode=manual", example=70)
|
||||
speed_pct: Optional[int] = Field(None, description="Alias for percent (30-100)", example=70)
|
||||
|
||||
class GovernorRequest(BaseModel):
|
||||
enabled: Optional[bool] = Field(None, description="Enable or disable the thermal governor")
|
||||
reset: bool = Field(False, description="Clear any active derate and reapply the full profile")
|
||||
|
||||
class SweepRequest(BaseModel):
|
||||
knob: str = Field("mem_offset_mhz", description="mem_offset_mhz | core_offset_mhz | lock_mem_mhz | lock_core_max")
|
||||
profile: str = Field("ollama", description="Profile to tune")
|
||||
workload: str = Field("auto", description="ollama (decode tok/s) | comfy (diffusion it/s) | auto")
|
||||
model: Optional[str] = Field(None, description="Model to benchmark with; defaults to the loaded one")
|
||||
start: Optional[int] = Field(None, description="First offset value")
|
||||
stop: Optional[int] = Field(None, description="Last offset value")
|
||||
step: Optional[int] = Field(None, description="Offset increment")
|
||||
repeats: int = Field(1, description="Benchmark runs per step")
|
||||
max_steps: int = Field(6, description="Cap on swept values for discrete clock knobs")
|
||||
include_unlocked: bool = Field(True, description="Include an unlocked (0) control step")
|
||||
apply_best: bool = Field(False, description="Write the winning value into the profile")
|
||||
|
||||
class RequestVramRequest(BaseModel):
|
||||
needed_gb: float = Field(0.0, description="How much free VRAM Ollama needs", example=12.0)
|
||||
|
||||
|
||||
# ==========================================
|
||||
# REST API ENDPOINTS
|
||||
@@ -71,51 +274,45 @@ class FanRequest(BaseModel):
|
||||
|
||||
@app.get("/api/stats", summary="Full System Snapshot", tags=["Telemetry"])
|
||||
async def get_all_stats() -> Dict[str, Any]:
|
||||
"""Gather complete live snapshot of GPU hardware, host RAM, Ollama, ComfyUI, and switch history."""
|
||||
gpu_stats = vram_arbitrator.get_gpu_hardware_stats()
|
||||
mem_stats = ram_optimizer.get_detailed_meminfo()
|
||||
ollama_state = await vram_arbitrator.get_ollama_live_state()
|
||||
comfy_state = await vram_arbitrator.get_comfyui_live_state()
|
||||
history = vram_arbitrator.get_switch_history()
|
||||
comfy_models = ram_optimizer.find_comfy_model_files()
|
||||
arbitrator_status = vram_arbitrator.arbitrator.get_status()
|
||||
|
||||
return {
|
||||
"timestamp": asyncio.get_event_loop().time(),
|
||||
"gpu": gpu_stats,
|
||||
"ram": mem_stats,
|
||||
"ollama": ollama_state,
|
||||
"comfyui": comfy_state,
|
||||
"arbitrator": arbitrator_status,
|
||||
"history": history,
|
||||
"comfy_models_count": len(comfy_models),
|
||||
}
|
||||
"""Latest unified snapshot of GPU hardware, host RAM, Ollama, ComfyUI and swap history."""
|
||||
return await broker.get()
|
||||
|
||||
|
||||
@app.get("/api/gpu", summary="GPU Sensors and VRAM Breakdown", tags=["Telemetry"])
|
||||
async def get_gpu_metrics() -> Dict[str, Any]:
|
||||
"""Retrieve detailed NVML sensors (utilization %, temp, power, fan, clocks, and per-process VRAM allocation)."""
|
||||
"""Detailed NVML sensors (utilization, temp, power, fan, clocks, throttle reasons, per-process VRAM)."""
|
||||
return vram_arbitrator.get_gpu_hardware_stats()
|
||||
|
||||
@app.get("/api/memory", summary="Host RAM and Page Cache Breakdown", tags=["Telemetry"])
|
||||
async def get_ram_metrics() -> Dict[str, Any]:
|
||||
"""Retrieve precise host 64GB DDR5 RAM breakdown, active cache size, and cache hit ratios."""
|
||||
"""Precise host RAM breakdown, active cache size and cache ratios."""
|
||||
return ram_optimizer.get_detailed_meminfo()
|
||||
|
||||
@app.get("/api/stream", summary="Real-Time SSE Telemetry Stream", tags=["Telemetry"])
|
||||
async def sse_telemetry_stream(request: Request):
|
||||
"""Server-Sent Events (SSE) streaming real-time statistics at 1Hz for dynamic dashboards."""
|
||||
"""Server-Sent Events stream of the shared 1Hz snapshot."""
|
||||
async def event_generator():
|
||||
while True:
|
||||
q = broker.subscribe()
|
||||
try:
|
||||
snap = await broker.get()
|
||||
yield f"data: {json.dumps(snap)}\n\n"
|
||||
while not broker.closing:
|
||||
if await request.is_disconnected():
|
||||
break
|
||||
try:
|
||||
stats = await get_all_stats()
|
||||
yield f"data: {json.dumps(stats)}\n\n"
|
||||
snap = await asyncio.wait_for(q.get(), timeout=5.0)
|
||||
if snap is None: # shutdown sentinel
|
||||
break
|
||||
yield f"data: {json.dumps(snap)}\n\n"
|
||||
except asyncio.TimeoutError:
|
||||
yield ": keepalive\n\n"
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"SSE stream error: {e}")
|
||||
yield f"data: {json.dumps({'error': str(e)})}\n\n"
|
||||
await asyncio.sleep(1.0)
|
||||
finally:
|
||||
broker.unsubscribe(q)
|
||||
|
||||
return StreamingResponse(
|
||||
event_generator(),
|
||||
@@ -129,80 +326,142 @@ async def sse_telemetry_stream(request: Request):
|
||||
|
||||
@app.post("/api/switch-model", summary="Hot-Swap Ollama LLM in VRAM", tags=["Orchestration"])
|
||||
async def api_switch_model(req: SwitchRequest):
|
||||
"""Hot-swap the active Ollama model in VRAM and measure exact load duration and token evaluation speed."""
|
||||
"""Hot-swap the active Ollama model, measuring real load bandwidth and token throughput."""
|
||||
if req.free_comfy_first:
|
||||
await vram_arbitrator.arbitrator.request_vram_for_ollama()
|
||||
res = await vram_arbitrator.switch_ollama_model(req.model, keep_alive=req.keep_alive or "30m")
|
||||
if not res.get("success"):
|
||||
raise HTTPException(status_code=500, detail=res.get("error"))
|
||||
return res
|
||||
|
||||
@app.post("/api/free-vram", summary="Soft-Yield Ollama VRAM", tags=["Orchestration"])
|
||||
async def api_free_vram():
|
||||
"""Instruct Ollama to instantly yield VRAM to 0MB in ~15ms while preserving model weights in the 64GB host RAM page cache."""
|
||||
return await vram_arbitrator.instant_free_ollama_vram()
|
||||
async def api_free_vram(confirm: bool = Query(True, description="Wait for the driver to actually release the allocation")):
|
||||
"""Yield Ollama's VRAM and wait for the release to be confirmed by NVML."""
|
||||
return await vram_arbitrator.instant_free_ollama_vram(confirm=confirm)
|
||||
|
||||
@app.post("/api/comfy-free", summary="Purge ComfyUI VRAM Cache", tags=["Orchestration"])
|
||||
async def api_comfy_free():
|
||||
"""Purge loaded diffusion models and VRAM cache from the ComfyUI pipeline."""
|
||||
return await vram_arbitrator.instant_free_comfyui_vram()
|
||||
|
||||
@app.post("/api/warm-all", summary="Pre-warm All Models into RAM Cache", tags=["Memory Optimization"])
|
||||
async def api_warm_all():
|
||||
"""Pre-fault and read all installed Ollama GGUF models and ComfyUI Safetensors checkpoints into the Linux OS Page Cache."""
|
||||
return await ram_optimizer.warm_all_models()
|
||||
@app.post("/api/request-vram", summary="Ask for VRAM on Ollama's behalf", tags=["Orchestration"])
|
||||
async def api_request_vram(req: RequestVramRequest):
|
||||
"""Force an immediate ComfyUI purge if there is not enough free VRAM for Ollama."""
|
||||
return await vram_arbitrator.arbitrator.request_vram_for_ollama(req.needed_gb)
|
||||
|
||||
@app.post("/api/warm-all", summary="Pre-warm Models into RAM Cache", tags=["Memory Optimization"])
|
||||
async def api_warm_all(req: Optional[WarmAllRequest] = None):
|
||||
"""Warm the highest-value models into the page cache within a byte budget."""
|
||||
return await ram_optimizer.warm_all_models(budget_gb=req.budget_gb if req else None)
|
||||
|
||||
@app.get("/api/warm-plan", summary="Preview the Warm Plan", tags=["Memory Optimization"])
|
||||
async def api_warm_plan(budget_gb: Optional[float] = Query(None, description="Override the byte budget")):
|
||||
"""Show what warming would read, in what order, and what it would skip — without doing it."""
|
||||
return ram_optimizer.build_warm_plan(budget_gb)
|
||||
|
||||
@app.post("/api/warm-model", summary="Pre-warm Single Model or File", tags=["Memory Optimization"])
|
||||
async def api_warm_model(req: WarmRequest):
|
||||
"""Pre-warm a specific Ollama model or individual file path into Linux RAM cache."""
|
||||
"""Pre-warm a specific Ollama model or file path into the Linux page cache."""
|
||||
if req.model_name and req.blob_only:
|
||||
return ram_optimizer.warm_ollama_blob(req.model_name, force=req.force)
|
||||
if req.model_name:
|
||||
return await ram_optimizer.warm_ollama_model(req.model_name, keep_alive="1m")
|
||||
elif req.filepath:
|
||||
return ram_optimizer.warm_file_to_ram(req.filepath)
|
||||
else:
|
||||
if req.filepath:
|
||||
return ram_optimizer.warm_file_to_ram(req.filepath, force=req.force)
|
||||
raise HTTPException(status_code=400, detail="model_name or filepath required")
|
||||
|
||||
@app.get("/api/cache/report", summary="Measured Page-Cache Residency", tags=["Memory Optimization"])
|
||||
async def api_cache_report(files: bool = Query(True), refresh: bool = Query(False)):
|
||||
"""Measured (not assumed) page-cache residency for every model on disk."""
|
||||
report = ram_optimizer.get_cache_report(include_files=files, force_refresh=refresh)
|
||||
report["capability"] = ram_optimizer.residency_capability()
|
||||
return report
|
||||
|
||||
@app.get("/api/models", summary="List All Installed Models", tags=["Catalog"])
|
||||
async def api_get_models():
|
||||
"""List all installed Ollama models and discovered ComfyUI model checkpoints/safetensors on disk with sizes and quantization levels."""
|
||||
async def api_get_models(refresh: bool = Query(False)):
|
||||
"""All installed Ollama models (with their on-disk blobs) and ComfyUI checkpoints."""
|
||||
catalog = ram_optimizer.get_model_catalog(force_refresh=refresh)
|
||||
ollama_state = await vram_arbitrator.get_ollama_live_state()
|
||||
comfy_models = ram_optimizer.find_comfy_model_files()
|
||||
return {
|
||||
"ollama_models": ollama_state.get("installed_models", []),
|
||||
"comfy_models": comfy_models,
|
||||
"ollama_blobs": catalog["ollama"],
|
||||
"comfy_models": catalog["comfy"],
|
||||
"cached_at": catalog["cached_at"],
|
||||
}
|
||||
|
||||
@app.get("/api/history", summary="Model Switch History Log", tags=["Analytics"])
|
||||
async def api_get_history(limit: int = Query(20, description="Max history items to return")):
|
||||
"""Get the recent history log of model switch events, swap durations (in ms), and RAM cache hit status."""
|
||||
history = vram_arbitrator.get_switch_history()
|
||||
return history[:limit]
|
||||
async def api_get_history(limit: int = Query(20, description="Max history items to return"),
|
||||
durable: bool = Query(False, description="Read from the persistent store instead of the in-memory ring")):
|
||||
"""Recent swap events, durations, achieved bandwidth and cache status."""
|
||||
if durable:
|
||||
return telemetry_store.recent_events(limit)
|
||||
return vram_arbitrator.get_switch_history()[:limit]
|
||||
|
||||
@app.post("/api/benchmark", summary="Run Latency Benchmark", tags=["Analytics"])
|
||||
async def api_run_benchmark(req: BenchmarkRequest):
|
||||
"""Run an automated benchmark swapping between available models to measure round-trip latency and RAM cache effectiveness."""
|
||||
"""Automated round-trip switch benchmark measuring latency and cache effectiveness."""
|
||||
from mcp_server import run_model_switch_benchmark
|
||||
res_str = await run_model_switch_benchmark(iterations=req.iterations or 2)
|
||||
return json.loads(res_str)
|
||||
|
||||
|
||||
# ==========================================
|
||||
# ANALYTICS (persisted)
|
||||
# ==========================================
|
||||
|
||||
@app.get("/api/analytics/profiles", summary="Which Overclock Profile Is Actually Faster", tags=["Analytics"])
|
||||
async def api_analytics_profiles(days: float = Query(7.0)):
|
||||
"""Decode throughput and thermals grouped by the profile that was active at the time."""
|
||||
return {"window_days": days, "profiles": telemetry_store.profile_comparison(days)}
|
||||
|
||||
@app.get("/api/analytics/swaps", summary="Swap Statistics", tags=["Analytics"])
|
||||
async def api_analytics_swaps(days: float = Query(7.0)):
|
||||
"""Aggregated swap/yield/purge latencies, cache-hit split and per-model throughput."""
|
||||
return telemetry_store.swap_stats(days)
|
||||
|
||||
@app.get("/api/analytics/timeseries", summary="Downsampled Telemetry History", tags=["Analytics"])
|
||||
async def api_analytics_timeseries(hours: float = Query(6.0), buckets: int = Query(240)):
|
||||
"""Long-range history for charts that outlive a page refresh."""
|
||||
return {"hours": hours, "points": telemetry_store.timeseries(hours, buckets)}
|
||||
|
||||
@app.get("/api/analytics/models", summary="Model Usage Ranking", tags=["Analytics"])
|
||||
async def api_analytics_models(days: float = Query(30.0)):
|
||||
"""Recency/frequency ranking used to prioritise the RAM warm budget."""
|
||||
return {"window_days": days, "models": telemetry_store.model_usage_ranking(days)}
|
||||
|
||||
@app.get("/api/db", summary="Telemetry Store Info", tags=["Analytics"])
|
||||
async def api_db_info():
|
||||
"""Where the persistent store lives and how much history it holds."""
|
||||
return telemetry_store.db_info()
|
||||
|
||||
|
||||
# ==========================================
|
||||
# OVERCLOCK MANAGEMENT
|
||||
# ==========================================
|
||||
|
||||
@app.get("/api/overclock", summary="Overclock Status & Profiles", tags=["Overclock"])
|
||||
async def api_overclock_status():
|
||||
"""Get live GPU overclock state, active profile, and all per-app profiles."""
|
||||
return overclock_manager.get_status()
|
||||
"""Live GPU overclock state, active profile, governor state and all per-app profiles."""
|
||||
status = overclock_manager.get_status()
|
||||
status["governor"] = thermal_governor.governor.get_status()
|
||||
return status
|
||||
|
||||
@app.post("/api/overclock/apply", summary="Apply Overclock Profile", tags=["Overclock"])
|
||||
async def api_overclock_apply(req: OverclockApplyRequest):
|
||||
"""Apply a named overclock profile (ollama | comfy | balanced) to the GPU immediately."""
|
||||
"""Apply a named overclock profile (ollama | comfy | balanced) immediately."""
|
||||
res = overclock_manager.apply_profile(req.profile)
|
||||
if not res.get("success"):
|
||||
raise HTTPException(status_code=400, detail=res.get("error"))
|
||||
return res
|
||||
|
||||
@app.post("/api/overclock/restore", summary="Restore Stock GPU State", tags=["Overclock"])
|
||||
async def api_overclock_restore():
|
||||
"""Drop all clock locks and offsets, restore default power limit and automatic fans."""
|
||||
return overclock_manager.restore_safe("manual request")
|
||||
|
||||
@app.get("/api/overclock/profiles", summary="List Overclock Profiles", tags=["Overclock"])
|
||||
async def api_overclock_profiles():
|
||||
"""List all overclock profiles with their current settings."""
|
||||
"""All overclock profiles with their current settings."""
|
||||
return overclock_manager.get_profiles()
|
||||
|
||||
@app.post("/api/overclock/profiles/{name}", summary="Update Overclock Profile", tags=["Overclock"])
|
||||
@@ -216,7 +475,7 @@ async def api_overclock_update_profile(name: str, req: OverclockProfileUpdate):
|
||||
@app.get("/api/overclock/fan", summary="Get GPU Fan Status", tags=["Overclock"])
|
||||
@app.get("/api/gpu/fan", summary="Get GPU Fan Status", tags=["Overclock"])
|
||||
async def api_get_fan_status():
|
||||
"""Get current GPU fan control mode and speed."""
|
||||
"""Current GPU fan control mode and speed."""
|
||||
return overclock_manager.get_fan_status()
|
||||
|
||||
@app.post("/api/overclock/fan", summary="Set GPU Fan Speed", tags=["Overclock"])
|
||||
@@ -228,15 +487,64 @@ async def api_set_fan(req: FanRequest):
|
||||
return overclock_manager.set_fan_speed(pct)
|
||||
return overclock_manager.set_fan_auto()
|
||||
|
||||
|
||||
# ==========================================
|
||||
# THERMAL GOVERNOR
|
||||
# ==========================================
|
||||
|
||||
@app.get("/api/governor", summary="Thermal Governor State", tags=["Governor"])
|
||||
async def api_governor_status():
|
||||
"""Current derate level, why it was applied, and the escalation history."""
|
||||
return thermal_governor.governor.get_status()
|
||||
|
||||
@app.post("/api/governor", summary="Control the Thermal Governor", tags=["Governor"])
|
||||
async def api_governor_control(req: GovernorRequest):
|
||||
"""Enable/disable the governor, or clear an active derate."""
|
||||
if req.enabled is not None:
|
||||
thermal_governor.governor.set_enabled(req.enabled)
|
||||
if req.reset:
|
||||
thermal_governor.governor.reset()
|
||||
return thermal_governor.governor.get_status()
|
||||
|
||||
|
||||
# ==========================================
|
||||
# AUTOTUNE
|
||||
# ==========================================
|
||||
|
||||
@app.get("/api/autotune", summary="Autotune Status & History", tags=["Autotune"])
|
||||
async def api_autotune_status():
|
||||
"""Sweep progress, the last result, and every recorded autotune step."""
|
||||
return autotune.get_status()
|
||||
|
||||
@app.post("/api/autotune/sweep", summary="Run an Overclock Sweep", tags=["Autotune"])
|
||||
async def api_autotune_sweep(req: SweepRequest):
|
||||
"""Walk a clock offset upward, measuring tok/s and watching for instability at each step."""
|
||||
res = await autotune.sweep(
|
||||
knob=req.knob, profile=req.profile, workload=req.workload, model=req.model,
|
||||
start=req.start, stop=req.stop, step=req.step,
|
||||
repeats=req.repeats, max_steps=req.max_steps,
|
||||
include_unlocked=req.include_unlocked, apply_best=req.apply_best,
|
||||
)
|
||||
if not res.get("success"):
|
||||
raise HTTPException(status_code=400, detail=res.get("error"))
|
||||
return res
|
||||
|
||||
@app.post("/api/autotune/cancel", summary="Cancel a Running Sweep", tags=["Autotune"])
|
||||
async def api_autotune_cancel():
|
||||
"""Stop the current sweep after the step in flight; the profile is restored either way."""
|
||||
return autotune.cancel()
|
||||
|
||||
|
||||
# Mount static web UI files
|
||||
app.mount("/static", StaticFiles(directory="/home/drjones/unified-model-manager/static"), name="static")
|
||||
app.mount("/static", StaticFiles(directory=f"{BASE_DIR}/static"), name="static")
|
||||
|
||||
@app.get("/", summary="Dashboard Web UI", tags=["UI"])
|
||||
async def root_index():
|
||||
with open("/home/drjones/unified-model-manager/static/index.html", "r") as f:
|
||||
with open(f"{BASE_DIR}/static/index.html", "r") as f:
|
||||
content = f.read()
|
||||
return HTMLResponse(content=content)
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run("server:app", host="0.0.0.0", port=9090, reload=False, log_level="info")
|
||||
uvicorn.run("server:app", host="0.0.0.0", port=9090, reload=False, log_level="info",
|
||||
timeout_graceful_shutdown=10)
|
||||
|
||||
235
static/app.js
235
static/app.js
@@ -33,6 +33,9 @@ function initSSE() {
|
||||
function updateDashboard(data) {
|
||||
if (!data) return;
|
||||
|
||||
// Governor state rides along in the shared snapshot — no extra polling needed.
|
||||
if (data.governor) renderGovernor(data.governor);
|
||||
|
||||
// 1. GPU VRAM Stats
|
||||
const gpu = data.gpu || {};
|
||||
const ram = data.ram || {};
|
||||
@@ -646,3 +649,235 @@ async function saveOverclockProfile() {
|
||||
alert(`Error: ${err}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// THERMAL GOVERNOR / RESIDENCY / ANALYTICS / AUTOTUNE
|
||||
// ============================================================================
|
||||
|
||||
let governorEnabled = true;
|
||||
|
||||
function renderGovernor(gov) {
|
||||
if (!gov) return;
|
||||
governorEnabled = gov.enabled;
|
||||
const levels = 3;
|
||||
const el = (id) => document.getElementById(id);
|
||||
if (!el('gov-label')) return;
|
||||
|
||||
el('gov-label').textContent = gov.label || '—';
|
||||
el('gov-label').className = 'text-2xl font-bold ' +
|
||||
(gov.level === 0 ? 'text-emerald-400' : gov.level < 3 ? 'text-amber-400' : 'text-rose-400');
|
||||
el('gov-level').textContent = `level ${gov.level} / ${levels}`;
|
||||
el('gov-bar').style.width = `${(gov.level / levels) * 100}%`;
|
||||
el('gov-esc').textContent = `${gov.escalate_at_c}°C`;
|
||||
el('gov-rec').textContent = `${gov.recover_below_c}°C`;
|
||||
el('gov-scale').textContent = `${Math.round((gov.offset_scale ?? 1) * 100)}%`;
|
||||
el('gov-reason').textContent = gov.last_reason || '—';
|
||||
|
||||
const toggle = el('gov-toggle');
|
||||
toggle.textContent = gov.enabled ? 'Enabled' : 'Disabled';
|
||||
toggle.className = 'px-2.5 py-1 text-xs font-semibold rounded-lg border transition ' +
|
||||
(gov.enabled ? 'bg-emerald-950/70 border-emerald-800 text-emerald-300 hover:bg-emerald-900'
|
||||
: 'bg-slate-800 border-slate-700 text-slate-400 hover:bg-slate-700');
|
||||
|
||||
const hist = el('gov-history');
|
||||
hist.innerHTML = (gov.history || []).map(h => {
|
||||
const t = new Date(h.ts * 1000).toLocaleTimeString();
|
||||
const up = h.to_level > h.from_level;
|
||||
return `<div class="flex justify-between ${up ? 'text-rose-400' : 'text-cyan-400'}">
|
||||
<span>${t} ${up ? '▲' : '▼'} L${h.to_level} ${h.label}</span>
|
||||
<span class="text-slate-500 truncate ml-2">${h.reason || ''}</span></div>`;
|
||||
}).join('') || '<div class="text-slate-600">No derates recorded — the card has stayed happy.</div>';
|
||||
}
|
||||
|
||||
async function toggleGovernor() {
|
||||
const res = await fetch('/api/governor', {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ enabled: !governorEnabled })
|
||||
});
|
||||
renderGovernor(await res.json());
|
||||
}
|
||||
|
||||
async function resetGovernor() {
|
||||
const res = await fetch('/api/governor', {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ reset: true })
|
||||
});
|
||||
renderGovernor(await res.json());
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- residency
|
||||
|
||||
async function fetchCacheReport(refresh = false) {
|
||||
const body = document.getElementById('cache-files');
|
||||
if (!body) return;
|
||||
if (refresh) body.innerHTML = '<div class="text-slate-500 text-xs">Scanning…</div>';
|
||||
try {
|
||||
const res = await fetch(`/api/cache/report?refresh=${refresh ? 'true' : 'false'}`);
|
||||
const d = await res.json();
|
||||
|
||||
document.getElementById('cache-total').textContent =
|
||||
`${d.resident_total_gb} / ${d.catalog_total_gb} GB`;
|
||||
document.getElementById('cache-meta').textContent =
|
||||
`${d.residency_pct}% resident · ${d.warm_files}/${d.files_scanned} warm · ${d.scan_ms}ms`;
|
||||
|
||||
const cap = d.capability || {};
|
||||
document.getElementById('cache-capability').innerHTML = cap.exact_everywhere
|
||||
? '<span class="text-emerald-500">Exact measurement available for every model file.</span>'
|
||||
: `<span class="text-amber-500">Ollama weights measured by read-rate probe</span> — ${cap.reason || ''}`;
|
||||
|
||||
body.innerHTML = (d.files || []).slice(0, 40).map(f => {
|
||||
const pct = f.resident_pct ?? 0;
|
||||
const color = pct >= 90 ? 'bg-emerald-500' : pct >= 25 ? 'bg-amber-500' : 'bg-slate-700';
|
||||
const kind = f.kind === 'ollama' ? 'text-cyan-400' : 'text-fuchsia-400';
|
||||
return `<div>
|
||||
<div class="flex justify-between text-[11px] font-mono">
|
||||
<span class="${kind} truncate mr-2" title="${f.full_path}">${f.name}</span>
|
||||
<span class="text-slate-400 whitespace-nowrap">${pct}% · ${f.size_gb}GB</span>
|
||||
</div>
|
||||
<div class="w-full bg-slate-950 rounded-full h-1 mt-0.5 overflow-hidden">
|
||||
<div class="${color} h-full transition-all" style="width:${pct}%"></div>
|
||||
</div>
|
||||
</div>`;
|
||||
}).join('');
|
||||
} catch (e) {
|
||||
body.innerHTML = `<div class="text-rose-400 text-xs">Residency scan failed: ${e}</div>`;
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchWarmPlan() {
|
||||
const body = document.getElementById('cache-files');
|
||||
body.innerHTML = '<div class="text-slate-500 text-xs">Building plan…</div>';
|
||||
try {
|
||||
const d = await (await fetch('/api/warm-plan')).json();
|
||||
document.getElementById('cache-total').textContent = `${d.planned_gb} GB planned`;
|
||||
document.getElementById('cache-meta').textContent =
|
||||
`budget ${d.budget_gb} GB · ${d.warm_count} to warm · ${d.skipped_count} skipped`;
|
||||
document.getElementById('cache-capability').innerHTML =
|
||||
'<span class="text-slate-500">Ranked by recency/frequency; already-resident files cost nothing.</span>';
|
||||
body.innerHTML = d.plan.map((p, i) => `<div class="flex justify-between text-[11px] font-mono">
|
||||
<span class="text-amber-400 truncate mr-2">${i + 1}. ${p.name}</span>
|
||||
<span class="text-slate-400 whitespace-nowrap">${p.size_gb}GB · ${p.resident_pct}% · s=${p.score}</span>
|
||||
</div>`).join('') || '<div class="text-emerald-500 text-xs">Everything worth warming is already resident.</div>';
|
||||
} catch (e) {
|
||||
body.innerHTML = `<div class="text-rose-400 text-xs">${e}</div>`;
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- analytics
|
||||
|
||||
async function fetchAnalytics() {
|
||||
const el = document.getElementById('analytics-body');
|
||||
if (!el) return;
|
||||
const days = document.getElementById('analytics-days').value;
|
||||
try {
|
||||
const d = await (await fetch(`/api/analytics/profiles?days=${days}`)).json();
|
||||
if (!d.profiles || !d.profiles.length) {
|
||||
el.innerHTML = '<div class="text-slate-500">No swaps recorded yet in this window. Switch a model to start building the comparison.</div>';
|
||||
return;
|
||||
}
|
||||
const best = Math.max(...d.profiles.map(p => p.avg_tok_s || 0));
|
||||
el.innerHTML = `<table class="w-full text-[11px] font-mono">
|
||||
<thead><tr class="text-slate-500 text-left border-b border-slate-800">
|
||||
<th class="pb-1">profile</th><th>tok/s</th><th>load GB/s</th><th>°C avg</th><th>W avg</th><th>SM MHz</th><th>n</th>
|
||||
</tr></thead><tbody>` +
|
||||
d.profiles.map(p => {
|
||||
const win = (p.avg_tok_s || 0) === best;
|
||||
return `<tr class="border-b border-slate-900 ${win ? 'text-emerald-400' : 'text-slate-300'}">
|
||||
<td class="py-1">${win ? '★ ' : ''}${p.profile ?? '—'}</td>
|
||||
<td>${p.avg_tok_s ?? '—'}</td><td>${p.avg_load_gbps ?? '—'}</td>
|
||||
<td>${p.avg_temp_c ?? '—'}</td><td>${p.avg_power_w ?? '—'}</td>
|
||||
<td>${p.avg_clock_sm ?? '—'}</td><td class="text-slate-500">${p.swaps}</td></tr>`;
|
||||
}).join('') + '</tbody></table>';
|
||||
} catch (e) {
|
||||
el.innerHTML = `<div class="text-rose-400">${e}</div>`;
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- autotune
|
||||
|
||||
let sweepPoll = null;
|
||||
|
||||
async function runSweep() {
|
||||
const btn = document.getElementById('at-run');
|
||||
const body = document.getElementById('autotune-body');
|
||||
const payload = {
|
||||
knob: document.getElementById('at-knob').value,
|
||||
profile: document.getElementById('at-profile').value,
|
||||
start: parseInt(document.getElementById('at-start').value, 10),
|
||||
stop: parseInt(document.getElementById('at-stop').value, 10),
|
||||
step: parseInt(document.getElementById('at-step').value, 10),
|
||||
apply_best: document.getElementById('at-apply').checked,
|
||||
};
|
||||
btn.disabled = true;
|
||||
btn.classList.add('opacity-50');
|
||||
body.innerHTML = '<div class="text-fuchsia-400">Sweeping… each step loads the model and benchmarks decode. This takes a few minutes.</div>';
|
||||
sweepPoll = setInterval(pollSweep, 2000);
|
||||
try {
|
||||
const res = await fetch('/api/autotune/sweep', {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
const d = await res.json();
|
||||
renderSweep(res.ok ? d : { error: d.detail });
|
||||
} catch (e) {
|
||||
body.innerHTML = `<div class="text-rose-400">${e}</div>`;
|
||||
} finally {
|
||||
clearInterval(sweepPoll);
|
||||
btn.disabled = false;
|
||||
btn.classList.remove('opacity-50');
|
||||
document.getElementById('autotune-state').textContent = 'idle';
|
||||
}
|
||||
}
|
||||
|
||||
async function pollSweep() {
|
||||
try {
|
||||
const d = await (await fetch('/api/autotune')).json();
|
||||
const s = document.getElementById('autotune-state');
|
||||
if (d.running && d.current) {
|
||||
s.textContent = `${d.current.knob}=${d.current.value} → ${d.current.tokens_per_sec} tok/s`;
|
||||
s.className = 'text-xs font-mono text-fuchsia-400';
|
||||
}
|
||||
} catch (e) { /* transient */ }
|
||||
}
|
||||
|
||||
async function cancelSweep() {
|
||||
await fetch('/api/autotune/cancel', { method: 'POST' });
|
||||
document.getElementById('autotune-state').textContent = 'cancelling…';
|
||||
}
|
||||
|
||||
function renderSweep(d) {
|
||||
const body = document.getElementById('autotune-body');
|
||||
if (d.error) { body.innerHTML = `<div class="text-rose-400">${d.error}</div>`; return; }
|
||||
const rows = (d.table || []).map(r => `<tr class="${r.stable ? 'text-slate-300' : 'text-rose-400'} border-b border-slate-900">
|
||||
<td class="py-0.5">${r.value}</td><td>${r.tokens_per_sec}</td><td>${r.temp_c}</td>
|
||||
<td>${r.power_w ?? '—'}</td><td>${r.clock_mem_mhz ?? '—'}</td>
|
||||
<td class="text-slate-500 truncate">${r.instability || 'stable'}</td></tr>`).join('');
|
||||
const best = d.best
|
||||
? `<div class="text-emerald-400 mb-2">★ Best stable: <span class="font-mono">${d.knob}=${d.best.value}</span>
|
||||
→ ${d.best.tokens_per_sec} tok/s at ${d.best.temp_c}°C
|
||||
${d.gain_vs_current_pct != null
|
||||
? `<span class="text-slate-400">(${d.gain_vs_current_pct > 0 ? '+' : ''}${d.gain_vs_current_pct}% vs your current setting)</span>`
|
||||
: d.gain_vs_first_step_pct != null
|
||||
? `<span class="text-slate-500">(spread across tried values: ${d.gain_vs_first_step_pct > 0 ? '+' : ''}${d.gain_vs_first_step_pct}% vs slowest)</span>`
|
||||
: ''}
|
||||
${d.applied_to_profile ? '<span class="text-fuchsia-400">· saved to profile</span>' : ''}</div>`
|
||||
: '<div class="text-amber-400 mb-2">No stable step produced throughput.</div>';
|
||||
const unstable = d.first_unstable
|
||||
? `<div class="text-rose-400 text-[11px] mb-2">Stopped climbing at ${d.first_unstable.value}: ${d.first_unstable.why}</div>` : '';
|
||||
body.innerHTML = best + unstable + `<table class="w-full text-[11px] font-mono">
|
||||
<thead><tr class="text-slate-500 text-left border-b border-slate-800">
|
||||
<th class="pb-1">offset</th><th>tok/s</th><th>°C</th><th>W</th><th>mem MHz</th><th>status</th>
|
||||
</tr></thead><tbody>${rows}</tbody></table>`;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- wiring
|
||||
|
||||
document.addEventListener('DOMContentLoaded', () => {
|
||||
fetchCacheReport(false);
|
||||
fetchAnalytics();
|
||||
setInterval(fetchAnalytics, 60000);
|
||||
setInterval(() => fetchCacheReport(false), 30000);
|
||||
fetch('/api/autotune').then(r => r.json()).then(d => {
|
||||
if (d.last_result) renderSweep(d.last_result);
|
||||
}).catch(() => {});
|
||||
});
|
||||
|
||||
@@ -665,6 +665,144 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ============ NEXT-LEVEL PANELS: governor / residency / analytics / autotune ============ -->
|
||||
<div class="grid grid-cols-1 xl:grid-cols-2 gap-5 mt-5">
|
||||
|
||||
<!-- Thermal Governor -->
|
||||
<div class="bg-slate-900/80 border border-slate-800 rounded-2xl p-5">
|
||||
<div class="flex items-center justify-between pb-3 border-b border-slate-800">
|
||||
<div class="flex items-center space-x-2">
|
||||
<div class="p-2 rounded-lg bg-rose-950/80 border border-rose-800 text-rose-400">
|
||||
<i class="fa-solid fa-temperature-arrow-down text-sm"></i>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="font-bold text-slate-100 text-sm">Thermal Governor</h3>
|
||||
<p class="text-xs text-slate-400">Walks the overclock back when the card complains</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="flex items-center space-x-2">
|
||||
<button onclick="toggleGovernor()" id="gov-toggle" class="px-2.5 py-1 text-xs font-semibold rounded-lg bg-slate-800 border border-slate-700 text-slate-300 hover:bg-slate-700 transition">Enabled</button>
|
||||
<button onclick="resetGovernor()" class="px-2.5 py-1 text-xs font-semibold rounded-lg bg-slate-800 border border-slate-700 text-slate-300 hover:bg-slate-700 transition">Reset</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="mt-4 space-y-3">
|
||||
<div class="flex items-baseline justify-between">
|
||||
<span id="gov-label" class="text-2xl font-bold text-emerald-400">full</span>
|
||||
<span id="gov-level" class="text-xs font-mono text-slate-500">level 0 / 3</span>
|
||||
</div>
|
||||
<div class="w-full bg-slate-950 rounded-full h-2 overflow-hidden border border-slate-800">
|
||||
<div id="gov-bar" class="h-full bg-gradient-to-r from-emerald-500 to-rose-500 transition-all duration-500" style="width:0%"></div>
|
||||
</div>
|
||||
<div class="grid grid-cols-3 gap-2 text-center pt-1">
|
||||
<div class="bg-slate-950/60 rounded-lg p-2 border border-slate-800">
|
||||
<div class="text-[10px] text-slate-500 uppercase">Escalate</div>
|
||||
<div id="gov-esc" class="text-sm font-mono text-rose-400">83°C</div>
|
||||
</div>
|
||||
<div class="bg-slate-950/60 rounded-lg p-2 border border-slate-800">
|
||||
<div class="text-[10px] text-slate-500 uppercase">Recover</div>
|
||||
<div id="gov-rec" class="text-sm font-mono text-cyan-400">72°C</div>
|
||||
</div>
|
||||
<div class="bg-slate-950/60 rounded-lg p-2 border border-slate-800">
|
||||
<div class="text-[10px] text-slate-500 uppercase">Offset Scale</div>
|
||||
<div id="gov-scale" class="text-sm font-mono text-slate-300">100%</div>
|
||||
</div>
|
||||
</div>
|
||||
<p class="text-xs text-slate-400"><span class="text-slate-500">Last action:</span> <span id="gov-reason" class="font-mono">cold start</span></p>
|
||||
<div id="gov-history" class="space-y-1 max-h-24 overflow-y-auto text-[11px] font-mono"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Measured page-cache residency -->
|
||||
<div class="bg-slate-900/80 border border-slate-800 rounded-2xl p-5">
|
||||
<div class="flex items-center justify-between pb-3 border-b border-slate-800">
|
||||
<div class="flex items-center space-x-2">
|
||||
<div class="p-2 rounded-lg bg-amber-950/80 border border-amber-800 text-amber-400">
|
||||
<i class="fa-solid fa-memory text-sm"></i>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="font-bold text-slate-100 text-sm">Measured Page-Cache Residency</h3>
|
||||
<p class="text-xs text-slate-400">What is genuinely in RAM, not what we hope is</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="flex items-center space-x-2">
|
||||
<button onclick="fetchWarmPlan()" class="px-2.5 py-1 text-xs font-semibold rounded-lg bg-slate-800 border border-slate-700 text-slate-300 hover:bg-slate-700 transition">Plan</button>
|
||||
<button onclick="fetchCacheReport(true)" class="px-2.5 py-1 text-xs font-semibold rounded-lg bg-amber-950/70 border border-amber-800 text-amber-300 hover:bg-amber-900 transition">Rescan</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="mt-4">
|
||||
<div class="flex items-baseline justify-between mb-2">
|
||||
<span id="cache-total" class="text-2xl font-bold text-amber-400">— GB</span>
|
||||
<span id="cache-meta" class="text-xs font-mono text-slate-500">—</span>
|
||||
</div>
|
||||
<div id="cache-capability" class="text-[11px] text-slate-500 mb-3"></div>
|
||||
<div id="cache-files" class="space-y-1.5 max-h-56 overflow-y-auto pr-1"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Profile analytics -->
|
||||
<div class="bg-slate-900/80 border border-slate-800 rounded-2xl p-5">
|
||||
<div class="flex items-center justify-between pb-3 border-b border-slate-800">
|
||||
<div class="flex items-center space-x-2">
|
||||
<div class="p-2 rounded-lg bg-cyan-950/80 border border-cyan-800 text-cyan-400">
|
||||
<i class="fa-solid fa-chart-column text-sm"></i>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="font-bold text-slate-100 text-sm">Is the Overclock Actually Working?</h3>
|
||||
<p class="text-xs text-slate-400">Decode throughput per profile, from persisted history</p>
|
||||
</div>
|
||||
</div>
|
||||
<select id="analytics-days" onchange="fetchAnalytics()" class="bg-slate-950 border border-slate-700 text-xs text-slate-300 rounded-lg px-2 py-1">
|
||||
<option value="1">24h</option><option value="7" selected>7d</option><option value="30">30d</option>
|
||||
</select>
|
||||
</div>
|
||||
<div id="analytics-body" class="mt-4 text-xs text-slate-400">Collecting data…</div>
|
||||
</div>
|
||||
|
||||
<!-- Autotune -->
|
||||
<div class="bg-slate-900/80 border border-slate-800 rounded-2xl p-5">
|
||||
<div class="flex items-center justify-between pb-3 border-b border-slate-800">
|
||||
<div class="flex items-center space-x-2">
|
||||
<div class="p-2 rounded-lg bg-fuchsia-950/80 border border-fuchsia-800 text-fuchsia-400">
|
||||
<i class="fa-solid fa-wave-square text-sm"></i>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="font-bold text-slate-100 text-sm">Overclock Autotune</h3>
|
||||
<p class="text-xs text-slate-400">Sweep a clock offset, measure tok/s, stop at instability</p>
|
||||
</div>
|
||||
</div>
|
||||
<span id="autotune-state" class="text-xs font-mono text-slate-500">idle</span>
|
||||
</div>
|
||||
<div class="mt-4 grid grid-cols-2 gap-3">
|
||||
<label class="text-xs text-slate-400">Knob
|
||||
<select id="at-knob" class="w-full mt-1 bg-slate-950 border border-slate-700 text-slate-300 rounded-lg px-2 py-1.5">
|
||||
<option value="mem_offset_mhz">Memory offset</option>
|
||||
<option value="core_offset_mhz">Core offset</option>
|
||||
</select>
|
||||
</label>
|
||||
<label class="text-xs text-slate-400">Profile
|
||||
<select id="at-profile" class="w-full mt-1 bg-slate-950 border border-slate-700 text-slate-300 rounded-lg px-2 py-1.5">
|
||||
<option value="ollama">ollama</option><option value="comfy">comfy</option><option value="balanced">balanced</option>
|
||||
</select>
|
||||
</label>
|
||||
<label class="text-xs text-slate-400">Start<input id="at-start" type="number" value="0" class="w-full mt-1 bg-slate-950 border border-slate-700 text-slate-300 rounded-lg px-2 py-1.5 font-mono"></label>
|
||||
<label class="text-xs text-slate-400">Stop<input id="at-stop" type="number" value="600" class="w-full mt-1 bg-slate-950 border border-slate-700 text-slate-300 rounded-lg px-2 py-1.5 font-mono"></label>
|
||||
<label class="text-xs text-slate-400">Step<input id="at-step" type="number" value="150" class="w-full mt-1 bg-slate-950 border border-slate-700 text-slate-300 rounded-lg px-2 py-1.5 font-mono"></label>
|
||||
<label class="text-xs text-slate-400 flex items-end space-x-2 pb-1">
|
||||
<input id="at-apply" type="checkbox" class="accent-fuchsia-500"><span>Save winner to profile</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="flex items-center space-x-2 mt-3">
|
||||
<button onclick="runSweep()" id="at-run" class="px-4 py-2 bg-gradient-to-r from-fuchsia-600 to-purple-600 hover:from-fuchsia-500 hover:to-purple-500 text-white text-xs font-bold rounded-lg shadow-md transition">
|
||||
<i class="fa-solid fa-play mr-1"></i>Run Sweep
|
||||
</button>
|
||||
<button onclick="cancelSweep()" class="px-3 py-2 bg-slate-800 border border-slate-700 text-slate-300 text-xs font-semibold rounded-lg hover:bg-slate-700 transition">Cancel</button>
|
||||
<span class="text-[11px] text-slate-500">Restores the profile when done, even on error.</span>
|
||||
</div>
|
||||
<div id="autotune-body" class="mt-3 text-xs"></div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</main>
|
||||
|
||||
<script src="/static/app.js"></script>
|
||||
|
||||
400
telemetry_store.py
Normal file
400
telemetry_store.py
Normal file
@@ -0,0 +1,400 @@
|
||||
"""SQLite time-series persistence for HyperSwap telemetry, swap events and autotune runs.
|
||||
|
||||
Everything the arbitrator learns used to die with the process (SWITCH_HISTORY was an
|
||||
in-memory deque of 50). This module keeps it on disk so we can answer the question the
|
||||
whole app exists to answer: does a given overclock profile actually deliver more tok/s?
|
||||
|
||||
Design notes:
|
||||
* WAL mode + a single writer thread -> the 1Hz sampler never blocks the event loop.
|
||||
* Telemetry rows are batched and flushed every FLUSH_INTERVAL_S.
|
||||
* Retention pruning runs opportunistically, keeping the DB bounded and small.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
logger = logging.getLogger("telemetry_store")
|
||||
|
||||
_BASE = os.path.dirname(os.path.abspath(__file__))
|
||||
DB_PATH = os.environ.get("HYPERSWAP_DB", os.path.join(_BASE, "hyperswap.db"))
|
||||
|
||||
TELEMETRY_RETENTION_DAYS = float(os.environ.get("HYPERSWAP_TELEMETRY_RETENTION_DAYS", "14"))
|
||||
EVENT_RETENTION_DAYS = float(os.environ.get("HYPERSWAP_EVENT_RETENTION_DAYS", "180"))
|
||||
FLUSH_INTERVAL_S = 2.0
|
||||
PRUNE_INTERVAL_S = 3600.0
|
||||
|
||||
SCHEMA = """
|
||||
PRAGMA journal_mode=WAL;
|
||||
PRAGMA synchronous=NORMAL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS telemetry (
|
||||
ts REAL NOT NULL,
|
||||
profile TEXT,
|
||||
gpu_util_pct REAL,
|
||||
mem_util_pct REAL,
|
||||
temp_c REAL,
|
||||
power_w REAL,
|
||||
power_limit_w REAL,
|
||||
fan_pct REAL,
|
||||
clock_sm_mhz REAL,
|
||||
clock_mem_mhz REAL,
|
||||
vram_used_bytes INTEGER,
|
||||
ollama_bytes INTEGER,
|
||||
comfy_bytes INTEGER,
|
||||
system_bytes INTEGER,
|
||||
ram_used_bytes INTEGER,
|
||||
ram_cached_bytes INTEGER,
|
||||
pcie_tx_kbps INTEGER,
|
||||
pcie_rx_kbps INTEGER,
|
||||
throttle_reasons TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_telemetry_ts ON telemetry(ts);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS events (
|
||||
ts REAL NOT NULL,
|
||||
event_type TEXT,
|
||||
source TEXT,
|
||||
target TEXT,
|
||||
profile TEXT,
|
||||
duration_ms REAL,
|
||||
load_duration_ms REAL,
|
||||
yield_confirm_ms REAL,
|
||||
tokens_per_sec REAL,
|
||||
bytes_loaded INTEGER,
|
||||
load_gbps REAL,
|
||||
cache_status TEXT,
|
||||
detail TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_ts ON events(ts);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_type ON events(event_type);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS autotune_runs (
|
||||
ts REAL NOT NULL,
|
||||
profile TEXT,
|
||||
knob TEXT,
|
||||
core_offset_mhz INTEGER,
|
||||
mem_offset_mhz INTEGER,
|
||||
tokens_per_sec REAL,
|
||||
load_gbps REAL,
|
||||
temp_c REAL,
|
||||
power_w REAL,
|
||||
stable INTEGER,
|
||||
instability TEXT,
|
||||
note TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_autotune_ts ON autotune_runs(ts);
|
||||
"""
|
||||
|
||||
|
||||
class _Writer(threading.Thread):
|
||||
"""Single background writer: batches telemetry, commits events immediately."""
|
||||
|
||||
daemon = True
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__(name="telemetry-writer")
|
||||
self.q: "queue.Queue[Optional[tuple]]" = queue.Queue(maxsize=10000)
|
||||
self._stop = threading.Event()
|
||||
self._last_prune = 0.0
|
||||
|
||||
def run(self) -> None:
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.executescript(SCHEMA)
|
||||
conn.commit()
|
||||
pending: List[tuple] = []
|
||||
last_flush = time.time()
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
item = self.q.get(timeout=0.5)
|
||||
except queue.Empty:
|
||||
item = None
|
||||
if item is not None:
|
||||
kind, sql, params = item
|
||||
if kind == "telemetry":
|
||||
pending.append((sql, params))
|
||||
else:
|
||||
try:
|
||||
conn.execute(sql, params)
|
||||
conn.commit()
|
||||
except Exception as e:
|
||||
logger.warning("event write failed: %s", e)
|
||||
now = time.time()
|
||||
if pending and (now - last_flush) >= FLUSH_INTERVAL_S:
|
||||
try:
|
||||
for sql, params in pending:
|
||||
conn.execute(sql, params)
|
||||
conn.commit()
|
||||
except Exception as e:
|
||||
logger.warning("telemetry flush failed: %s", e)
|
||||
pending.clear()
|
||||
last_flush = now
|
||||
if now - self._last_prune > PRUNE_INTERVAL_S:
|
||||
self._last_prune = now
|
||||
try:
|
||||
conn.execute("DELETE FROM telemetry WHERE ts < ?",
|
||||
(now - TELEMETRY_RETENTION_DAYS * 86400,))
|
||||
conn.execute("DELETE FROM events WHERE ts < ?",
|
||||
(now - EVENT_RETENTION_DAYS * 86400,))
|
||||
conn.commit()
|
||||
except Exception as e:
|
||||
logger.debug("prune failed: %s", e)
|
||||
try:
|
||||
for sql, params in pending:
|
||||
conn.execute(sql, params)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop.set()
|
||||
|
||||
|
||||
_writer: Optional[_Writer] = None
|
||||
_writer_lock = threading.Lock()
|
||||
|
||||
|
||||
def start() -> None:
|
||||
global _writer
|
||||
with _writer_lock:
|
||||
if _writer is None or not _writer.is_alive():
|
||||
_writer = _Writer()
|
||||
_writer.start()
|
||||
logger.info("telemetry store started at %s", DB_PATH)
|
||||
|
||||
|
||||
def stop() -> None:
|
||||
global _writer
|
||||
with _writer_lock:
|
||||
if _writer is not None:
|
||||
_writer.stop()
|
||||
_writer.join(timeout=3.0)
|
||||
_writer = None
|
||||
|
||||
|
||||
def _submit(kind: str, sql: str, params: tuple) -> None:
|
||||
w = _writer
|
||||
if w is None:
|
||||
return
|
||||
try:
|
||||
w.q.put_nowait((kind, sql, params))
|
||||
except queue.Full:
|
||||
logger.debug("telemetry queue full, dropping sample")
|
||||
|
||||
|
||||
_TELEMETRY_SQL = """
|
||||
INSERT INTO telemetry (ts, profile, gpu_util_pct, mem_util_pct, temp_c, power_w, power_limit_w,
|
||||
fan_pct, clock_sm_mhz, clock_mem_mhz, vram_used_bytes, ollama_bytes, comfy_bytes, system_bytes,
|
||||
ram_used_bytes, ram_cached_bytes, pcie_tx_kbps, pcie_rx_kbps, throttle_reasons)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
"""
|
||||
|
||||
|
||||
def record_telemetry(gpu: Dict[str, Any], ram: Dict[str, Any], profile: Optional[str] = None,
|
||||
throttle_reasons: Optional[str] = None) -> None:
|
||||
"""Persist a single 1Hz telemetry sample. Never raises."""
|
||||
try:
|
||||
if not gpu.get("available"):
|
||||
return
|
||||
bd = gpu.get("breakdown", {})
|
||||
_submit("telemetry", _TELEMETRY_SQL, (
|
||||
time.time(), profile,
|
||||
gpu.get("gpu_util_pct"), gpu.get("mem_util_pct"), gpu.get("temperature_c"),
|
||||
gpu.get("power_w"), gpu.get("power_limit_w"), gpu.get("fan_pct"),
|
||||
gpu.get("clock_graphics_mhz"), gpu.get("clock_mem_mhz"),
|
||||
gpu.get("vram_used_bytes"),
|
||||
int(bd.get("ollama_gb", 0) * (1024 ** 3)),
|
||||
int(bd.get("comfyui_gb", 0) * (1024 ** 3)),
|
||||
int(bd.get("system_gb", 0) * (1024 ** 3)),
|
||||
ram.get("used_bytes"), ram.get("cached_bytes"),
|
||||
gpu.get("pcie_tx_kbps"), gpu.get("pcie_rx_kbps"),
|
||||
throttle_reasons,
|
||||
))
|
||||
except Exception as e:
|
||||
logger.debug("record_telemetry failed: %s", e)
|
||||
|
||||
|
||||
_EVENT_SQL = """
|
||||
INSERT INTO events (ts, event_type, source, target, profile, duration_ms, load_duration_ms,
|
||||
yield_confirm_ms, tokens_per_sec, bytes_loaded, load_gbps, cache_status, detail)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
"""
|
||||
|
||||
|
||||
def record_event(event: Dict[str, Any], profile: Optional[str] = None) -> None:
|
||||
"""Persist a swap/yield/purge event. Never raises."""
|
||||
try:
|
||||
_submit("event", _EVENT_SQL, (
|
||||
event.get("ts", time.time()),
|
||||
event.get("event_type"), event.get("source"), event.get("target"),
|
||||
profile or event.get("profile"),
|
||||
event.get("duration_ms"), event.get("load_duration_ms"),
|
||||
event.get("yield_confirm_ms"), event.get("tokens_per_sec"),
|
||||
event.get("bytes_loaded"), event.get("load_gbps"),
|
||||
event.get("cache_status"), event.get("detail"),
|
||||
))
|
||||
except Exception as e:
|
||||
logger.debug("record_event failed: %s", e)
|
||||
|
||||
|
||||
_AUTOTUNE_SQL = """
|
||||
INSERT INTO autotune_runs (ts, profile, knob, core_offset_mhz, mem_offset_mhz, tokens_per_sec,
|
||||
load_gbps, temp_c, power_w, stable, instability, note)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
"""
|
||||
|
||||
|
||||
def record_autotune(row: Dict[str, Any]) -> None:
|
||||
try:
|
||||
_submit("event", _AUTOTUNE_SQL, (
|
||||
row.get("ts", time.time()), row.get("profile"), row.get("knob"),
|
||||
row.get("core_offset_mhz"), row.get("mem_offset_mhz"),
|
||||
row.get("tokens_per_sec"), row.get("load_gbps"),
|
||||
row.get("temp_c"), row.get("power_w"),
|
||||
1 if row.get("stable") else 0,
|
||||
row.get("instability"), row.get("note"),
|
||||
))
|
||||
except Exception as e:
|
||||
logger.debug("record_autotune failed: %s", e)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ queries
|
||||
|
||||
def _read_conn() -> sqlite3.Connection:
|
||||
conn = sqlite3.connect(f"file:{DB_PATH}?mode=ro", uri=True, timeout=5.0)
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
def _rows(sql: str, params: tuple = ()) -> List[Dict[str, Any]]:
|
||||
if not os.path.exists(DB_PATH):
|
||||
return []
|
||||
try:
|
||||
with _read_conn() as conn:
|
||||
return [dict(r) for r in conn.execute(sql, params).fetchall()]
|
||||
except Exception as e:
|
||||
logger.debug("query failed: %s", e)
|
||||
return []
|
||||
|
||||
|
||||
def profile_comparison(days: float = 7.0) -> List[Dict[str, Any]]:
|
||||
"""The headline question: which overclock profile actually produces more tok/s?
|
||||
|
||||
Joins decode throughput from switch events against thermals sampled while that
|
||||
profile was active.
|
||||
"""
|
||||
since = time.time() - days * 86400
|
||||
perf = _rows("""
|
||||
SELECT profile,
|
||||
COUNT(*) AS swaps,
|
||||
AVG(tokens_per_sec) AS avg_tok_s,
|
||||
MAX(tokens_per_sec) AS max_tok_s,
|
||||
AVG(load_gbps) AS avg_load_gbps,
|
||||
AVG(load_duration_ms) AS avg_load_ms
|
||||
FROM events
|
||||
WHERE ts > ? AND event_type = 'LLM Model Switch' AND tokens_per_sec > 0
|
||||
GROUP BY profile
|
||||
""", (since,))
|
||||
thermals = {r["profile"]: r for r in _rows("""
|
||||
SELECT profile,
|
||||
AVG(temp_c) AS avg_temp_c,
|
||||
MAX(temp_c) AS max_temp_c,
|
||||
AVG(power_w) AS avg_power_w,
|
||||
AVG(clock_sm_mhz) AS avg_clock_sm,
|
||||
AVG(clock_mem_mhz) AS avg_clock_mem,
|
||||
COUNT(*) AS samples
|
||||
FROM telemetry
|
||||
WHERE ts > ? AND gpu_util_pct > 5
|
||||
GROUP BY profile
|
||||
""", (since,))}
|
||||
out = []
|
||||
for row in perf:
|
||||
merged = dict(row)
|
||||
merged.update(thermals.get(row["profile"], {}))
|
||||
for k, v in list(merged.items()):
|
||||
if isinstance(v, float):
|
||||
merged[k] = round(v, 2)
|
||||
out.append(merged)
|
||||
out.sort(key=lambda r: r.get("avg_tok_s") or 0, reverse=True)
|
||||
return out
|
||||
|
||||
|
||||
def swap_stats(days: float = 7.0) -> Dict[str, Any]:
|
||||
since = time.time() - days * 86400
|
||||
by_type = _rows("""
|
||||
SELECT event_type, COUNT(*) AS n,
|
||||
AVG(duration_ms) AS avg_ms, MIN(duration_ms) AS min_ms, MAX(duration_ms) AS max_ms,
|
||||
AVG(yield_confirm_ms) AS avg_confirm_ms
|
||||
FROM events WHERE ts > ? GROUP BY event_type ORDER BY n DESC
|
||||
""", (since,))
|
||||
cache = _rows("""
|
||||
SELECT cache_status, COUNT(*) AS n, AVG(load_gbps) AS avg_gbps
|
||||
FROM events WHERE ts > ? AND cache_status IS NOT NULL GROUP BY cache_status
|
||||
""", (since,))
|
||||
models = _rows("""
|
||||
SELECT target AS model, COUNT(*) AS loads, AVG(tokens_per_sec) AS avg_tok_s,
|
||||
AVG(load_gbps) AS avg_gbps, MAX(ts) AS last_used
|
||||
FROM events WHERE ts > ? AND event_type = 'LLM Model Switch'
|
||||
GROUP BY target ORDER BY loads DESC LIMIT 25
|
||||
""", (since,))
|
||||
return {"by_type": by_type, "by_cache_status": cache, "by_model": models, "window_days": days}
|
||||
|
||||
|
||||
def model_usage_ranking(days: float = 30.0) -> List[Dict[str, Any]]:
|
||||
"""Recency+frequency score per model, used to prioritise the RAM warm budget."""
|
||||
since = time.time() - days * 86400
|
||||
now = time.time()
|
||||
rows = _rows("""
|
||||
SELECT target AS model, COUNT(*) AS loads, MAX(ts) AS last_used
|
||||
FROM events WHERE ts > ? AND target IS NOT NULL AND event_type IN
|
||||
('LLM Model Switch','Model Warm') GROUP BY target
|
||||
""", (since,))
|
||||
for r in rows:
|
||||
age_h = max((now - (r["last_used"] or since)) / 3600.0, 0.01)
|
||||
# frequency, decayed by recency (half-life ~24h)
|
||||
r["score"] = round(r["loads"] * (0.5 ** (age_h / 24.0)) + 1.0 / age_h, 4)
|
||||
r["age_hours"] = round(age_h, 2)
|
||||
rows.sort(key=lambda r: r["score"], reverse=True)
|
||||
return rows
|
||||
|
||||
|
||||
def timeseries(hours: float = 6.0, buckets: int = 240) -> List[Dict[str, Any]]:
|
||||
"""Downsampled history for long-range dashboard charts."""
|
||||
since = time.time() - hours * 3600
|
||||
width = max((hours * 3600) / max(buckets, 1), 1.0)
|
||||
return _rows("""
|
||||
SELECT CAST(ts / ? AS INTEGER) * ? AS bucket_ts,
|
||||
AVG(gpu_util_pct) AS gpu_util_pct, AVG(temp_c) AS temp_c,
|
||||
AVG(power_w) AS power_w, AVG(fan_pct) AS fan_pct,
|
||||
AVG(vram_used_bytes) AS vram_used_bytes,
|
||||
AVG(ollama_bytes) AS ollama_bytes, AVG(comfy_bytes) AS comfy_bytes,
|
||||
AVG(ram_cached_bytes) AS ram_cached_bytes,
|
||||
AVG(clock_sm_mhz) AS clock_sm_mhz, AVG(clock_mem_mhz) AS clock_mem_mhz
|
||||
FROM telemetry WHERE ts > ?
|
||||
GROUP BY bucket_ts ORDER BY bucket_ts
|
||||
""", (width, width, since))
|
||||
|
||||
|
||||
def recent_events(limit: int = 50) -> List[Dict[str, Any]]:
|
||||
return _rows("SELECT * FROM events ORDER BY ts DESC LIMIT ?", (limit,))
|
||||
|
||||
|
||||
def autotune_history(limit: int = 200) -> List[Dict[str, Any]]:
|
||||
return _rows("SELECT * FROM autotune_runs ORDER BY ts DESC LIMIT ?", (limit,))
|
||||
|
||||
|
||||
def db_info() -> Dict[str, Any]:
|
||||
info = {"path": DB_PATH, "exists": os.path.exists(DB_PATH)}
|
||||
if info["exists"]:
|
||||
info["size_mb"] = round(os.path.getsize(DB_PATH) / (1024 ** 2), 2)
|
||||
for tbl in ("telemetry", "events", "autotune_runs"):
|
||||
r = _rows(f"SELECT COUNT(*) AS n FROM {tbl}")
|
||||
info[f"{tbl}_rows"] = r[0]["n"] if r else 0
|
||||
r = _rows("SELECT MIN(ts) AS a, MAX(ts) AS b FROM telemetry")
|
||||
if r and r[0]["a"]:
|
||||
info["coverage_hours"] = round((r[0]["b"] - r[0]["a"]) / 3600.0, 2)
|
||||
return info
|
||||
175
thermal_governor.py
Normal file
175
thermal_governor.py
Normal file
@@ -0,0 +1,175 @@
|
||||
"""Thermal governor: walks the overclock back when the card says it is unhappy.
|
||||
|
||||
Nothing in HyperSwap used to de-escalate. A profile pinned the fans to a manual PWM and
|
||||
locked the clocks, and those settings stayed exactly where they were put no matter what
|
||||
the card reported. This module watches the telemetry the sampler already collects (so it
|
||||
costs no extra NVML calls) and derates the active profile when the GPU is hot or
|
||||
throttling, then restores it once things settle.
|
||||
|
||||
Hysteresis is deliberate: escalation needs HOT_SAMPLES consecutive bad samples and
|
||||
recovery needs COOL_SAMPLES consecutive good ones, so a single spike during a diffusion
|
||||
step does not cause profile thrash.
|
||||
"""
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import overclock_manager
|
||||
|
||||
logger = logging.getLogger("thermal_governor")
|
||||
|
||||
# Derate ladder: each step scales the profile's clock offsets and raises the fan floor.
|
||||
DERATE_LADDER = [
|
||||
{"level": 0, "offset_scale": 1.00, "fan_floor": 0, "label": "full"},
|
||||
{"level": 1, "offset_scale": 0.60, "fan_floor": 80, "label": "derated -40%"},
|
||||
{"level": 2, "offset_scale": 0.25, "fan_floor": 90, "label": "derated -75%"},
|
||||
{"level": 3, "offset_scale": 0.00, "fan_floor": 100, "label": "stock clocks, fans max"},
|
||||
]
|
||||
|
||||
TEMP_ESCALATE_C = 83.0
|
||||
TEMP_RECOVER_C = 72.0
|
||||
HOT_SAMPLES = 5 # ~5 s at 1 Hz before we act
|
||||
COOL_SAMPLES = 30 # ~30 s of calm before we give the clocks back
|
||||
REAPPLY_COOLDOWN_S = 20.0
|
||||
|
||||
# Throttle reasons that mean the hardware is protecting itself, not just hitting a cap.
|
||||
HARD_THROTTLES = {"hw_thermal_slowdown", "sw_thermal_slowdown", "hw_slowdown",
|
||||
"hw_power_brake_slowdown"}
|
||||
|
||||
|
||||
class ThermalGovernor:
|
||||
def __init__(self) -> None:
|
||||
self.enabled = True
|
||||
self.level = 0
|
||||
self.hot_streak = 0
|
||||
self.cool_streak = 0
|
||||
self.last_change = 0.0
|
||||
self.last_reason = "cold start"
|
||||
self.history: List[Dict[str, Any]] = []
|
||||
self._lock = threading.Lock()
|
||||
self._base_profile: Optional[str] = None
|
||||
|
||||
# ---------------------------------------------------------------- observation
|
||||
|
||||
def observe(self, gpu: Dict[str, Any], active_profile: Optional[str] = None) -> None:
|
||||
"""Feed one telemetry sample. Cheap and non-blocking; actuation runs in a thread."""
|
||||
if not self.enabled or not gpu.get("available"):
|
||||
return
|
||||
temp = gpu.get("temperature_c") or 0
|
||||
reasons = set(gpu.get("throttle_reasons") or [])
|
||||
hard = bool(reasons & HARD_THROTTLES)
|
||||
|
||||
hot = temp >= TEMP_ESCALATE_C or hard
|
||||
cool = temp <= TEMP_RECOVER_C and not hard
|
||||
|
||||
with self._lock:
|
||||
if hot:
|
||||
self.hot_streak += 1
|
||||
self.cool_streak = 0
|
||||
elif cool:
|
||||
self.cool_streak += 1
|
||||
self.hot_streak = 0
|
||||
else:
|
||||
self.hot_streak = 0
|
||||
self.cool_streak = 0
|
||||
|
||||
now = time.time()
|
||||
if now - self.last_change < REAPPLY_COOLDOWN_S:
|
||||
return
|
||||
|
||||
if self.hot_streak >= HOT_SAMPLES and self.level < len(DERATE_LADDER) - 1:
|
||||
why = (f"{temp:.0f}°C" if temp >= TEMP_ESCALATE_C else "") + \
|
||||
(f" throttling: {','.join(sorted(reasons & HARD_THROTTLES))}" if hard else "")
|
||||
self._step(self.level + 1, why.strip(), active_profile, temp)
|
||||
elif self.cool_streak >= COOL_SAMPLES and self.level > 0:
|
||||
self._step(self.level - 1, f"stable at {temp:.0f}°C", active_profile, temp)
|
||||
|
||||
def _step(self, new_level: int, reason: str, active_profile: Optional[str],
|
||||
temp: float) -> None:
|
||||
old = self.level
|
||||
self.level = new_level
|
||||
self.hot_streak = self.cool_streak = 0
|
||||
self.last_change = time.time()
|
||||
self.last_reason = reason
|
||||
profile = active_profile or overclock_manager.ACTIVE_PROFILE
|
||||
self._base_profile = profile
|
||||
entry = {
|
||||
"ts": time.time(), "from_level": old, "to_level": new_level,
|
||||
"label": DERATE_LADDER[new_level]["label"], "reason": reason,
|
||||
"temp_c": temp, "profile": profile,
|
||||
}
|
||||
self.history.insert(0, entry)
|
||||
del self.history[50:]
|
||||
direction = "escalating" if new_level > old else "recovering"
|
||||
logger.warning(f"Thermal governor {direction} to level {new_level} "
|
||||
f"({DERATE_LADDER[new_level]['label']}) — {reason}")
|
||||
threading.Thread(target=self._actuate, args=(profile,), daemon=True).start()
|
||||
|
||||
def _actuate(self, profile: str) -> None:
|
||||
try:
|
||||
overclock_manager.apply_profile(profile, overrides=self.overrides_for(profile))
|
||||
except Exception as e:
|
||||
logger.error(f"Governor failed to apply derate: {e}")
|
||||
|
||||
# ---------------------------------------------------------------- overrides
|
||||
|
||||
def overrides_for(self, profile: str) -> Dict[str, Any]:
|
||||
"""Translate the current derate level into profile overrides."""
|
||||
step = DERATE_LADDER[self.level]
|
||||
if self.level == 0:
|
||||
return {}
|
||||
cfg = overclock_manager.load_profiles().get(profile, {})
|
||||
scale = step["offset_scale"]
|
||||
overrides: Dict[str, Any] = {
|
||||
"core_offset_mhz": int(cfg.get("core_offset_mhz", 0) * scale),
|
||||
"mem_offset_mhz": int(cfg.get("mem_offset_mhz", 0) * scale),
|
||||
}
|
||||
if step["level"] >= 2:
|
||||
# Stop pinning the core clock high when the card is already backing off.
|
||||
overrides["lock_core_min"] = 0
|
||||
overrides["lock_core_max"] = 0
|
||||
floor = step["fan_floor"]
|
||||
if floor:
|
||||
overrides["fan_mode"] = "manual"
|
||||
overrides["fan_speed_pct"] = max(int(cfg.get("fan_speed_pct", 0)), floor)
|
||||
return overrides
|
||||
|
||||
# ---------------------------------------------------------------- control
|
||||
|
||||
def reset(self) -> Dict[str, Any]:
|
||||
with self._lock:
|
||||
self.level = 0
|
||||
self.hot_streak = self.cool_streak = 0
|
||||
self.last_change = time.time()
|
||||
self.last_reason = "manual reset"
|
||||
profile = self._base_profile or overclock_manager.ACTIVE_PROFILE
|
||||
if profile and profile != "stock":
|
||||
overclock_manager.apply_profile(profile)
|
||||
return self.get_status()
|
||||
|
||||
def set_enabled(self, enabled: bool) -> Dict[str, Any]:
|
||||
self.enabled = bool(enabled)
|
||||
if not enabled and self.level > 0:
|
||||
self.reset()
|
||||
return self.get_status()
|
||||
|
||||
def get_status(self) -> Dict[str, Any]:
|
||||
step = DERATE_LADDER[self.level]
|
||||
return {
|
||||
"enabled": self.enabled,
|
||||
"level": self.level,
|
||||
"label": step["label"],
|
||||
"offset_scale": step["offset_scale"],
|
||||
"fan_floor": step["fan_floor"],
|
||||
"last_reason": self.last_reason,
|
||||
"hot_streak": self.hot_streak,
|
||||
"cool_streak": self.cool_streak,
|
||||
"escalate_at_c": TEMP_ESCALATE_C,
|
||||
"recover_below_c": TEMP_RECOVER_C,
|
||||
"seconds_since_change": round(time.time() - self.last_change, 1) if self.last_change else None,
|
||||
"history": self.history[:10],
|
||||
}
|
||||
|
||||
|
||||
governor = ThermalGovernor()
|
||||
@@ -11,6 +11,8 @@ import json
|
||||
import websockets
|
||||
|
||||
import overclock_manager
|
||||
import ram_optimizer
|
||||
import telemetry_store
|
||||
|
||||
try:
|
||||
import pynvml
|
||||
@@ -24,9 +26,131 @@ logger = logging.getLogger("vram_arbitrator")
|
||||
OLLAMA_API_BASE = "http://localhost:11434"
|
||||
COMFY_API_BASE = "http://127.0.0.1:8188"
|
||||
|
||||
# Circular buffer for transition events
|
||||
# Circular buffer for transition events (the durable log lives in telemetry_store)
|
||||
SWITCH_HISTORY = deque(maxlen=50)
|
||||
|
||||
# Bandwidth thresholds for classifying how a model reached VRAM, calibrated by measuring
|
||||
# the same 12.87 GB model loaded cold and warm on this box (2026-08-28):
|
||||
#
|
||||
# 3.1% resident -> 34.3 s -> 0.38 GB/s
|
||||
# 100% resident -> 4.9 s -> 2.63 GB/s
|
||||
#
|
||||
# The first cut at these numbers assumed a page-cache-fed load would approach the bus
|
||||
# rate and set the cache-hit bar at 5 GB/s. It does not: Ollama's load_duration covers
|
||||
# host-to-device transfer and model initialisation as well as the file read, so a fully
|
||||
# resident model still reports ~2.6 GB/s while the page cache itself reads at 6.4 GB/s.
|
||||
# A 5 GB/s bar could therefore never be met, and every warm load was being reported as
|
||||
# a partial hit. Thresholds now sit either side of the measured 6.9x separation.
|
||||
RAM_HIT_GBPS = 2.0
|
||||
PARTIAL_HIT_GBPS = 0.8
|
||||
|
||||
# How long Ollama's VRAM may take to actually drain before we stop waiting.
|
||||
# Ollama will not unload a model while a generation is in flight, so a short ceiling
|
||||
# reports a timeout for what is really just a busy model finishing its request. Observed
|
||||
# here: two yields hit the old 3 s limit with 8.2 GB still held while ComfyUI was starting.
|
||||
# Waiting longer is the safer failure mode -- the alternative is diffusion allocating into
|
||||
# VRAM that is still occupied.
|
||||
YIELD_CONFIRM_TIMEOUT_S = 10.0
|
||||
YIELD_CONFIRM_POLL_S = 0.02
|
||||
YIELD_RESIDUAL_BYTES = 256 * 1024 ** 2 # treat <256 MB as "released"
|
||||
|
||||
# Connection-pooled clients. Re-creating an AsyncClient per call meant a fresh TCP
|
||||
# handshake on every one of the watchdog's polls.
|
||||
_clients: Dict[str, httpx.AsyncClient] = {}
|
||||
|
||||
|
||||
def _client(base_url: str, timeout: float) -> httpx.AsyncClient:
|
||||
key = f"{base_url}|{timeout}"
|
||||
c = _clients.get(key)
|
||||
if c is None or c.is_closed:
|
||||
c = httpx.AsyncClient(
|
||||
base_url=base_url,
|
||||
timeout=timeout,
|
||||
limits=httpx.Limits(max_keepalive_connections=4, max_connections=8),
|
||||
)
|
||||
_clients[key] = c
|
||||
return c
|
||||
|
||||
|
||||
async def close_clients() -> None:
|
||||
for c in list(_clients.values()):
|
||||
try:
|
||||
await c.aclose()
|
||||
except Exception:
|
||||
pass
|
||||
_clients.clear()
|
||||
|
||||
|
||||
# Bit flags from nvmlDeviceGetCurrentClocksThrottleReasons, decoded for the governor.
|
||||
THROTTLE_REASONS = {
|
||||
0x0000000000000001: "gpu_idle",
|
||||
0x0000000000000002: "applications_clocks_setting",
|
||||
0x0000000000000004: "sw_power_cap",
|
||||
0x0000000000000008: "hw_slowdown",
|
||||
0x0000000000000010: "sync_boost",
|
||||
0x0000000000000020: "sw_thermal_slowdown",
|
||||
0x0000000000000040: "hw_thermal_slowdown",
|
||||
0x0000000000000080: "hw_power_brake_slowdown",
|
||||
0x0000000000000100: "display_clock_setting",
|
||||
}
|
||||
|
||||
|
||||
def decode_throttle_reasons(bits: int) -> List[str]:
|
||||
return [name for mask, name in THROTTLE_REASONS.items() if bits & mask]
|
||||
|
||||
|
||||
def get_process_vram_bytes() -> Dict[str, int]:
|
||||
"""Fast NVML-only VRAM attribution, used by the yield barrier's tight poll loop.
|
||||
|
||||
Deliberately avoids psutil lookups: this runs every 20 ms while we wait for VRAM
|
||||
to actually drain.
|
||||
"""
|
||||
out = {"ollama_bytes": 0, "comfyui_bytes": 0, "other_bytes": 0, "free_bytes": 0}
|
||||
if not NVML_AVAILABLE:
|
||||
return out
|
||||
try:
|
||||
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
|
||||
out["free_bytes"] = pynvml.nvmlDeviceGetMemoryInfo(handle).free
|
||||
procs = list(pynvml.nvmlDeviceGetComputeRunningProcesses(handle))
|
||||
try:
|
||||
procs += list(pynvml.nvmlDeviceGetGraphicsRunningProcesses(handle))
|
||||
except Exception:
|
||||
pass
|
||||
merged: Dict[int, int] = {}
|
||||
for p in procs:
|
||||
merged[p.pid] = max(merged.get(p.pid, 0), p.usedGpuMemory or 0)
|
||||
for pid, used in merged.items():
|
||||
kind = _PID_KIND_CACHE.get(pid)
|
||||
if kind is None:
|
||||
kind = _classify_pid(pid)
|
||||
_PID_KIND_CACHE[pid] = kind
|
||||
if kind == "ollama":
|
||||
out["ollama_bytes"] += used
|
||||
elif kind == "comfy":
|
||||
out["comfyui_bytes"] += used
|
||||
else:
|
||||
out["other_bytes"] += used
|
||||
except Exception as e:
|
||||
logger.debug(f"get_process_vram_bytes failed: {e}")
|
||||
return out
|
||||
|
||||
|
||||
_PID_KIND_CACHE: Dict[int, str] = {}
|
||||
|
||||
|
||||
def _classify_pid(pid: int) -> str:
|
||||
try:
|
||||
proc = psutil.Process(pid)
|
||||
pname = proc.name().lower()
|
||||
cmdline = " ".join(proc.cmdline()).lower()
|
||||
except Exception:
|
||||
return "other"
|
||||
if "ollama" in pname or "llama-server" in cmdline:
|
||||
return "ollama"
|
||||
if "comfy" in cmdline or "main.py" in cmdline:
|
||||
return "comfy"
|
||||
return "other"
|
||||
|
||||
|
||||
def get_gpu_hardware_stats() -> Dict[str, Any]:
|
||||
"""Retrieve comprehensive GPU hardware and process metrics via NVML."""
|
||||
@@ -77,6 +201,49 @@ def get_gpu_hardware_stats() -> Dict[str, Any]:
|
||||
clock_graphics = 0
|
||||
clock_mem = 0
|
||||
|
||||
# PCIe throughput (KB/s) — TX + RX. Key metric for the RAM-cache
|
||||
# PCIe-speed swap thesis (assimilated from pmady/gpu-mcp-server).
|
||||
pcie_tx_kbps = 0
|
||||
pcie_rx_kbps = 0
|
||||
try:
|
||||
pcie_tx_kbps = pynvml.nvmlDeviceGetPcieThroughput(handle, pynvml.NVML_PCIE_UTIL_TX_BYTES)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
pcie_rx_kbps = pynvml.nvmlDeviceGetPcieThroughput(handle, pynvml.NVML_PCIE_UTIL_RX_BYTES)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Why the GPU is not running at full clocks — the thermal governor reads this.
|
||||
throttle_bits = 0
|
||||
throttle_reasons: List[str] = []
|
||||
try:
|
||||
throttle_bits = pynvml.nvmlDeviceGetCurrentClocksThrottleReasons(handle)
|
||||
throttle_reasons = decode_throttle_reasons(throttle_bits)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Power management limit (watts) — the OC ceiling.
|
||||
power_limit_w = 0.0
|
||||
try:
|
||||
power_limit_w = round(pynvml.nvmlDeviceGetPowerManagementLimit(handle) / 1000.0, 1)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Driver + CUDA version (completeness).
|
||||
driver_version = ""
|
||||
cuda_version = ""
|
||||
try:
|
||||
dv = pynvml.nvmlSystemGetDriverVersion()
|
||||
driver_version = dv.decode("utf-8") if isinstance(dv, bytes) else str(dv)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
cv = pynvml.nvmlSystemGetCudaDriverVersion()
|
||||
cuda_version = cv # int like 12030 == CUDA 12.3
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Discover processes on GPU
|
||||
proc_breakdown = {
|
||||
"ollama_bytes": 0,
|
||||
@@ -142,6 +309,13 @@ def get_gpu_hardware_stats() -> Dict[str, Any]:
|
||||
"mem_util_pct": util_rates.memory,
|
||||
"temperature_c": temp_c,
|
||||
"power_w": power_w,
|
||||
"power_limit_w": power_limit_w,
|
||||
"pcie_tx_kbps": pcie_tx_kbps,
|
||||
"pcie_rx_kbps": pcie_rx_kbps,
|
||||
"throttle_bits": throttle_bits,
|
||||
"throttle_reasons": throttle_reasons,
|
||||
"driver_version": driver_version,
|
||||
"cuda_version": cuda_version,
|
||||
"fan_pct": fan_pct,
|
||||
"fans": fans,
|
||||
"num_fans": len(fans),
|
||||
@@ -174,9 +348,9 @@ async def get_ollama_live_state() -> Dict[str, Any]:
|
||||
"installed_models": []
|
||||
}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=3.0) as client:
|
||||
client = _client(OLLAMA_API_BASE, 3.0)
|
||||
# Check running models (ps)
|
||||
ps_resp = await client.get(f"{OLLAMA_API_BASE}/api/ps")
|
||||
ps_resp = await client.get("/api/ps")
|
||||
if ps_resp.status_code == 200:
|
||||
state["online"] = True
|
||||
models = ps_resp.json().get("models", [])
|
||||
@@ -190,7 +364,7 @@ async def get_ollama_live_state() -> Dict[str, Any]:
|
||||
state["expires_at"] = first.get("expires_at")
|
||||
|
||||
# Check all tags
|
||||
tags_resp = await client.get(f"{OLLAMA_API_BASE}/api/tags")
|
||||
tags_resp = await client.get("/api/tags")
|
||||
if tags_resp.status_code == 200:
|
||||
state["installed_models"] = tags_resp.json().get("models", [])
|
||||
except Exception as e:
|
||||
@@ -211,9 +385,9 @@ async def get_comfyui_live_state() -> Dict[str, Any]:
|
||||
"vram_total_mb": 0,
|
||||
}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=3.0) as client:
|
||||
client = _client(COMFY_API_BASE, 3.0)
|
||||
# Check system stats
|
||||
stats_resp = await client.get(f"{COMFY_API_BASE}/system_stats")
|
||||
stats_resp = await client.get("/system_stats")
|
||||
if stats_resp.status_code == 200:
|
||||
state["online"] = True
|
||||
data = stats_resp.json()
|
||||
@@ -224,7 +398,7 @@ async def get_comfyui_live_state() -> Dict[str, Any]:
|
||||
state["vram_total_mb"] = round(dev.get("vram_total", 0) / (1024**2), 1)
|
||||
|
||||
# Check queue
|
||||
queue_resp = await client.get(f"{COMFY_API_BASE}/queue")
|
||||
queue_resp = await client.get("/queue")
|
||||
if queue_resp.status_code == 200:
|
||||
qdata = queue_resp.json()
|
||||
running = qdata.get("queue_running", [])
|
||||
@@ -239,58 +413,178 @@ async def get_comfyui_live_state() -> Dict[str, Any]:
|
||||
|
||||
return state
|
||||
|
||||
async def instant_free_ollama_vram(model_name: Optional[str] = None) -> Dict[str, Any]:
|
||||
"""Tell Ollama to instantly yield VRAM without evicting from OS page cache."""
|
||||
async def _await_vram_release(baseline_bytes: int,
|
||||
timeout_s: float = YIELD_CONFIRM_TIMEOUT_S) -> Dict[str, Any]:
|
||||
"""Block until Ollama's VRAM has actually drained, or we give up.
|
||||
|
||||
Posting keep_alive:0 only *asks* Ollama to unload; the driver frees the allocation
|
||||
some milliseconds later. Returning before that happens is how ComfyUI ends up
|
||||
allocating into VRAM that is still occupied, which shows up as a CUDA OOM mid-graph.
|
||||
"""
|
||||
t0 = time.perf_counter()
|
||||
if not model_name:
|
||||
last = baseline_bytes
|
||||
while True:
|
||||
snap = get_process_vram_bytes()
|
||||
last = snap["ollama_bytes"]
|
||||
if last <= YIELD_RESIDUAL_BYTES:
|
||||
return {
|
||||
"confirmed": True,
|
||||
"confirm_ms": round((time.perf_counter() - t0) * 1000, 2),
|
||||
"residual_bytes": last,
|
||||
"free_bytes": snap["free_bytes"],
|
||||
}
|
||||
if (time.perf_counter() - t0) >= timeout_s:
|
||||
return {
|
||||
"confirmed": False,
|
||||
"confirm_ms": round((time.perf_counter() - t0) * 1000, 2),
|
||||
"residual_bytes": last,
|
||||
"free_bytes": snap["free_bytes"],
|
||||
"error": f"Ollama still holding {round(last / (1024**3), 2)} GB after {timeout_s}s",
|
||||
}
|
||||
await asyncio.sleep(YIELD_CONFIRM_POLL_S)
|
||||
|
||||
|
||||
def _record(event: Dict[str, Any]) -> None:
|
||||
"""Push an event to both the in-memory ring and the durable store."""
|
||||
event.setdefault("ts", time.time())
|
||||
event.setdefault("timestamp", time.strftime("%H:%M:%S"))
|
||||
SWITCH_HISTORY.appendleft(event)
|
||||
telemetry_store.record_event(event, profile=overclock_manager.ACTIVE_PROFILE)
|
||||
|
||||
|
||||
async def instant_free_ollama_vram(model_name: Optional[str] = None,
|
||||
confirm: bool = True) -> Dict[str, Any]:
|
||||
"""Yield Ollama's VRAM and wait for the driver to actually release it.
|
||||
|
||||
The returned duration_ms is now the real end-to-end release time, not just how long
|
||||
the HTTP POST took.
|
||||
"""
|
||||
t0 = time.perf_counter()
|
||||
if model_name:
|
||||
targets = [model_name]
|
||||
else:
|
||||
# Unload *every* resident model, not just loaded_models[0]. Ollama will happily
|
||||
# keep several models in VRAM at once; releasing only the first left the rest
|
||||
# allocated, which the confirm barrier caught as "still holding 8.2 GB after 3s".
|
||||
ollama_state = await get_ollama_live_state()
|
||||
model_name = ollama_state.get("active_model_name")
|
||||
targets = [m.get("name") for m in ollama_state.get("loaded_models", []) if m.get("name")]
|
||||
if not targets and ollama_state.get("active_model_name"):
|
||||
targets = [ollama_state["active_model_name"]]
|
||||
|
||||
if not model_name:
|
||||
return {"success": True, "message": "No active Ollama model in VRAM", "duration_ms": 0}
|
||||
if not targets:
|
||||
return {"success": True, "message": "No active Ollama model in VRAM",
|
||||
"duration_ms": 0, "confirmed": True}
|
||||
|
||||
model_name = targets[0] if len(targets) == 1 else f"{len(targets)} models"
|
||||
baseline = get_process_vram_bytes()["ollama_bytes"]
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
resp = await client.post(
|
||||
f"{OLLAMA_API_BASE}/api/generate",
|
||||
json={"model": model_name, "keep_alive": 0},
|
||||
)
|
||||
client = _client(OLLAMA_API_BASE, 5.0)
|
||||
await asyncio.gather(*[
|
||||
client.post("/api/generate", json={"model": t, "keep_alive": 0})
|
||||
for t in targets
|
||||
], return_exceptions=True)
|
||||
request_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
|
||||
barrier = {"confirmed": None, "confirm_ms": 0.0, "residual_bytes": baseline}
|
||||
if confirm:
|
||||
barrier = await _await_vram_release(baseline)
|
||||
|
||||
duration_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
event = {
|
||||
"timestamp": time.strftime("%H:%M:%S"),
|
||||
freed_gb = round(max(baseline - barrier.get("residual_bytes", 0), 0) / (1024**3), 2)
|
||||
|
||||
_record({
|
||||
"event_type": "Ollama VRAM Yield",
|
||||
"source": model_name,
|
||||
"source": ", ".join(targets)[:200],
|
||||
"target": "VRAM 0MB (Kept in RAM)",
|
||||
"duration_ms": duration_ms,
|
||||
"cache_status": "RAM-Cached",
|
||||
"yield_confirm_ms": barrier.get("confirm_ms"),
|
||||
"cache_status": "RAM-Cached" if barrier.get("confirmed") else "Yield Timeout",
|
||||
"detail": barrier.get("error"),
|
||||
})
|
||||
return {
|
||||
"success": True,
|
||||
"model": model_name,
|
||||
"models_unloaded": targets,
|
||||
"duration_ms": duration_ms,
|
||||
"request_ms": request_ms,
|
||||
"confirm_ms": barrier.get("confirm_ms"),
|
||||
"confirmed": barrier.get("confirmed"),
|
||||
"freed_gb": freed_gb,
|
||||
"residual_gb": round(barrier.get("residual_bytes", 0) / (1024**3), 2),
|
||||
"free_vram_gb": round(barrier.get("free_bytes", 0) / (1024**3), 2),
|
||||
"error": barrier.get("error"),
|
||||
}
|
||||
SWITCH_HISTORY.appendleft(event)
|
||||
return {"success": True, "model": model_name, "duration_ms": duration_ms}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
return {"success": False, "error": str(e),
|
||||
"duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
|
||||
async def instant_free_comfyui_vram() -> Dict[str, Any]:
|
||||
"""Tell ComfyUI to purge loaded diffusion models from VRAM."""
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
resp = await client.post(
|
||||
f"{COMFY_API_BASE}/free",
|
||||
json={"unload_models": True, "free_memory": True},
|
||||
)
|
||||
client = _client(COMFY_API_BASE, 5.0)
|
||||
await client.post("/free", json={"unload_models": True, "free_memory": True})
|
||||
duration_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
event = {
|
||||
"timestamp": time.strftime("%H:%M:%S"),
|
||||
snap = get_process_vram_bytes()
|
||||
_record({
|
||||
"event_type": "ComfyUI VRAM Purge",
|
||||
"source": "ComfyUI Pipeline",
|
||||
"target": "VRAM Free",
|
||||
"duration_ms": duration_ms,
|
||||
"cache_status": "Cleaned",
|
||||
}
|
||||
SWITCH_HISTORY.appendleft(event)
|
||||
return {"success": True, "duration_ms": duration_ms}
|
||||
})
|
||||
return {"success": True, "duration_ms": duration_ms,
|
||||
"free_vram_gb": round(snap["free_bytes"] / (1024**3), 2)}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
return {"success": False, "error": str(e),
|
||||
"duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
|
||||
_MODEL_SIZE_CACHE: Dict[str, int] = {}
|
||||
|
||||
|
||||
def _model_size_bytes(model_name: str) -> int:
|
||||
"""On-disk weight size for an Ollama model, used to turn load time into bandwidth."""
|
||||
if model_name in _MODEL_SIZE_CACHE:
|
||||
return _MODEL_SIZE_CACHE[model_name]
|
||||
try:
|
||||
for f in ram_optimizer.find_ollama_model_files():
|
||||
_MODEL_SIZE_CACHE[f["model"]] = f["size_bytes"]
|
||||
except Exception as e:
|
||||
logger.debug(f"model size lookup failed: {e}")
|
||||
return _MODEL_SIZE_CACHE.get(model_name, 0)
|
||||
|
||||
|
||||
def classify_load(size_bytes: int, load_duration_ms: float) -> Dict[str, Any]:
|
||||
"""Classify how a model reached VRAM, from achieved bandwidth rather than a constant.
|
||||
|
||||
The old rule was `load_duration_ms < 2500`, which called a 27B Q2_K read from NVMe a
|
||||
cache hit and a small model read from RAM a cold load. Bandwidth separates them
|
||||
cleanly: page cache feeds PCIe at many GB/s, this NVMe does not.
|
||||
"""
|
||||
if load_duration_ms <= 1.0:
|
||||
return {"cache_status": "Already in VRAM", "load_gbps": None, "is_ram_hit": True}
|
||||
if not size_bytes:
|
||||
# No size on record — fall back to the old heuristic, but say so.
|
||||
# Without a size we cannot compute bandwidth at all; this is a guess and is
|
||||
# labelled as one. 8s roughly splits the measured warm (4.9s) and cold (34.3s)
|
||||
# loads for a mid-size model, but it is meaningless for very small or large ones.
|
||||
return {
|
||||
"cache_status": "RAM Cache Hit ⚡" if load_duration_ms < 8000 else "Cold Disk Load 💾",
|
||||
"load_gbps": None,
|
||||
"is_ram_hit": load_duration_ms < 8000,
|
||||
"detail": "size unknown, fell back to a duration guess",
|
||||
}
|
||||
gbps = (size_bytes / (1024**3)) / (load_duration_ms / 1000.0)
|
||||
if gbps >= RAM_HIT_GBPS:
|
||||
status = "RAM Cache Hit ⚡"
|
||||
elif gbps >= PARTIAL_HIT_GBPS:
|
||||
status = "Partial Cache 🌤"
|
||||
else:
|
||||
status = "Cold Disk Load 💾"
|
||||
return {"cache_status": status, "load_gbps": round(gbps, 2), "is_ram_hit": gbps >= RAM_HIT_GBPS}
|
||||
|
||||
|
||||
async def switch_ollama_model(target_model: str, keep_alive: str = "30m") -> Dict[str, Any]:
|
||||
"""High-speed hot-swap to target Ollama model, tracking swap metrics."""
|
||||
@@ -299,13 +593,13 @@ async def switch_ollama_model(target_model: str, keep_alive: str = "30m") -> Dic
|
||||
prev_model = cur_state.get("active_model_name") or "None"
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=180.0) as client:
|
||||
client = _client(OLLAMA_API_BASE, 180.0)
|
||||
resp = await client.post(
|
||||
f"{OLLAMA_API_BASE}/api/generate",
|
||||
json={"model": target_model, "prompt": "Ready check", "stream": False, "keep_alive": keep_alive},
|
||||
"/api/generate",
|
||||
json={"model": target_model, "prompt": "Ready check", "stream": False,
|
||||
"keep_alive": keep_alive},
|
||||
)
|
||||
total_duration = time.perf_counter() - t0
|
||||
total_duration_ms = round(total_duration * 1000, 2)
|
||||
total_duration_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
@@ -314,20 +608,21 @@ async def switch_ollama_model(target_model: str, keep_alive: str = "30m") -> Dic
|
||||
eval_count = data.get("eval_count", 0)
|
||||
tokens_per_sec = round((eval_count / (eval_dur_ms / 1000)) if eval_dur_ms > 0 else 0, 1)
|
||||
|
||||
# Check if it was a RAM cache hit (load duration < 1500ms for large model indicates RAM hit)
|
||||
is_ram_hit = load_dur_ms < 2500
|
||||
size_bytes = _model_size_bytes(target_model)
|
||||
cls = classify_load(size_bytes, load_dur_ms)
|
||||
|
||||
event = {
|
||||
"timestamp": time.strftime("%H:%M:%S"),
|
||||
_record({
|
||||
"event_type": "LLM Model Switch",
|
||||
"source": prev_model,
|
||||
"target": target_model,
|
||||
"duration_ms": total_duration_ms,
|
||||
"load_duration_ms": load_dur_ms,
|
||||
"tokens_per_sec": tokens_per_sec,
|
||||
"cache_status": "RAM Cache Hit ⚡" if is_ram_hit else "Cold Disk Load 💾",
|
||||
}
|
||||
SWITCH_HISTORY.appendleft(event)
|
||||
"bytes_loaded": size_bytes,
|
||||
"load_gbps": cls["load_gbps"],
|
||||
"cache_status": cls["cache_status"],
|
||||
"detail": cls.get("detail"),
|
||||
})
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
@@ -336,31 +631,59 @@ async def switch_ollama_model(target_model: str, keep_alive: str = "30m") -> Dic
|
||||
"total_duration_ms": total_duration_ms,
|
||||
"load_duration_ms": load_dur_ms,
|
||||
"tokens_per_sec": tokens_per_sec,
|
||||
"is_ram_hit": is_ram_hit,
|
||||
"model_size_gb": round(size_bytes / (1024**3), 2) if size_bytes else None,
|
||||
"load_gbps": cls["load_gbps"],
|
||||
"cache_status": cls["cache_status"],
|
||||
"is_ram_hit": cls["is_ram_hit"],
|
||||
"response": data.get("response", ""),
|
||||
}
|
||||
else:
|
||||
return {"success": False, "error": f"HTTP {resp.status_code}: {resp.text}", "duration_ms": total_duration_ms}
|
||||
return {"success": False, "error": f"HTTP {resp.status_code}: {resp.text}",
|
||||
"duration_ms": total_duration_ms}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
return {"success": False, "error": str(e),
|
||||
"duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
|
||||
def get_switch_history() -> List[Dict[str, Any]]:
|
||||
return list(SWITCH_HISTORY)
|
||||
|
||||
|
||||
class AutoArbitrator:
|
||||
"""Real-time bidirectional background arbitrator for seamless Ollama <-> ComfyUI hot-swapping."""
|
||||
"""Real-time bidirectional background arbitrator for seamless Ollama <-> ComfyUI hot-swapping.
|
||||
|
||||
Two behavioural changes worth knowing about:
|
||||
|
||||
* ComfyUI's VRAM is no longer purged 1.5 s after every finished prompt. Iterating on
|
||||
a workflow is the common case, and purging between runs forced a full checkpoint
|
||||
reload each time. The purge now waits for COMFY_IDLE_PURGE_S of genuinely empty
|
||||
queue, and happens immediately only when Ollama actually needs the VRAM.
|
||||
* The watchdog no longer polls two ComfyUI endpoints every 300 ms. The WebSocket is
|
||||
the primary signal; polling is a fallback that runs at 1 Hz and only hits /queue,
|
||||
backing off further while the socket is healthy.
|
||||
"""
|
||||
|
||||
COMFY_IDLE_PURGE_S = 30.0
|
||||
WATCHDOG_INTERVAL_S = 1.0
|
||||
WATCHDOG_INTERVAL_WS_OK_S = 3.0
|
||||
|
||||
def __init__(self):
|
||||
self.running = False
|
||||
self.ws_task: Optional[asyncio.Task] = None
|
||||
self.poll_task: Optional[asyncio.Task] = None
|
||||
self.idle_task: Optional[asyncio.Task] = None
|
||||
self.last_yield_time = 0.0
|
||||
self.last_comfy_free_time = 0.0
|
||||
self.connected_ws = False
|
||||
self.last_action = "Idle"
|
||||
self.comfy_was_active = False
|
||||
self.comfy_idle_since: Optional[float] = None
|
||||
self.oc_profile = None
|
||||
self.pending_purge = False
|
||||
# While a tuning sweep is running, the arbitrator must not fight it: a ComfyUI
|
||||
# benchmark would otherwise trip trigger_comfy_priority, which reapplies the whole
|
||||
# 'comfy' profile and silently overwrites the clock the sweep is measuring.
|
||||
self.oc_suspended = False
|
||||
self.stats = {"yields": 0, "purges": 0, "yield_timeouts": 0, "deferred_purges": 0}
|
||||
|
||||
async def start(self):
|
||||
if self.running:
|
||||
@@ -368,24 +691,27 @@ class AutoArbitrator:
|
||||
self.running = True
|
||||
self.ws_task = asyncio.create_task(self._ws_listener())
|
||||
self.poll_task = asyncio.create_task(self._poll_watchdog())
|
||||
self.idle_task = asyncio.create_task(self._idle_purge_loop())
|
||||
logger.info("AutoArbitrator background engine started (Bidirectional).")
|
||||
# Apply the default (balanced) overclock profile on startup.
|
||||
try:
|
||||
await asyncio.get_event_loop().run_in_executor(None, overclock_manager.apply_profile, "balanced")
|
||||
await asyncio.get_running_loop().run_in_executor(
|
||||
None, overclock_manager.apply_profile, "balanced")
|
||||
self.oc_profile = "balanced"
|
||||
except Exception as e:
|
||||
logger.warning(f"Startup overclock apply failed: {e}")
|
||||
|
||||
async def stop(self):
|
||||
self.running = False
|
||||
if self.ws_task:
|
||||
self.ws_task.cancel()
|
||||
if self.poll_task:
|
||||
self.poll_task.cancel()
|
||||
for task in (self.ws_task, self.poll_task, self.idle_task):
|
||||
if task:
|
||||
task.cancel()
|
||||
await close_clients()
|
||||
logger.info("AutoArbitrator background engine stopped.")
|
||||
|
||||
async def trigger_comfy_priority(self, reason: str = "ComfyUI prompt detected"):
|
||||
"""Instantly yield Ollama VRAM to 0MB when ComfyUI needs to run diffusion models."""
|
||||
"""Yield Ollama's VRAM — and confirm it is gone — before diffusion allocates."""
|
||||
self.comfy_was_active = True
|
||||
self.comfy_idle_since = None
|
||||
self._apply_oc_profile("comfy")
|
||||
now = time.time()
|
||||
if now - self.last_yield_time < 1.0:
|
||||
@@ -394,51 +720,97 @@ class AutoArbitrator:
|
||||
ollama_state = await get_ollama_live_state()
|
||||
if ollama_state.get("active_model_name"):
|
||||
model = ollama_state["active_model_name"]
|
||||
logger.info(f"⚡ ComfyUI active ({reason}) -> Auto-yielding Ollama model '{model}' from VRAM...")
|
||||
logger.info(f"⚡ ComfyUI active ({reason}) -> Auto-yielding Ollama model '{model}'...")
|
||||
self.last_yield_time = time.time()
|
||||
res = await instant_free_ollama_vram(model)
|
||||
dur = res.get("duration_ms", 0)
|
||||
self.last_action = f"Auto-yielded '{model}' for ComfyUI ({dur}ms)"
|
||||
res = await instant_free_ollama_vram(model, confirm=True)
|
||||
self.stats["yields"] += 1
|
||||
if res.get("confirmed"):
|
||||
self.last_action = (f"Yielded '{model}' for ComfyUI in "
|
||||
f"{res.get('confirm_ms')}ms (confirmed {res.get('freed_gb')}GB free)")
|
||||
else:
|
||||
self.stats["yield_timeouts"] += 1
|
||||
self.last_action = (f"⚠ Yield of '{model}' NOT confirmed: "
|
||||
f"{res.get('residual_gb')}GB still held")
|
||||
logger.warning(f"VRAM yield barrier timed out: {res.get('error')}")
|
||||
logger.info(f"Ollama auto-yield completed: {res}")
|
||||
|
||||
async def trigger_comfy_completed(self):
|
||||
"""Purge ComfyUI VRAM cache when generation finishes, keeping VRAM 100% free for Ollama."""
|
||||
async def trigger_comfy_completed(self, immediate: bool = False):
|
||||
"""Mark the end of a generation. The actual purge is deferred unless forced."""
|
||||
self.comfy_was_active = False
|
||||
if self.comfy_idle_since is None:
|
||||
self.comfy_idle_since = time.time()
|
||||
self._apply_oc_profile("ollama")
|
||||
if immediate:
|
||||
await self._purge_comfy_now("Ollama needs VRAM")
|
||||
else:
|
||||
self.pending_purge = True
|
||||
self.stats["deferred_purges"] += 1
|
||||
self.last_action = (f"ComfyUI idle — holding its checkpoints for "
|
||||
f"{int(self.COMFY_IDLE_PURGE_S)}s in case you iterate")
|
||||
|
||||
async def _purge_comfy_now(self, reason: str):
|
||||
now = time.time()
|
||||
if now - self.last_comfy_free_time < 3.0:
|
||||
return
|
||||
self.last_comfy_free_time = now
|
||||
self.comfy_was_active = False
|
||||
self._apply_oc_profile("ollama")
|
||||
logger.info("⚡ ComfyUI finished generation -> Auto-purging ComfyUI VRAM cache for Ollama...")
|
||||
self.pending_purge = False
|
||||
logger.info(f"⚡ Purging ComfyUI VRAM cache ({reason})...")
|
||||
res = await instant_free_comfyui_vram()
|
||||
dur = res.get("duration_ms", 0)
|
||||
self.last_action = f"Auto-purged ComfyUI VRAM ({dur}ms) - Ready for Ollama"
|
||||
logger.info(f"ComfyUI auto-purge completed: {res}")
|
||||
self.stats["purges"] += 1
|
||||
self.last_action = f"Purged ComfyUI VRAM ({res.get('duration_ms')}ms) — {reason}"
|
||||
logger.info(f"ComfyUI purge completed: {res}")
|
||||
|
||||
async def _idle_purge_loop(self):
|
||||
"""Purge ComfyUI's VRAM only after a real idle gap, not between iterations."""
|
||||
while self.running:
|
||||
try:
|
||||
if self.pending_purge and self.comfy_idle_since and not self.comfy_was_active:
|
||||
idle_for = time.time() - self.comfy_idle_since
|
||||
if idle_for >= self.COMFY_IDLE_PURGE_S:
|
||||
await self._purge_comfy_now(
|
||||
f"idle {int(idle_for)}s")
|
||||
except Exception as e:
|
||||
logger.debug(f"idle purge loop error: {e}")
|
||||
await asyncio.sleep(2.0)
|
||||
|
||||
async def request_vram_for_ollama(self, needed_gb: float = 0.0) -> Dict[str, Any]:
|
||||
"""Called when Ollama needs VRAM now: purge ComfyUI immediately rather than waiting."""
|
||||
snap = get_process_vram_bytes()
|
||||
free_gb = snap["free_bytes"] / (1024**3)
|
||||
if needed_gb and free_gb >= needed_gb:
|
||||
return {"purged": False, "free_gb": round(free_gb, 2), "reason": "enough free VRAM"}
|
||||
if snap["comfyui_bytes"] > YIELD_RESIDUAL_BYTES:
|
||||
await self._purge_comfy_now(f"Ollama requested {needed_gb or '?'}GB")
|
||||
snap = get_process_vram_bytes()
|
||||
return {"purged": True, "free_gb": round(snap["free_bytes"] / (1024**3), 2)}
|
||||
return {"purged": False, "free_gb": round(free_gb, 2), "reason": "ComfyUI holds no VRAM"}
|
||||
|
||||
async def _ws_listener(self):
|
||||
client_id = "hyperswap-arbitrator"
|
||||
ws_url = f"ws://127.0.0.1:8188/ws?clientId={client_id}"
|
||||
backoff = 2.0
|
||||
|
||||
while self.running:
|
||||
try:
|
||||
async with websockets.connect(ws_url, ping_interval=10, ping_timeout=10) as ws:
|
||||
self.connected_ws = True
|
||||
backoff = 2.0
|
||||
logger.info("AutoArbitrator connected to ComfyUI WebSocket.")
|
||||
while self.running:
|
||||
msg = await ws.recv()
|
||||
if isinstance(msg, str):
|
||||
if not isinstance(msg, str):
|
||||
continue
|
||||
try:
|
||||
data = json.loads(msg)
|
||||
msg_type = data.get("type")
|
||||
msg_data = data.get("data", {})
|
||||
|
||||
if msg_type == "status":
|
||||
queue_rem = msg_data.get("status", {}).get("exec_info", {}).get("queue_remaining", 0)
|
||||
queue_rem = (msg_data.get("status", {})
|
||||
.get("exec_info", {}).get("queue_remaining", 0))
|
||||
if queue_rem > 0:
|
||||
await self.trigger_comfy_priority(f"Queue remaining: {queue_rem}")
|
||||
elif queue_rem == 0 and self.comfy_was_active:
|
||||
# Prompt queue finished
|
||||
await asyncio.sleep(1.5)
|
||||
await self.trigger_comfy_completed()
|
||||
elif msg_type in ("execution_start", "execution_cached"):
|
||||
await self.trigger_comfy_priority(f"Event: {msg_type}")
|
||||
@@ -446,12 +818,12 @@ class AutoArbitrator:
|
||||
node = msg_data.get("node")
|
||||
if node is not None:
|
||||
await self.trigger_comfy_priority(f"Executing node: {node}")
|
||||
elif node is None and self.comfy_was_active:
|
||||
# Finished executing graph
|
||||
await asyncio.sleep(1.5)
|
||||
elif self.comfy_was_active:
|
||||
await self.trigger_comfy_completed()
|
||||
elif msg_type == "execution_success":
|
||||
await asyncio.sleep(1.5)
|
||||
await self.trigger_comfy_completed()
|
||||
elif msg_type == "execution_error":
|
||||
logger.warning(f"ComfyUI execution error: {msg_data}")
|
||||
await self.trigger_comfy_completed()
|
||||
except Exception as e:
|
||||
logger.debug(f"WS parse error: {e}")
|
||||
@@ -461,41 +833,66 @@ class AutoArbitrator:
|
||||
self.connected_ws = False
|
||||
logger.debug(f"WS connection error: {e}")
|
||||
|
||||
await asyncio.sleep(2.0)
|
||||
await asyncio.sleep(backoff)
|
||||
backoff = min(backoff * 1.5, 15.0)
|
||||
|
||||
async def _poll_watchdog(self):
|
||||
"""Watchdog polling /queue every 300ms for robust bidirectional arbitration."""
|
||||
"""Fallback for when the WebSocket is down. One cheap /queue call, 1 Hz.
|
||||
|
||||
The previous version hit /system_stats and /queue every 300 ms on fresh TCP
|
||||
connections — roughly 6.6 requests/second against ComfyUI, forever.
|
||||
"""
|
||||
while self.running:
|
||||
interval = self.WATCHDOG_INTERVAL_WS_OK_S if self.connected_ws else self.WATCHDOG_INTERVAL_S
|
||||
try:
|
||||
comfy_state = await get_comfyui_live_state()
|
||||
is_executing = comfy_state.get("queue_running", 0) > 0 or comfy_state.get("queue_remaining", 0) > 0 or comfy_state.get("executing", False)
|
||||
if is_executing:
|
||||
await self.trigger_comfy_priority("Polling detected active queue/execution")
|
||||
elif self.comfy_was_active and not is_executing:
|
||||
await asyncio.sleep(1.5)
|
||||
client = _client(COMFY_API_BASE, 3.0)
|
||||
resp = await client.get("/queue")
|
||||
if resp.status_code == 200:
|
||||
q = resp.json()
|
||||
busy = len(q.get("queue_running", [])) > 0 or len(q.get("queue_pending", [])) > 0
|
||||
if busy:
|
||||
await self.trigger_comfy_priority("Watchdog saw an active queue")
|
||||
elif self.comfy_was_active:
|
||||
await self.trigger_comfy_completed()
|
||||
except Exception:
|
||||
pass
|
||||
await asyncio.sleep(0.3)
|
||||
await asyncio.sleep(interval)
|
||||
|
||||
def suspend_oc(self, reason: str = "tuning sweep") -> None:
|
||||
self.oc_suspended = True
|
||||
logger.info(f"Overclock auto-switching suspended ({reason})")
|
||||
|
||||
def resume_oc(self, profile: Optional[str] = None) -> None:
|
||||
self.oc_suspended = False
|
||||
# Forget the cached profile so the next transition actually reapplies.
|
||||
self.oc_profile = profile
|
||||
logger.info("Overclock auto-switching resumed")
|
||||
|
||||
def _apply_oc_profile(self, profile: str):
|
||||
"""Apply an overclock profile in a background thread; only fire on transition."""
|
||||
if self.oc_profile == profile:
|
||||
if self.oc_suspended or self.oc_profile == profile:
|
||||
return
|
||||
self.oc_profile = profile
|
||||
try:
|
||||
loop = asyncio.get_event_loop()
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.run_in_executor(None, overclock_manager.apply_profile, profile)
|
||||
logger.info(f"🎛️ Overclock profile switched -> '{profile}'")
|
||||
except Exception as e:
|
||||
logger.warning(f"Overclock profile switch failed ({profile}): {e}")
|
||||
|
||||
def get_status(self) -> Dict[str, Any]:
|
||||
idle_for = (time.time() - self.comfy_idle_since) if self.comfy_idle_since else None
|
||||
return {
|
||||
"running": self.running,
|
||||
"connected_ws": self.connected_ws,
|
||||
"last_action": self.last_action,
|
||||
"mode": "Bidirectional Hot-Swap (ComfyUI <-> Ollama)",
|
||||
"comfy_active": self.comfy_was_active,
|
||||
"pending_purge": self.pending_purge,
|
||||
"comfy_idle_s": round(idle_for, 1) if idle_for is not None else None,
|
||||
"idle_purge_after_s": self.COMFY_IDLE_PURGE_S,
|
||||
"oc_profile": self.oc_profile,
|
||||
"counters": dict(self.stats),
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user