From 5431144b2e97088a4ee201e11bc8e2819a711791 Mon Sep 17 00:00:00 2001 From: drjones Date: Fri, 28 Aug 2026 08:57:35 -0700 Subject: [PATCH 1/6] Add barrier-confirmed yielding, measured residency, persistence and closed-loop tuning Nine changes, in rough order of how much they affect real behaviour: 1. VRAM yield is now a barrier. Posting keep_alive:0 only asks Ollama to unload; measured here, the HTTP call returns in 63ms while the driver takes a further 77ms to release 14.9GB. Returning inside that window is how ComfyUI ends up allocating into VRAM that is still occupied. instant_free_ollama_vram() polls NVML until the allocation is actually gone and reports request/confirm split. 2. ComfyUI VRAM is no longer purged 1.5s after every prompt, which forced a full checkpoint reload on each workflow iteration. It is held for 30s of genuinely empty queue, with an immediate purge when Ollama actually asks for the memory. 3. Cache-hit classification uses achieved bandwidth (size / load duration) rather than a fixed `load_duration < 2500ms`. That constant called a 12.9GB model read at 2.9GB/s a cold load, and a 0.5GB model read from NVMe a cache hit. 4. Page-cache residency is measured, not assumed. mincore(2) reported 128GB resident on a box with 46GB of page cache: the kernel only permits page-cache introspection on files you own, and the Ollama blobs are owned by uid ollama, for which mincore answers "all resident" instead of failing. Uses cachestat(2) where permitted and a randomised read-rate probe elsewhere, labelling which was used. Fixed-offset probing was self-fulfilling, so windows are random and cold ones are returned with FADV_DONTNEED. 5. Warming is budgeted and ranked by recency/frequency instead of reading every file top-to-bottom, which on 64GB of RAM just evicts whatever was warmed first. 6. Telemetry and events persist to SQLite (~0.38 MB/hour) instead of living in a 50-entry in-memory deque, so /api/analytics/profiles can finally answer whether an overclock profile actually delivers more tok/s. 7. Thermal governor walks the overclock back on sustained heat or hardware throttling, with hysteresis, fed from the existing sampler. 8. Autotune sweeps a clock offset, benchmarks decode at each step, watches for Xid errors and degenerate output, and restores the profile in a finally block. 9. Stock clocks/power/fans are restored on shutdown and via systemd ExecStopPost. Nothing previously undid a locked clock or a manually pinned fan. Also: one shared 1Hz telemetry sampler fanned out to SSE subscribers rather than every client re-running the whole snapshot; wall-clock timestamps in place of the event loop's monotonic clock; cached nvidia-smi shell-outs; quieter httpx logging. Co-Authored-By: Claude Opus 5 --- .gitignore | 5 + README.md | 63 +++- autotune.py | 263 +++++++++++++++ overclock_manager.py | 71 +++- overclock_profiles.json | 8 +- ram_optimizer.py | 658 +++++++++++++++++++++++++++++++----- server.py | 408 ++++++++++++++++++----- static/app.js | 231 +++++++++++++ static/index.html | 138 ++++++++ telemetry_store.py | 400 ++++++++++++++++++++++ thermal_governor.py | 175 ++++++++++ vram_arbitrator.py | 721 ++++++++++++++++++++++++++++++---------- 12 files changed, 2779 insertions(+), 362 deletions(-) create mode 100644 autotune.py create mode 100644 telemetry_store.py create mode 100644 thermal_governor.py diff --git a/.gitignore b/.gitignore index bee8803..aa003ec 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,8 @@ __pycache__/ .venv/ venv/ .DS_Store + +# persistent telemetry store +hyperswap.db +hyperswap.db-wal +hyperswap.db-shm diff --git a/README.md b/README.md index 02390db..67df288 100644 --- a/README.md +++ b/README.md @@ -20,17 +20,18 @@ ## 1. Feature Matrix ### โšก Bidirectional VRAM Hot-Swapping & Arbitration -* **Sub-25ms Soft-Yield**: Instantly releases Ollama VRAM allocations (`keep_alive: 0`) down to 0 MB when ComfyUI needs to run diffusion workloads without evicting weights from system RAM. -* **Auto-Purge for ComfyUI**: Automatically purges diffusion pipeline checkpoints and VRAM buffers when an image/video generation job finishes, releasing 100% of VRAM back to Ollama. -* **Real-Time ComfyUI WebSocket & Watchdog Listener**: Subscribes directly to `ws://127.0.0.1:8188/ws` and runs a 300ms watchdog loop to detect prompt queueing and node execution in real time. +* **Confirmed Soft-Yield (barrier, not fire-and-forget)**: Releases Ollama VRAM allocations (`keep_alive: 0`) down to 0 MB, then **waits on NVML until the driver has actually freed the allocation** before letting ComfyUI proceed. Posting `keep_alive: 0` only *asks* Ollama to unload; on this box the HTTP call returns in ~63 ms while the driver takes a further ~77 ms to release 14.9 GB. Returning during that window is how diffusion ends up allocating into VRAM that is still occupied. +* **Idle-Aware ComfyUI Purge**: Diffusion checkpoints are held for `COMFY_IDLE_PURGE_S` (30 s) of genuinely empty queue rather than purged 1.5 s after every prompt โ€” iterating on a workflow no longer pays a full checkpoint reload per run. An immediate purge still happens the moment Ollama actually asks for VRAM (`POST /api/request-vram`). +* **Real-Time ComfyUI WebSocket & Watchdog Listener**: Subscribes directly to `ws://127.0.0.1:8188/ws`. The WebSocket is the primary signal; a connection-pooled watchdog polls `/queue` at 1 Hz purely as a fallback, backing off to 3 s while the socket is healthy. * **Process-Level VRAM Attribution**: Live NVML process inspection attributes exact GPU memory usage across Ollama (`llama-server`), ComfyUI (`python`), and Desktop display servers (`gnome-shell`, `Xorg`). -* **Hot-Swap Transition History**: Circular buffer logs all model switch events, swap durations (in ms), tokens/sec throughput, and RAM cache hit status (`RAM Cache Hit โšก` vs `Cold Disk Load ๐Ÿ’พ`). +* **Bandwidth-Classified Transition History**: Every switch is classified by the bandwidth it actually achieved (`model size รท load duration`) rather than a fixed duration threshold: `RAM Cache Hit โšก` (โ‰ฅ5 GB/s), `Partial Cache ๐ŸŒค` (โ‰ฅ1.5 GB/s), `Cold Disk Load ๐Ÿ’พ` (below that). The previous `load_duration < 2500 ms` rule called a 12.9 GB model read at 2.9 GB/s a "cold disk load" and a 0.5 GB model read from NVMe a "cache hit". ### ๐Ÿง  64GB Host RAM Cache & Page Pre-warmer * **Zero-Latency Model Discovery**: Automatic cataloging of all local Ollama models (`/usr/share/ollama/.ollama/models`, `~/.ollama/models`) and ComfyUI model directories (`checkpoints`, `diffusion_models`, `unet`, `vae`, `clip`, `loras`, `controlnet`). -* **POSIX `fadvise` & Pinned Pre-warmer**: Pre-faults multi-gigabyte GGUFs and Safetensors into the Linux OS Page Cache so that reloading models across PCIe 4.0 x16 runs at ~31.5 GB/s (sub-second VRAM loads). -* **Granular Pre-warming Controls**: Pre-warm all discovered models in bulk or target individual models/safetensors on demand. -* **Memory Telemetry**: Real-time breakdown of Total Host RAM, Applications Memory, Active Model Page Cache, Free Memory, and Cache Residency Ratio. +* **POSIX `fadvise` & Pinned Pre-warmer**: Pre-faults multi-gigabyte GGUFs and Safetensors into the Linux OS Page Cache so that reloading models runs at page-cache speed rather than disk speed. +* **Measured Residency via `cachestat(2)`**: Residency is measured, not assumed. `cachestat(2)` gives exact cached-page counts per file. Where the kernel refuses it โ€” it only permits introspection of files you own, and Ollama's blobs are owned by uid `ollama` โ€” HyperSwap falls back to a randomised read-rate probe and labels the result as such. Files it cannot measure are reported as unmeasurable rather than guessed at. +* **Budgeted, Ranked Warming**: This box has 64 GB of RAM and >270 GB of model files; reading everything simply evicts whatever was warmed first. Files are ranked by recency/frequency (from the persisted event log) and warmed until a byte budget is spent, skipping anything already resident. `GET /api/warm-plan` previews the decision without executing it. +* **Memory Telemetry**: Real-time breakdown of Total Host RAM, Applications Memory, Active Model Page Cache, Free Memory, and measured Cache Residency Ratio. ### ๐ŸŽ›๏ธ Dynamic Overclocking & Thermal Management * **Workload-Aware Overclock Profiles**: @@ -44,11 +45,25 @@ * **Hardware Fan Control**: Switch between `auto` and `manual` PWM control (30%โ€“100%) with synchronized dual-fan actuation (`[fan:0]` and `[fan:1]`). * **Automated Lockstep Profile Switching**: AutoArbitrator automatically switches hardware profiles in lockstep with the active workload (`comfy` on generation start, `ollama` on completion). +### ๐ŸŒก๏ธ Thermal Governor (closed-loop de-escalation) +* Every overclock lever here is sticky: a profile locks clocks and pins the fans to a manual PWM, and nothing used to undo that. The governor watches the telemetry the sampler already collects (so it costs no extra NVML calls) and walks the overclock back through a four-step derate ladder when the card runs hot or reports a hardware throttle. +* **Hysteresis by design**: escalation needs 5 consecutive bad samples, recovery needs 30 consecutive good ones, with a 20 s cooldown between changes โ€” a single spike during a diffusion step will not cause profile thrash. +* **Guaranteed restore**: stock clocks, default power limit and automatic fans are restored by the server's shutdown hook *and* by a systemd `ExecStopPost=`, so a `SIGKILL` cannot leave the card with locked clocks and fans pinned at 100%. + +### ๐Ÿ”ฌ Overclock Autotune (`autotune.py`) +* Walks a clock offset upward, running a fixed decode benchmark at each step, and reports the **fastest stable** value with its measured gain over baseline. +* **Instability detection**: kernel `Xid`/`NVRM` messages via `journalctl -k`, benchmark failure, degenerate output, and a temperature ceiling. The sweep stops climbing the moment a step looks unstable. +* **Safety**: refuses to start while ComfyUI is executing, and restores the original profile in a `finally` block โ€” including on exception or cancellation. + +### ๐Ÿ—„๏ธ Persistent Telemetry Store (`telemetry_store.py`) +* Swap history used to be an in-memory `deque(maxlen=50)` that evaporated on every restart. Telemetry and events now persist to SQLite (WAL, single writer thread, batched 1 Hz inserts, automatic retention pruning) at roughly **0.4 MB per hour**. +* This is what makes the app's central question answerable: **`GET /api/analytics/profiles` compares decode throughput per overclock profile**, joined against the thermals recorded while that profile was active. + ### ๐Ÿ“Š Real-Time Web Telemetry Dashboard (`:9090`) * **Live Hardware Telemetry**: GPU utilization %, GPU temperature (ยฐC), power draw (W), fan speeds (%), and graphics/memory clock frequencies (MHz). * **Live Dual-Axis Time-Series Chart**: Real-time graphical visualization of VRAM usage (GB) and Host RAM Cache (GB) with zero frontend polling overhead. * **Interactive Control Center**: Trigger model hot-swaps, soft-yields, cache pre-warms, fan adjustments, and benchmarks directly from the web interface. -* **Server-Sent Events (SSE)**: Pushes unified 1Hz telemetry updates via `GET /api/stream`. +* **Server-Sent Events (SSE)**: A single background sampler produces one 1 Hz snapshot and fans it out to every subscriber via `GET /api/stream`. Previously each connected client independently re-ran the whole snapshot โ€” NVML, `/proc/meminfo`, an HTTP round-trip each to Ollama and ComfyUI, and a recursive walk of the ComfyUI models tree with a `stat()` per checkpoint โ€” once per second, so opening the dashboard in three tabs tripled the load on the thing it was measuring. Slow clients drop stale frames instead of stalling the sampler. ### ๐Ÿค– Model Context Protocol (MCP 2.0) Server * **12 Native Agentic Tools**: Allows AI agents (Antigravity CLI, Claude Desktop, Cursor) to manage GPU resources, trigger model hot-swaps, tune fan curves, and inspect telemetry. @@ -92,7 +107,7 @@ flowchart TD ### The Physics of Sub-Second Switching * **Host RAM as Staging**: Active LLMs and diffusion checkpoints remain resident in the 64GB Linux Page Cache. * **PCIe 4.0 x16 Hot-Swapping**: Transferring weights across PCIe 4.0 x16 achieves **~31.5 GB/s** bandwidth, reducing model loads from 30+ seconds (disk) to **under 1.5 seconds**. -* **Soft-Yielding**: Dropping Ollama's VRAM allocation via `keep_alive: 0` takes **~15ms** while preserving the weights in host RAM. +* **Soft-Yielding**: Dropping Ollama's VRAM allocation via `keep_alive: 0` preserves the weights in host RAM. Measured on this box: the HTTP request returns in **~63 ms**, and the driver finishes releasing 14.9 GB **~77 ms after that**. HyperSwap waits for the second number before handing VRAM to ComfyUI โ€” the earlier "~15 ms" figure timed the request, not the release. --- @@ -119,12 +134,36 @@ The HyperSwap server runs on port `9090` by default. Interactive OpenAPI/Swagger | Endpoint | Method | Description | | :--- | :--- | :--- | | `/api/switch-model` | `POST` | Hot-swaps the active Ollama LLM in VRAM and tracks transition timing. | -| `/api/free-vram` | `POST` | Instructs Ollama to soft-yield VRAM down to 0 MB in ~15ms while retaining RAM cache. | +| `/api/free-vram` | `POST` | Soft-yields Ollama VRAM to 0 MB and **waits for NVML to confirm the release** (`?confirm=false` to skip). Returns `request_ms`, `confirm_ms` and the GB actually freed. | | `/api/comfy-free` | `POST` | Instructs ComfyUI to purge loaded diffusion weights and VRAM cache. | -| `/api/warm-all` | `POST` | Pre-faults all installed Ollama models and ComfyUI Safetensors into the Linux page cache. | -| `/api/warm-model` | `POST` | Pre-warms a specific model or file into RAM. | +| `/api/request-vram` | `POST` | Ollama-priority path: purges ComfyUI immediately if there is not enough free VRAM. | +| `/api/warm-all` | `POST` | Warms the highest-value models into page cache within a byte budget (`budget_gb`). | +| `/api/warm-plan` | `GET` | Previews what warming would read, in what order, and what it would skip โ€” without doing it. | +| `/api/warm-model` | `POST` | Pre-warms a specific model or file into RAM (`blob_only` warms weights without touching VRAM). | +| `/api/cache/report` | `GET` | Measured page-cache residency per model file, with the measurement method used for each. | | `/api/benchmark` | `POST` | Runs an automated back-and-forth model swap benchmark and calculates average latency. | +### Analytics Endpoints (persisted) + +| Endpoint | Method | Description | +| :--- | :--- | :--- | +| `/api/analytics/profiles` | `GET` | **Decode throughput per overclock profile**, joined with the thermals recorded under it. | +| `/api/analytics/swaps` | `GET` | Aggregated swap/yield/purge latencies, cache-hit split, and per-model throughput. | +| `/api/analytics/timeseries` | `GET` | Downsampled telemetry history for charts that outlive a page refresh. | +| `/api/analytics/models` | `GET` | Recency/frequency model ranking used to prioritise the warm budget. | +| `/api/history?durable=true` | `GET` | Swap history from the persistent store rather than the in-memory ring. | +| `/api/db` | `GET` | Store location, row counts and how many hours of history are held. | + +### Governor & Autotune Endpoints + +| Endpoint | Method | Description | +| :--- | :--- | :--- | +| `/api/governor` | `GET` / `POST` | Current derate level and why; enable/disable, or clear an active derate. | +| `/api/overclock/restore` | `POST` | Drop all clock locks and offsets, restore default power limit and automatic fans. | +| `/api/autotune` | `GET` | Sweep progress, last result, and every recorded autotune step. | +| `/api/autotune/sweep` | `POST` | Walk a clock offset upward, measuring tok/s and watching for instability at each step. | +| `/api/autotune/cancel` | `POST` | Stop the current sweep after the step in flight; the profile is restored either way. | + --- ## 4. Model Context Protocol (MCP 2.0) Reference diff --git a/autotune.py b/autotune.py new file mode 100644 index 0000000..c819096 --- /dev/null +++ b/autotune.py @@ -0,0 +1,263 @@ +"""Closed-loop overclock autotuner. + +The profiles in this repo were hand-tuned and had already drifted apart from the defaults +in overclock_manager.py, with no record of which numbers were actually faster. This module +answers that empirically: it walks a clock offset upward, measures real decode throughput +at each step, watches for instability, and reports the highest setting that was both +stable and fastest. + +Safety properties: + * The original profile is always restored, including on exception or cancellation. + * A sweep refuses to start while ComfyUI is executing, so it cannot corrupt someone's + render by yanking clocks mid-graph. + * Every step is bounded by a temperature ceiling and checked for kernel Xid messages, + and the sweep stops climbing the moment a step looks unstable. +""" +import asyncio +import logging +import subprocess +import time +from typing import Any, Dict, List, Optional + +import overclock_manager +import telemetry_store +import vram_arbitrator + +logger = logging.getLogger("autotune") + +BENCH_PROMPT = ("Write a detailed technical explanation of how virtual memory paging " + "works in a modern operating system kernel.") +BENCH_TOKENS = 160 +SETTLE_S = 2.5 +TEMP_CEILING_C = 84.0 + +KNOBS = { + "mem_offset_mhz": {"default_start": 0, "default_stop": 1000, "default_step": 100}, + "core_offset_mhz": {"default_start": 0, "default_stop": 300, "default_step": 25}, +} + + +def _xid_since(since_ts: float) -> List[str]: + """Look for NVIDIA Xid errors in the kernel log โ€” the clearest instability signal.""" + try: + since = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(since_ts)) + proc = subprocess.run( + ["journalctl", "-k", "--since", since, "--no-pager", "-q"], + capture_output=True, text=True, timeout=10, + ) + return [ln.strip() for ln in proc.stdout.splitlines() + if "Xid" in ln or "NVRM:" in ln] + except Exception as e: + logger.debug(f"Xid check unavailable: {e}") + return [] + + +async def _decode_benchmark(model: str) -> Dict[str, Any]: + """One fixed decode run. Throughput here is the thing being optimised.""" + client = vram_arbitrator._client(vram_arbitrator.OLLAMA_API_BASE, 300.0) + t0 = time.perf_counter() + resp = await client.post("/api/generate", json={ + "model": model, + "prompt": BENCH_PROMPT, + "stream": False, + "keep_alive": "10m", + "options": {"num_predict": BENCH_TOKENS, "temperature": 0.0, "seed": 42}, + }) + wall_ms = round((time.perf_counter() - t0) * 1000, 2) + if resp.status_code != 200: + return {"ok": False, "error": f"HTTP {resp.status_code}: {resp.text[:200]}", + "wall_ms": wall_ms} + data = resp.json() + eval_ms = data.get("eval_duration", 0) / 1e6 + eval_count = data.get("eval_count", 0) + text = data.get("response", "") or "" + return { + "ok": True, + "tokens_per_sec": round(eval_count / (eval_ms / 1000), 2) if eval_ms > 0 else 0.0, + "eval_count": eval_count, + "eval_ms": round(eval_ms, 2), + "prompt_eval_ms": round(data.get("prompt_eval_duration", 0) / 1e6, 2), + "wall_ms": wall_ms, + "response_chars": len(text), + # A model producing almost nothing, or pure repetition, is a corruption signal. + "degenerate": eval_count < BENCH_TOKENS * 0.5 or len(set(text.split())) < 8, + } + + +class SweepState: + def __init__(self) -> None: + self.running = False + self.cancel = False + self.current: Optional[Dict[str, Any]] = None + self.last_result: Optional[Dict[str, Any]] = None + + +state = SweepState() + + +async def sweep(knob: str = "mem_offset_mhz", + profile: str = "ollama", + model: Optional[str] = None, + start: Optional[int] = None, + stop: Optional[int] = None, + step: Optional[int] = None, + repeats: int = 1, + apply_best: bool = False) -> Dict[str, Any]: + """Sweep one clock offset and return the fastest stable value.""" + if knob not in KNOBS: + return {"success": False, "error": f"unknown knob '{knob}'; try {list(KNOBS)}"} + if state.running: + return {"success": False, "error": "a sweep is already running"} + + comfy = await vram_arbitrator.get_comfyui_live_state() + if comfy.get("executing") or comfy.get("queue_remaining"): + return {"success": False, "error": "ComfyUI is busy; refusing to change clocks mid-render"} + + if not model: + ollama = await vram_arbitrator.get_ollama_live_state() + model = ollama.get("active_model_name") + if not model: + installed = ollama.get("installed_models") or [] + if not installed: + return {"success": False, "error": "no Ollama model available to benchmark"} + model = installed[0].get("name") + + defaults = KNOBS[knob] + start = defaults["default_start"] if start is None else start + stop = defaults["default_stop"] if stop is None else stop + step = defaults["default_step"] if step is None else step + if step <= 0 or stop < start: + return {"success": False, "error": "invalid sweep range"} + + baseline_cfg = overclock_manager.load_profiles().get(profile, {}) + state.running = True + state.cancel = False + results: List[Dict[str, Any]] = [] + t_start = time.time() + + try: + # Load the model once up front so the first step does not pay the load cost. + await _decode_benchmark(model) + + value = start + while value <= stop and not state.cancel: + overclock_manager.apply_profile(profile, overrides={knob: value}) + await asyncio.sleep(SETTLE_S) + step_started = time.time() + + samples = [] + for _ in range(max(repeats, 1)): + samples.append(await _decode_benchmark(model)) + if state.cancel: + break + + gpu = vram_arbitrator.get_gpu_hardware_stats() + xids = _xid_since(step_started) + ok_samples = [s for s in samples if s.get("ok") and not s.get("degenerate")] + temp = gpu.get("temperature_c", 0) or 0 + + instability = [] + if xids: + instability.append(f"kernel Xid: {xids[0][:120]}") + if len(ok_samples) < len(samples): + instability.append("benchmark failed or produced degenerate output") + if temp >= TEMP_CEILING_C: + instability.append(f"temperature ceiling hit ({temp}ยฐC)") + + tok_s = round(max((s["tokens_per_sec"] for s in ok_samples), default=0.0), 2) + row = { + "knob": knob, + "value": value, + "profile": profile, + "model": model, + "tokens_per_sec": tok_s, + "temp_c": temp, + "power_w": gpu.get("power_w"), + "clock_sm_mhz": gpu.get("clock_graphics_mhz"), + "clock_mem_mhz": gpu.get("clock_mem_mhz"), + "throttle_reasons": gpu.get("throttle_reasons"), + "stable": not instability, + "instability": "; ".join(instability) or None, + "samples": samples, + } + results.append(row) + telemetry_store.record_autotune({ + "profile": profile, "knob": knob, + "core_offset_mhz": value if knob == "core_offset_mhz" else baseline_cfg.get("core_offset_mhz"), + "mem_offset_mhz": value if knob == "mem_offset_mhz" else baseline_cfg.get("mem_offset_mhz"), + "tokens_per_sec": tok_s, "temp_c": temp, "power_w": gpu.get("power_w"), + "stable": row["stable"], "instability": row["instability"], + "note": f"sweep {knob} {start}..{stop} step {step}", + }) + state.current = {"knob": knob, "value": value, "stop": stop, + "tokens_per_sec": tok_s, "stable": row["stable"]} + logger.info(f"autotune {knob}={value}: {tok_s} tok/s, {temp}ยฐC, " + f"stable={row['stable']} {row['instability'] or ''}") + + if not row["stable"]: + logger.warning(f"autotune stopping climb at {knob}={value}: {row['instability']}") + break + value += step + + stable = [r for r in results if r["stable"] and r["tokens_per_sec"] > 0] + best = max(stable, key=lambda r: r["tokens_per_sec"]) if stable else None + baseline = next((r for r in results if r["value"] == start), None) + gain_pct = None + if best and baseline and baseline["tokens_per_sec"] > 0: + gain_pct = round((best["tokens_per_sec"] / baseline["tokens_per_sec"] - 1) * 100, 2) + + applied = None + if apply_best and best: + overclock_manager.set_profile(profile, {knob: best["value"]}) + applied = {knob: best["value"], "profile": profile} + logger.info(f"autotune wrote {knob}={best['value']} into profile '{profile}'") + + result = { + "success": True, + "knob": knob, + "profile": profile, + "model": model, + "range": {"start": start, "stop": stop, "step": step}, + "steps_run": len(results), + "duration_s": round(time.time() - t_start, 1), + "cancelled": state.cancel, + "best": {k: best[k] for k in ("value", "tokens_per_sec", "temp_c", "clock_mem_mhz", + "clock_sm_mhz")} if best else None, + "baseline_tokens_per_sec": baseline["tokens_per_sec"] if baseline else None, + "gain_pct": gain_pct, + "applied_to_profile": applied, + "first_unstable": next(({"value": r["value"], "why": r["instability"]} + for r in results if not r["stable"]), None), + "table": [{k: r[k] for k in ("value", "tokens_per_sec", "temp_c", "power_w", + "clock_mem_mhz", "clock_sm_mhz", "stable", + "instability")} for r in results], + } + state.last_result = result + return result + finally: + # Always hand the card back exactly as we found it. + state.running = False + state.current = None + try: + overclock_manager.apply_profile(profile) + logger.info(f"autotune restored profile '{profile}'") + except Exception as e: + logger.error(f"autotune failed to restore profile, forcing stock: {e}") + overclock_manager.restore_safe("autotune restore failed") + + +def get_status() -> Dict[str, Any]: + return { + "running": state.running, + "current": state.current, + "last_result": state.last_result, + "knobs": KNOBS, + "history": telemetry_store.autotune_history(100), + } + + +def cancel() -> Dict[str, Any]: + if not state.running: + return {"cancelled": False, "reason": "no sweep running"} + state.cancel = True + return {"cancelled": True} diff --git a/overclock_manager.py b/overclock_manager.py index 5f5d3b0..1fb7368 100644 --- a/overclock_manager.py +++ b/overclock_manager.py @@ -207,14 +207,20 @@ def _apply_offsets(core_mhz: int, mem_mhz: int) -> Dict[str, Any]: } -def apply_profile(name: str) -> Dict[str, Any]: - """Apply a named overclock profile to the GPU. Returns a full result report.""" +def apply_profile(name: str, overrides: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: + """Apply a named overclock profile to the GPU. Returns a full result report. + + `overrides` lets the thermal governor and the autotuner apply a modified version of a + profile (a derated offset, a probe clock) without mutating what is stored on disk. + """ global ACTIVE_PROFILE, _LAST_RESULT profiles = load_profiles() if name not in profiles: return {"success": False, "error": f"unknown profile '{name}'", "profile": name} - cfg = profiles[name] + cfg = dict(profiles[name]) + if overrides: + cfg.update(overrides) fan_mode = cfg.get("fan_mode", "auto") fan_speed = int(cfg.get("fan_speed_pct", 0)) @@ -230,15 +236,31 @@ def apply_profile(name: str) -> Dict[str, Any]: } result["gpu"] = get_gpu_state() result["fan_status"] = get_fan_status() + result["overrides"] = overrides or {} + _STATE_CACHE["value"] = None + _FAN_CACHE["value"] = None ACTIVE_PROFILE = name _LAST_RESULT = result logger.info(f"Overclock profile applied: {name} -> {json.dumps(result, default=str)}") return result -def get_gpu_state() -> Dict[str, Any]: - """Read back live GPU clocks/power/limits via nvidia-smi.""" +_STATE_CACHE: Dict[str, Any] = {"ts": 0.0, "value": None} +_FAN_CACHE: Dict[str, Any] = {"ts": 0.0, "value": None} +STATE_TTL_S = 2.0 + + +def get_gpu_state(force: bool = False) -> Dict[str, Any]: + """Read back live GPU clocks/power/limits via nvidia-smi. + + Cached for STATE_TTL_S: this forks `sudo nvidia-smi`, and the dashboard polls the + status endpoint every few seconds. NVML already covers the live 1 Hz telemetry. + """ + import time as _time + if not force and _STATE_CACHE["value"] is not None and \ + (_time.time() - _STATE_CACHE["ts"]) < STATE_TTL_S: + return _STATE_CACHE["value"] state: Dict[str, Any] = {} r = _smi( "--query-gpu=driver_version,name,memory.total,power.limit,power.max_limit,power.default_limit," @@ -257,6 +279,7 @@ def get_gpu_state() -> Dict[str, Any]: state[k] = float(parts[i]) except ValueError: state[k] = parts[i] + _STATE_CACHE.update({"ts": __import__("time").time(), "value": state}) return state @@ -298,11 +321,16 @@ def set_fan_auto() -> Dict[str, Any]: ok = r["rc"] == 0 if ok: FAN_MANUAL = False + _FAN_CACHE["value"] = None return {"success": ok, "manual": False, "fan_speed_pct": None, "detail": r.get("out") or r.get("err")} -def get_fan_status() -> Dict[str, Any]: - """Read current fan control mode + target speed.""" +def get_fan_status(force: bool = False) -> Dict[str, Any]: + """Read current fan control mode + target speed (cached; forks nvidia-settings).""" + import time as _time + if not force and _FAN_CACHE["value"] is not None and \ + (_time.time() - _FAN_CACHE["ts"]) < STATE_TTL_S: + return _FAN_CACHE["value"] global FAN_MANUAL target = None manual = FAN_MANUAL @@ -320,7 +348,34 @@ def get_fan_status() -> Dict[str, Any]: target = int(line.split("):")[-1].split(".")[0].strip()) except Exception: pass - return {"manual": manual, "mode": "manual" if manual else "auto", "target_speed_pct": target} + result = {"manual": manual, "mode": "manual" if manual else "auto", "target_speed_pct": target} + _FAN_CACHE.update({"ts": __import__("time").time(), "value": result}) + return result + + +def restore_safe(reason: str = "shutdown") -> Dict[str, Any]: + """Return the card to stock: no clock locks, no offsets, default power, automatic fans. + + This matters because every lever here is sticky. If the service dies while a profile is + applied, the GPU keeps the locked clocks and, worse, keeps the fans pinned at whatever + manual PWM was last set. Nothing was undoing that. + """ + logger.warning(f"Restoring GPU to safe stock state ({reason})") + result = { + "reason": reason, + "clock_lock": _apply_clock_lock(0, 0), + "mem_lock": _apply_mem_lock(0), + "offsets": _apply_offsets(0, 0), + "fan": set_fan_auto(), + } + # Hand the power limit back to the card's own default rather than assuming 370 W. + state = get_gpu_state() + default_w = state.get("power_default_w") + if isinstance(default_w, (int, float)) and default_w > 0: + result["power_limit"] = _apply_power_limit(int(default_w)) + global ACTIVE_PROFILE + ACTIVE_PROFILE = "stock" + return result def get_status() -> Dict[str, Any]: diff --git a/overclock_profiles.json b/overclock_profiles.json index 573a6b7..d871404 100644 --- a/overclock_profiles.json +++ b/overclock_profiles.json @@ -2,8 +2,8 @@ "ollama": { "label": "Ollama \u2014 LLM decode (memory-bandwidth bound)", "power_limit_w": 370, - "core_offset_mhz": 150, - "mem_offset_mhz": 825, + "core_offset_mhz": 35, + "mem_offset_mhz": 200, "lock_core_min": 0, "lock_core_max": 0, "lock_mem_mhz": 0, @@ -14,12 +14,12 @@ "label": "ComfyUI \u2014 diffusion (core-compute bound)", "power_limit_w": 370, "core_offset_mhz": 100, - "mem_offset_mhz": 500, + "mem_offset_mhz": 150, "lock_core_min": 2900, "lock_core_max": 3105, "lock_mem_mhz": 0, "fan_mode": "manual", - "fan_speed_pct": 75 + "fan_speed_pct": 100 }, "balanced": { "label": "Balanced \u2014 stock boost, power unlocked", diff --git a/ram_optimizer.py b/ram_optimizer.py index a8306ee..bf0bff2 100644 --- a/ram_optimizer.py +++ b/ram_optimizer.py @@ -1,16 +1,48 @@ -"""RAM Optimizer and Model Pre-warmer for High-Speed Switching.""" -import os -import glob -import time -import httpx +"""RAM Optimizer and Model Pre-warmer for High-Speed Switching. + +Two things changed here versus the naive version: + + 1. Residency is *measured*, not assumed. mincore(2) tells us exactly what fraction of + each model file is resident in the Linux page cache, so "RAM Cache Hit" stops being + a guess based on how long a load took. + 2. Warming is *budgeted*. This box has 64 GB of RAM and >33 GB of models; reading every + file top-to-bottom simply evicts whatever was warmed first. Files are now scored by + recency/frequency (from the telemetry store) and warmed until a byte budget is hit, + skipping anything already resident. +""" +import ctypes +import ctypes.util +import json import logging -from typing import Dict, List, Any +import os +import random +import time +from typing import Dict, List, Any, Optional, Tuple + +import httpx + +import telemetry_store logger = logging.getLogger("ram_optimizer") OLLAMA_API_BASE = "http://localhost:11434" COMFY_API_BASE = "http://127.0.0.1:8188" -COMFY_MODELS_DIR = "/home/drjones/ComfyUI/models" +COMFY_MODELS_DIR = os.environ.get("HYPERSWAP_COMFY_MODELS", "/home/drjones/ComfyUI/models") +OLLAMA_MODEL_DIRS = [ + "/usr/share/ollama/.ollama/models", + os.path.expanduser("~/.ollama/models"), +] + +PAGE_SIZE = os.sysconf("SC_PAGE_SIZE") +# Files bigger than this are sampled rather than fully mapped for residency. +RESIDENCY_FULL_MAP_LIMIT = 2 * 1024 ** 3 +RESIDENCY_SAMPLE_WINDOWS = 64 +RESIDENCY_WINDOW_BYTES = 16 * 1024 * 1024 +# A file at/above this residency is considered warm and is skipped by the warmer. +WARM_SKIP_THRESHOLD_PCT = 90.0 + +CATALOG_TTL_S = 30.0 + def get_detailed_meminfo() -> Dict[str, Any]: """Parse /proc/meminfo for precise page cache and RAM stats.""" @@ -25,7 +57,7 @@ def get_detailed_meminfo() -> Dict[str, Any]: info[key] = int(val) * 1024 # Convert kB to bytes except Exception as e: logger.error(f"Failed to read /proc/meminfo: {e}") - + total = info.get("MemTotal", 0) free = info.get("MemFree", 0) available = info.get("MemAvailable", 0) @@ -51,43 +83,439 @@ def get_detailed_meminfo() -> Dict[str, Any]: "cache_ratio_pct": round((cached / total * 100) if total > 0 else 0, 1), } -def warm_file_to_ram(filepath: str, chunk_size: int = 16 * 1024 * 1024) -> Dict[str, Any]: - """Pre-fault/read file into Linux OS Page Cache at maximum disk read speed.""" + +# ---------------------------------------------------------------- page residency +# +# Measuring page-cache residency turned out to be the subtle part. +# +# * cachestat(2) (Linux 6.5+) is the right tool: exact cached-page counts for an fd, +# no mmap, microseconds per call. But the kernel only permits it on files you own +# or can write -- the Ollama blobs are owned by uid `ollama`, so it returns EPERM. +# * mincore(2) does NOT fail closed for those files on this kernel: it reports every +# page as resident, which produced 128 GB of "resident" model weights on a box with +# 46 GB of page cache. It is therefore not used at all. +# +# So: cachestat where permitted, and an explicit read-throughput probe where it is not. +# Anything we cannot measure is reported as unmeasurable rather than guessed at. + +_libc = None +_SYS_cachestat = 451 # x86_64 + + +class _CachestatRange(ctypes.Structure): + _fields_ = [("off", ctypes.c_uint64), ("len", ctypes.c_uint64)] + + +class _Cachestat(ctypes.Structure): + _fields_ = [ + ("nr_cache", ctypes.c_uint64), + ("nr_dirty", ctypes.c_uint64), + ("nr_writeback", ctypes.c_uint64), + ("nr_evicted", ctypes.c_uint64), + ("nr_recently_evicted", ctypes.c_uint64), + ] + + +def _get_libc(): + global _libc + if _libc is None: + _libc = ctypes.CDLL(ctypes.util.find_library("c") or "libc.so.6", use_errno=True) + return _libc + + +def _cachestat(fd: int, offset: int, length: int) -> Optional[_Cachestat]: + """Raw cachestat(2). Returns None if the kernel refuses (EPERM/ENOSYS).""" + libc = _get_libc() + rng = _CachestatRange(offset, length) + cs = _Cachestat() + ctypes.set_errno(0) + rc = libc.syscall(ctypes.c_long(_SYS_cachestat), ctypes.c_int(fd), + ctypes.byref(rng), ctypes.byref(cs), ctypes.c_uint(0)) + if rc != 0: + return None + return cs + + +PROBE_WINDOWS = 12 +PROBE_WINDOW_BYTES = 2 * 1024 * 1024 +# Measured on this box: cold NVMe reads land around 0.35-0.5 GB/s, page-cache reads at +# 3.2-13 GB/s. 1.5 GB/s sits in the empty middle of that gap. +PROBE_CACHED_GBPS = 1.5 + + +def _throughput_probe(fd: int, size: int) -> Dict[str, Any]: + """Infer residency by timing reads of small windows spread across the file. + + Used only where cachestat is not permitted (Ollama's blobs are owned by uid `ollama`). + + Two details matter for correctness: + + * Offsets are random per call. A fixed stride made the probe self-fulfilling: the + first pass faulted its 24 MB of sample windows into the page cache, and every pass + after that re-read exactly those windows and reported 100% resident for a file that + was almost entirely cold. + * Windows that read cold are handed straight back with FADV_DONTNEED. Those pages are + pollution the probe itself created, and leaving them behind would slowly warm the + cache with data nobody asked for. + """ + windows = min(PROBE_WINDOWS, max(int(size // PROBE_WINDOW_BYTES), 1)) + if windows <= 0: + return {"resident_pct": 0.0, "windows": 0} + + max_off = max(size - PROBE_WINDOW_BYTES, 0) + offsets = sorted(random.randint(0, max_off) for _ in range(windows)) if max_off else [0] + buf = bytearray(PROBE_WINDOW_BYTES) + cached = 0 + rates = [] + for off in offsets: + length = min(PROBE_WINDOW_BYTES, size - off) + if length <= 0: + continue + view = memoryview(buf)[:length] + t0 = time.perf_counter() + os.preadv(fd, [view], off) + dt = time.perf_counter() - t0 + gbps = (length / (1024 ** 3)) / dt if dt > 0 else 0.0 + rates.append(gbps) + if gbps >= PROBE_CACHED_GBPS: + cached += 1 + else: + # We just pulled this off disk; put it back the way we found it. + try: + os.posix_fadvise(fd, off, length, os.POSIX_FADV_DONTNEED) + except Exception: + pass + n = len(rates) + return { + "resident_pct": round((cached / n * 100) if n else 0.0, 1), + "windows": n, + "median_gbps": round(sorted(rates)[n // 2], 2) if n else 0.0, + "sampled_gb": round(n * PROBE_WINDOW_BYTES / (1024 ** 3), 3), + } + + +def page_residency(filepath: str, allow_probe: bool = True) -> Dict[str, Any]: + """Measure what fraction of a file is resident in the Linux page cache.""" + try: + size = os.path.getsize(filepath) + except OSError as e: + return {"success": False, "error": str(e), "resident_pct": 0.0, "measurable": False} + if size == 0: + return {"success": True, "resident_pct": 0.0, "size_bytes": 0, "measurable": True, + "method": "empty"} + + try: + fd = os.open(filepath, os.O_RDONLY) + except OSError as e: + return {"success": False, "error": str(e), "resident_pct": 0.0, "measurable": False} + try: + cs = _cachestat(fd, 0, size) + if cs is not None: + total_pages = (size + PAGE_SIZE - 1) // PAGE_SIZE + pct = round((cs.nr_cache / total_pages * 100) if total_pages else 0.0, 1) + method, measurable = "cachestat", True + extra = {"dirty_pages": cs.nr_dirty, "evicted_pages": cs.nr_evicted} + elif allow_probe: + probe = _throughput_probe(fd, size) + pct = probe["resident_pct"] + method, measurable = "probe", True + extra = {"probe_windows": probe["windows"], "probe_median_gbps": probe.get("median_gbps")} + else: + return {"success": True, "filepath": filepath, "size_bytes": size, + "size_gb": round(size / (1024**3), 3), "resident_pct": None, + "measurable": False, "method": "unavailable", "warm": None, + "reason": "cachestat not permitted for this file (not owned by us)"} + + return { + "success": True, + "filepath": filepath, + "size_bytes": size, + "size_gb": round(size / (1024**3), 3), + "resident_pct": pct, + "resident_bytes": int(size * pct / 100.0), + "method": method, + "measurable": measurable, + "warm": pct >= WARM_SKIP_THRESHOLD_PCT, + **extra, + } + except Exception as e: + return {"success": False, "error": str(e), "resident_pct": 0.0, + "size_bytes": size, "measurable": False} + finally: + os.close(fd) + + +def residency_capability() -> Dict[str, Any]: + """Report whether exact residency is available, and how to enable it if not.""" + catalog = get_model_catalog() + blocked = [] + for f in catalog["ollama"]: + try: + fd = os.open(f["full_path"], os.O_RDONLY) + except OSError: + continue + try: + if _cachestat(fd, 0, 4096) is None: + blocked.append(f["full_path"]) + finally: + os.close(fd) + break # one probe is enough; blobs share a directory and owner + if not blocked: + return {"exact_everywhere": True} + owner = "" + try: + import pwd + owner = pwd.getpwuid(os.stat(blocked[0]).st_uid).pw_name + except Exception: + owner = str(os.stat(blocked[0]).st_uid) + return { + "exact_everywhere": False, + "method_for_blocked": "probe", + "reason": f"cachestat(2) is only permitted on files you own or can write; " + f"Ollama blobs are owned by '{owner}'", + "hint": f"exact numbers for Ollama weights need read/write access, e.g. " + f"'sudo usermod -aG {owner} $USER' plus group-write on the blobs directory", + } + + +# ---------------------------------------------------------------- catalogs + +_catalog_cache: Dict[str, Any] = {"ts": 0.0, "sig": None, "comfy": [], "ollama": []} + + +def _dir_signature(root: str) -> Tuple: + """Cheap fingerprint of a model tree: (mtime, entry count) per subdirectory.""" + sig = [] + if not os.path.isdir(root): + return tuple(sig) + for dirpath, dirnames, filenames in os.walk(root): + try: + sig.append((dirpath, os.stat(dirpath).st_mtime_ns, len(filenames))) + except OSError: + continue + return tuple(sig) + + +def find_ollama_model_files() -> List[Dict[str, Any]]: + """Map installed Ollama models to their on-disk GGUF blobs via the manifest tree. + + Knowing the blob path is what lets us warm (or measure) a specific model's weights + without pulling them into VRAM. + """ + results: List[Dict[str, Any]] = [] + seen = set() + for root in OLLAMA_MODEL_DIRS: + manifests = os.path.join(root, "manifests") + blobs = os.path.join(root, "blobs") + if not os.path.isdir(manifests): + continue + for dirpath, _, filenames in os.walk(manifests): + for tag in filenames: + manifest_path = os.path.join(dirpath, tag) + try: + with open(manifest_path) as f: + manifest = json.load(f) + except Exception: + continue + rel = os.path.relpath(dirpath, manifests) + parts = rel.split(os.sep) + # registry/namespace/name -> "name:tag", keeping non-library namespaces + name = parts[-1] if parts else rel + namespace = parts[-2] if len(parts) >= 2 else "library" + model_name = f"{name}:{tag}" if namespace == "library" else f"{namespace}/{name}:{tag}" + for layer in manifest.get("layers", []): + if layer.get("mediaType") != "application/vnd.ollama.image.model": + continue + digest = (layer.get("digest") or "").replace(":", "-") + blob_path = os.path.join(blobs, digest) + if not os.path.exists(blob_path): + continue + key = (model_name, blob_path) + if key in seen: + continue + seen.add(key) + size = layer.get("size") or os.path.getsize(blob_path) + results.append({ + "model": model_name, + "filename": digest, + "full_path": blob_path, + "size_bytes": size, + "size_gb": round(size / (1024**3), 3), + "kind": "ollama", + }) + return results + + +def find_comfy_model_files(force_refresh: bool = False) -> List[Dict[str, Any]]: + """Discover all model files under ComfyUI models (cached). + + This used to run inside the 1Hz telemetry snapshot, meaning a full recursive walk plus + a stat() of every checkpoint once per second per connected dashboard. It is now cached + behind a directory-mtime fingerprint. + """ + _refresh_catalog(force_refresh) + return _catalog_cache["comfy"] + + +def get_model_catalog(force_refresh: bool = False) -> Dict[str, Any]: + _refresh_catalog(force_refresh) + return { + "comfy": _catalog_cache["comfy"], + "ollama": _catalog_cache["ollama"], + "cached_at": _catalog_cache["ts"], + } + + +def _refresh_catalog(force: bool = False) -> None: + now = time.time() + if not force and (now - _catalog_cache["ts"]) < CATALOG_TTL_S: + return + sig = _dir_signature(COMFY_MODELS_DIR) + if not force and sig == _catalog_cache["sig"] and _catalog_cache["comfy"]: + _catalog_cache["ts"] = now + return + + extensions = (".safetensors", ".ckpt", ".pt", ".bin", ".gguf", ".sft") + results = [] + if os.path.exists(COMFY_MODELS_DIR): + for root, _, files in os.walk(COMFY_MODELS_DIR): + for file in files: + if not file.endswith(extensions): + continue + full_path = os.path.join(root, file) + try: + st = os.stat(full_path) + except OSError: + continue + results.append({ + "filename": file, + "rel_path": os.path.relpath(full_path, COMFY_MODELS_DIR), + "full_path": full_path, + "category": os.path.relpath(root, COMFY_MODELS_DIR).split(os.sep)[0], + "size_bytes": st.st_size, + "size_mb": round(st.st_size / (1024**2), 2), + "size_gb": round(st.st_size / (1024**3), 3), + "mtime": st.st_mtime, + "kind": "comfy", + }) + _catalog_cache.update({"ts": now, "sig": sig, "comfy": results, + "ollama": find_ollama_model_files()}) + + +# ---------------------------------------------------------------- residency report + +_report_cache: Dict[str, Any] = {"ts": 0.0, "report": None} +REPORT_TTL_S = 15.0 + + +def get_cache_report(include_files: bool = True, force_refresh: bool = False) -> Dict[str, Any]: + """Measured page-cache residency across the whole model catalog. + + Deduplicated by blob path: several Ollama tags routinely point at the same GGUF, and + counting each tag separately produced more "resident" bytes than the box has RAM. + """ + now = time.time() + cached = _report_cache["report"] + if cached and not force_refresh and (now - _report_cache["ts"]) < REPORT_TTL_S: + return cached if include_files else {**cached, "files": []} + + t0 = time.perf_counter() + catalog = get_model_catalog() + by_path: Dict[str, Dict[str, Any]] = {} + for f in list(catalog["ollama"]) + list(catalog["comfy"]): + path = f["full_path"] + name = f.get("model") or f.get("rel_path") or f.get("filename") + if path in by_path: + by_path[path]["aliases"].append(name) + continue + by_path[path] = {"entry": f, "name": name, "aliases": []} + + entries = [] + total_bytes = resident_bytes = 0 + for path, meta in by_path.items(): + f = meta["entry"] + res = page_residency(path) + size = f.get("size_bytes") or res.get("size_bytes") or 0 + rb = res.get("resident_bytes", 0) + total_bytes += size + resident_bytes += rb + entries.append({ + "name": meta["name"], + "aliases": meta["aliases"], + "kind": f.get("kind"), + "full_path": path, + "size_gb": round(size / (1024**3), 3), + "resident_pct": res.get("resident_pct", 0.0), + "resident_gb": round(rb / (1024**3), 3), + "warm": res.get("warm", False), + }) + entries.sort(key=lambda e: e["resident_gb"], reverse=True) + + report = { + "scan_ms": round((time.perf_counter() - t0) * 1000, 1), + "files_scanned": len(entries), + "unique_blobs": len(by_path), + "catalog_total_gb": round(total_bytes / (1024**3), 2), + "resident_total_gb": round(resident_bytes / (1024**3), 2), + "residency_pct": round((resident_bytes / total_bytes * 100) if total_bytes else 0, 1), + "warm_files": sum(1 for e in entries if e["warm"]), + "files": entries, + } + _report_cache.update({"ts": now, "report": report}) + return report if include_files else {**report, "files": []} + + +# ---------------------------------------------------------------- warming + +def warm_file_to_ram(filepath: str, chunk_size: int = 16 * 1024 * 1024, + skip_if_warm: bool = True) -> Dict[str, Any]: + """Pre-fault a file into the Linux page cache, skipping it if already resident.""" if not os.path.exists(filepath): return {"success": False, "error": f"File not found: {filepath}", "duration_ms": 0} - + + before = page_residency(filepath) + if skip_if_warm and before.get("warm"): + return { + "success": True, "filepath": filepath, "skipped": True, + "reason": "already resident", "resident_pct": before.get("resident_pct"), + "size_mb": round(before.get("size_bytes", 0) / (1024**2), 2), + "duration_ms": 0.0, "bytes_read": 0, + } + t0 = time.perf_counter() file_size = os.path.getsize(filepath) bytes_read = 0 try: with open(filepath, "rb") as f: - # Hint kernel that we will read this sequentially try: os.posix_fadvise(f.fileno(), 0, file_size, os.POSIX_FADV_WILLNEED) except Exception: pass - buf = bytearray(chunk_size) while True: n = f.readinto(buf) if not n: break bytes_read += n - + duration = time.perf_counter() - t0 - duration_ms = round(duration * 1000, 2) - speed_mb_s = round((bytes_read / (1024**2)) / duration if duration > 0 else 0, 2) + after = page_residency(filepath) return { "success": True, "filepath": filepath, + "skipped": False, "size_bytes": file_size, "size_mb": round(file_size / (1024**2), 2), "bytes_read": bytes_read, - "duration_ms": duration_ms, - "speed_mb_s": speed_mb_s, + "duration_ms": round(duration * 1000, 2), + "speed_mb_s": round((bytes_read / (1024**2)) / duration if duration > 0 else 0, 2), + "resident_pct_before": before.get("resident_pct", 0.0), + "resident_pct_after": after.get("resident_pct", 0.0), } except Exception as e: - return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + return {"success": False, "error": str(e), + "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + async def warm_ollama_model(model_name: str, keep_alive: str = "5m") -> Dict[str, Any]: """Warm an Ollama model into memory and measure time.""" @@ -101,76 +529,148 @@ async def warm_ollama_model(model_name: str, keep_alive: str = "5m") -> Dict[str duration = time.perf_counter() - t0 if resp.status_code == 200: data = resp.json() - return { + res = { "success": True, "model": model_name, "duration_ms": round(duration * 1000, 2), "load_duration_ms": round(data.get("load_duration", 0) / 1e6, 2), "total_duration_ms": round(data.get("total_duration", 0) / 1e6, 2), } - else: - return { - "success": False, - "model": model_name, - "error": f"HTTP {resp.status_code}: {resp.text}", - "duration_ms": round(duration * 1000, 2), - } + telemetry_store.record_event({ + "event_type": "Model Warm", "source": "warmer", "target": model_name, + "duration_ms": res["duration_ms"], "load_duration_ms": res["load_duration_ms"], + }) + return res + return { + "success": False, "model": model_name, + "error": f"HTTP {resp.status_code}: {resp.text}", + "duration_ms": round(duration * 1000, 2), + } except Exception as e: - return {"success": False, "model": model_name, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + return {"success": False, "model": model_name, "error": str(e), + "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} -def find_comfy_model_files() -> List[Dict[str, Any]]: - """Discover all model files under ComfyUI models.""" - results = [] - extensions = ("*.safetensors", "*.ckpt", "*.pt", "*.bin") - if os.path.exists(COMFY_MODELS_DIR): - for root, _, files in os.walk(COMFY_MODELS_DIR): - for file in files: - if any(file.endswith(ext.replace("*", "")) for ext in extensions): - full_path = os.path.join(root, file) - rel_path = os.path.relpath(full_path, COMFY_MODELS_DIR) - size = os.path.getsize(full_path) - results.append({ - "filename": file, - "rel_path": rel_path, - "full_path": full_path, - "size_bytes": size, - "size_mb": round(size / (1024**2), 2), - "size_gb": round(size / (1024**3), 3), - }) - return results -async def warm_all_models() -> Dict[str, Any]: - """Warm all available Ollama and ComfyUI models into Linux RAM Cache.""" - t0 = time.perf_counter() - warmed_ollama = [] - warmed_comfy = [] - - # 1. Ollama models +def warm_ollama_blob(model_name: str) -> Dict[str, Any]: + """Warm a specific Ollama model's GGUF into page cache without touching VRAM.""" + for f in find_ollama_model_files(): + if f["model"] == model_name: + res = warm_file_to_ram(f["full_path"]) + res["model"] = model_name + return res + return {"success": False, "error": f"no blob found for model '{model_name}'"} + + +def _warm_priority(days: float = 30.0) -> Dict[str, float]: + """Recency/frequency score per model name, from the persisted event log.""" try: - async with httpx.AsyncClient(timeout=10.0) as client: - tags_resp = await client.get(f"{OLLAMA_API_BASE}/api/tags") - if tags_resp.status_code == 200: - models = tags_resp.json().get("models", []) - for m in models: - name = m.get("name") - res = await warm_ollama_model(name, keep_alive="1m") - warmed_ollama.append(res) - except Exception as e: - logger.error(f"Error discovering Ollama models: {e}") - - # 2. ComfyUI models - comfy_files = find_comfy_model_files() - for f in comfy_files: - res = warm_file_to_ram(f["full_path"]) - warmed_comfy.append(res) - - total_duration_ms = round((time.perf_counter() - t0) * 1000, 2) - meminfo = get_detailed_meminfo() - + return {r["model"]: r["score"] for r in telemetry_store.model_usage_ranking(days)} + except Exception: + return {} + + +def build_warm_plan(budget_gb: Optional[float] = None) -> Dict[str, Any]: + """Decide *what* to warm, in what order, within a byte budget. + + Warming everything on a 64 GB box with 33+ GB of models just evicts the earliest + files, so we rank by usage (Ollama, from history) and recency (ComfyUI, by mtime), + then fill until the budget is spent. Already-resident files cost nothing. + """ + mem = get_detailed_meminfo() + if budget_gb is None: + # Leave headroom so warming never pushes the box into reclaim. + budget_gb = max((mem["available_bytes"] * 0.7) / (1024**3), 1.0) + budget_bytes = int(budget_gb * (1024**3)) + + catalog = get_model_catalog() + scores = _warm_priority() + now = time.time() + + candidates = [] + for f in catalog["ollama"]: + candidates.append({**f, "score": scores.get(f["model"], 0.0) + 0.5, + "name": f["model"]}) + for f in catalog["comfy"]: + age_days = max((now - f.get("mtime", now)) / 86400.0, 0.01) + candidates.append({**f, "score": scores.get(f["rel_path"], 0.0) + 1.0 / (1.0 + age_days), + "name": f["rel_path"]}) + + candidates.sort(key=lambda c: c["score"], reverse=True) + + plan, spent, skipped = [], 0, [] + seen_paths = set() + for c in candidates: + if c["full_path"] in seen_paths: + continue + seen_paths.add(c["full_path"]) + res = page_residency(c["full_path"]) + entry = { + "name": c["name"], "kind": c["kind"], "full_path": c["full_path"], + "size_gb": c.get("size_gb", 0), "score": round(c["score"], 4), + "resident_pct": res.get("resident_pct", 0.0), + } + if res.get("warm"): + entry["action"] = "already-warm" + skipped.append(entry) + continue + need = int(c.get("size_bytes", 0) * (1 - res.get("resident_pct", 0) / 100.0)) + if spent + need > budget_bytes: + entry["action"] = "over-budget" + skipped.append(entry) + continue + spent += need + entry["action"] = "warm" + entry["bytes_to_read"] = need + plan.append(entry) + + return { + "budget_gb": round(budget_gb, 2), + "planned_gb": round(spent / (1024**3), 2), + "warm_count": len(plan), + "skipped_count": len(skipped), + "plan": plan, + "skipped": skipped, + "meminfo": mem, + } + + +async def warm_all_models(budget_gb: Optional[float] = None, + include_vram_load: bool = False) -> Dict[str, Any]: + """Warm the highest-value models into the page cache within a byte budget.""" + t0 = time.perf_counter() + plan = build_warm_plan(budget_gb) + warmed = [] + for entry in plan["plan"]: + res = warm_file_to_ram(entry["full_path"]) + res["name"] = entry["name"] + res["kind"] = entry["kind"] + warmed.append(res) + # Budgets are computed up front, but the page cache is shared with the rest of + # the box; bail out if we start pushing the system into reclaim. + if get_detailed_meminfo()["available_gb"] < 4.0: + logger.warning("warm_all_models: stopping early, MemAvailable below 4 GB") + break + + if include_vram_load: + try: + async with httpx.AsyncClient(timeout=10.0) as client: + tags = await client.get(f"{OLLAMA_API_BASE}/api/tags") + if tags.status_code == 200: + top = sorted(tags.json().get("models", []), + key=lambda m: _warm_priority().get(m.get("name"), 0), + reverse=True)[:1] + for m in top: + await warm_ollama_model(m.get("name"), keep_alive="1m") + except Exception as e: + logger.debug(f"optional VRAM preload skipped: {e}") + return { "status": "completed", - "total_duration_ms": total_duration_ms, - "ollama_models_warmed": warmed_ollama, - "comfy_files_warmed": warmed_comfy, - "meminfo_after": meminfo, + "total_duration_ms": round((time.perf_counter() - t0) * 1000, 2), + "budget_gb": plan["budget_gb"], + "planned_gb": plan["planned_gb"], + "files_warmed": warmed, + "bytes_read": sum(w.get("bytes_read", 0) for w in warmed), + "skipped": plan["skipped"], + "meminfo_after": get_detailed_meminfo(), } diff --git a/server.py b/server.py index 5ebfb7f..c172792 100644 --- a/server.py +++ b/server.py @@ -1,27 +1,169 @@ """FastAPI Backend Server with SSE Real-Time Telemetry and Model Orchestration API.""" import asyncio +import contextlib import json import logging -from typing import Dict, Any, Optional, List +import time +from contextlib import asynccontextmanager +from typing import Dict, Any, Optional, List, Set + from fastapi import FastAPI, Request, HTTPException, Query from fastapi.responses import HTMLResponse, StreamingResponse, JSONResponse from fastapi.staticfiles import StaticFiles from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel, Field -import ram_optimizer -import vram_arbitrator +import autotune import overclock_manager +import ram_optimizer +import telemetry_store +import thermal_governor +import vram_arbitrator logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s") logger = logging.getLogger("model_manager_server") +# The sampler makes four HTTP calls a second; at INFO, httpx narrates every one of them. +logging.getLogger("httpx").setLevel(logging.WARNING) +logging.getLogger("httpcore").setLevel(logging.WARNING) + +BASE_DIR = "/home/drjones/unified-model-manager" + + +# ========================================== +# TELEMETRY BROKER +# ========================================== + +class TelemetryBroker: + """One sampler, many subscribers. + + Every SSE client used to run its own copy of the full snapshot once per second: + NVML queries, /proc/meminfo, an HTTP round-trip each to Ollama and ComfyUI, and โ€” the + expensive one โ€” a recursive walk of the ComfyUI models tree with a stat() per + checkpoint. Opening the dashboard in three tabs tripled the load on the very thing it + was measuring. Now a single background task samples at 1 Hz and fans the snapshot out. + + The sampler is also the natural feed for the thermal governor and the persistence + layer, so neither needs to poll the GPU on its own. + """ + + def __init__(self, interval_s: float = 1.0) -> None: + self.interval_s = interval_s + self.snapshot: Dict[str, Any] = {} + self.subscribers: Set[asyncio.Queue] = set() + self.task: Optional[asyncio.Task] = None + self.running = False + self.samples = 0 + self.last_sample_ms = 0.0 + + async def start(self) -> None: + if self.running: + return + self.running = True + self.task = asyncio.create_task(self._loop()) + + async def stop(self) -> None: + self.running = False + if self.task: + self.task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await self.task + + def subscribe(self) -> asyncio.Queue: + q: asyncio.Queue = asyncio.Queue(maxsize=2) + self.subscribers.add(q) + return q + + def unsubscribe(self, q: asyncio.Queue) -> None: + self.subscribers.discard(q) + + async def _loop(self) -> None: + while self.running: + t0 = time.perf_counter() + try: + snap = await self._sample() + self.snapshot = snap + self.samples += 1 + self.last_sample_ms = round((time.perf_counter() - t0) * 1000, 2) + + # Feed the governor and the durable store from the sample we already have. + thermal_governor.governor.observe(snap.get("gpu", {}), + overclock_manager.ACTIVE_PROFILE) + telemetry_store.record_telemetry( + snap.get("gpu", {}), snap.get("ram", {}), + profile=overclock_manager.ACTIVE_PROFILE, + throttle_reasons=",".join(snap.get("gpu", {}).get("throttle_reasons") or []), + ) + + for q in list(self.subscribers): + if q.full(): + # Slow client: drop the stale frame rather than stalling the sampler. + with contextlib.suppress(asyncio.QueueEmpty): + q.get_nowait() + with contextlib.suppress(asyncio.QueueFull): + q.put_nowait(snap) + except asyncio.CancelledError: + raise + except Exception as e: + logger.error(f"telemetry sampler error: {e}") + await asyncio.sleep(max(self.interval_s - (time.perf_counter() - t0), 0.05)) + + async def _sample(self) -> Dict[str, Any]: + gpu_stats = vram_arbitrator.get_gpu_hardware_stats() + mem_stats = ram_optimizer.get_detailed_meminfo() + ollama_state, comfy_state = await asyncio.gather( + vram_arbitrator.get_ollama_live_state(), + vram_arbitrator.get_comfyui_live_state(), + ) + return { + "timestamp": time.time(), # wall clock, not the event loop's monotonic clock + "monotonic": asyncio.get_running_loop().time(), + "gpu": gpu_stats, + "ram": mem_stats, + "ollama": ollama_state, + "comfyui": comfy_state, + "arbitrator": vram_arbitrator.arbitrator.get_status(), + "governor": thermal_governor.governor.get_status(), + "overclock": {"active_profile": overclock_manager.ACTIVE_PROFILE}, + "history": vram_arbitrator.get_switch_history(), + "comfy_models_count": len(ram_optimizer.find_comfy_model_files()), + "sampler": {"samples": self.samples, "last_sample_ms": self.last_sample_ms, + "subscribers": len(self.subscribers)}, + } + + async def get(self) -> Dict[str, Any]: + """Latest snapshot, sampling on demand if the loop has not produced one yet.""" + if not self.snapshot: + self.snapshot = await self._sample() + return self.snapshot + + +broker = TelemetryBroker() + + +@asynccontextmanager +async def lifespan(app: FastAPI): + telemetry_store.start() + await broker.start() + await vram_arbitrator.arbitrator.start() + yield + await vram_arbitrator.arbitrator.stop() + await broker.stop() + # Never leave the card with locked clocks and pinned fans after we exit. + try: + overclock_manager.restore_safe("server shutdown") + except Exception as e: + logger.error(f"restore_safe on shutdown failed: {e}") + telemetry_store.stop() + + app = FastAPI( title="HyperSwap // GPU Program Swapper & Telemetry API", - version="1.0.0", + version="2.0.0", description="High-performance VRAM arbitration and 64GB RAM cache orchestrator for simultaneous Ollama and ComfyUI workloads on Linux.", docs_url="/docs", redoc_url="/redoc", + lifespan=lifespan, ) app.add_middleware( @@ -32,26 +174,24 @@ app.add_middleware( allow_headers=["*"], ) -@app.on_event("startup") -async def on_startup(): - await vram_arbitrator.arbitrator.start() - -@app.on_event("shutdown") -async def on_shutdown(): - await vram_arbitrator.arbitrator.stop() # Pydantic Request Models class SwitchRequest(BaseModel): model: str = Field(..., description="Name of the Ollama model to hot-swap to in VRAM", example="qwen3.8fast:latest") keep_alive: Optional[str] = Field("30m", description="Keep-alive duration in VRAM (e.g. 5m, 30m, 0)", example="30m") + free_comfy_first: bool = Field(False, description="Purge ComfyUI VRAM first if it is holding memory") class WarmRequest(BaseModel): - model_name: Optional[str] = Field(None, description="Ollama model name to warm into OS page cache", example="gemma4:26b") - filepath: Optional[str] = Field(None, description="Absolute file path of Safetensors/GGUF to warm into RAM", example="/home/drjones/ComfyUI/models/checkpoints/v1-5-pruned-emaonly-fp16.safetensors") + model_name: Optional[str] = Field(None, description="Ollama model name to warm", example="gemma4:26b") + filepath: Optional[str] = Field(None, description="Absolute file path of Safetensors/GGUF to warm into RAM") + blob_only: bool = Field(False, description="Warm the model's weights into page cache without loading VRAM") + +class WarmAllRequest(BaseModel): + budget_gb: Optional[float] = Field(None, description="Byte budget for warming; defaults to 70% of MemAvailable", example=24.0) class BenchmarkRequest(BaseModel): iterations: Optional[int] = Field(2, description="Number of back-and-forth switch iterations to measure", example=2) - models: Optional[List[str]] = Field(None, description="Optional pair of models to benchmark between", example=["qwen3.8fast:latest", "smtek/Qwen3.8-27B:Q2_K_XL"]) + models: Optional[List[str]] = Field(None, description="Optional pair of models to benchmark between") class OverclockApplyRequest(BaseModel): profile: str = Field(..., description="Profile name: ollama | comfy | balanced", example="ollama") @@ -64,6 +204,23 @@ class FanRequest(BaseModel): percent: Optional[int] = Field(None, description="Fan speed 30-100 when mode=manual", example=70) speed_pct: Optional[int] = Field(None, description="Alias for percent (30-100)", example=70) +class GovernorRequest(BaseModel): + enabled: Optional[bool] = Field(None, description="Enable or disable the thermal governor") + reset: bool = Field(False, description="Clear any active derate and reapply the full profile") + +class SweepRequest(BaseModel): + knob: str = Field("mem_offset_mhz", description="mem_offset_mhz | core_offset_mhz") + profile: str = Field("ollama", description="Profile to tune") + model: Optional[str] = Field(None, description="Model to benchmark with; defaults to the loaded one") + start: Optional[int] = Field(None, description="First offset value") + stop: Optional[int] = Field(None, description="Last offset value") + step: Optional[int] = Field(None, description="Offset increment") + repeats: int = Field(1, description="Benchmark runs per step") + apply_best: bool = Field(False, description="Write the winning value into the profile") + +class RequestVramRequest(BaseModel): + needed_gb: float = Field(0.0, description="How much free VRAM Ollama needs", example=12.0) + # ========================================== # REST API ENDPOINTS @@ -71,52 +228,44 @@ class FanRequest(BaseModel): @app.get("/api/stats", summary="Full System Snapshot", tags=["Telemetry"]) async def get_all_stats() -> Dict[str, Any]: - """Gather complete live snapshot of GPU hardware, host RAM, Ollama, ComfyUI, and switch history.""" - gpu_stats = vram_arbitrator.get_gpu_hardware_stats() - mem_stats = ram_optimizer.get_detailed_meminfo() - ollama_state = await vram_arbitrator.get_ollama_live_state() - comfy_state = await vram_arbitrator.get_comfyui_live_state() - history = vram_arbitrator.get_switch_history() - comfy_models = ram_optimizer.find_comfy_model_files() - arbitrator_status = vram_arbitrator.arbitrator.get_status() - - return { - "timestamp": asyncio.get_event_loop().time(), - "gpu": gpu_stats, - "ram": mem_stats, - "ollama": ollama_state, - "comfyui": comfy_state, - "arbitrator": arbitrator_status, - "history": history, - "comfy_models_count": len(comfy_models), - } + """Latest unified snapshot of GPU hardware, host RAM, Ollama, ComfyUI and swap history.""" + return await broker.get() @app.get("/api/gpu", summary="GPU Sensors and VRAM Breakdown", tags=["Telemetry"]) async def get_gpu_metrics() -> Dict[str, Any]: - """Retrieve detailed NVML sensors (utilization %, temp, power, fan, clocks, and per-process VRAM allocation).""" + """Detailed NVML sensors (utilization, temp, power, fan, clocks, throttle reasons, per-process VRAM).""" return vram_arbitrator.get_gpu_hardware_stats() @app.get("/api/memory", summary="Host RAM and Page Cache Breakdown", tags=["Telemetry"]) async def get_ram_metrics() -> Dict[str, Any]: - """Retrieve precise host 64GB DDR5 RAM breakdown, active cache size, and cache hit ratios.""" + """Precise host RAM breakdown, active cache size and cache ratios.""" return ram_optimizer.get_detailed_meminfo() @app.get("/api/stream", summary="Real-Time SSE Telemetry Stream", tags=["Telemetry"]) async def sse_telemetry_stream(request: Request): - """Server-Sent Events (SSE) streaming real-time statistics at 1Hz for dynamic dashboards.""" + """Server-Sent Events stream of the shared 1Hz snapshot.""" async def event_generator(): - while True: - if await request.is_disconnected(): - break - try: - stats = await get_all_stats() - yield f"data: {json.dumps(stats)}\n\n" - except Exception as e: - logger.error(f"SSE stream error: {e}") - yield f"data: {json.dumps({'error': str(e)})}\n\n" - await asyncio.sleep(1.0) - + q = broker.subscribe() + try: + snap = await broker.get() + yield f"data: {json.dumps(snap)}\n\n" + while True: + if await request.is_disconnected(): + break + try: + snap = await asyncio.wait_for(q.get(), timeout=15.0) + yield f"data: {json.dumps(snap)}\n\n" + except asyncio.TimeoutError: + yield ": keepalive\n\n" + except asyncio.CancelledError: + raise + except Exception as e: + logger.error(f"SSE stream error: {e}") + yield f"data: {json.dumps({'error': str(e)})}\n\n" + finally: + broker.unsubscribe(q) + return StreamingResponse( event_generator(), media_type="text/event-stream", @@ -129,80 +278,142 @@ async def sse_telemetry_stream(request: Request): @app.post("/api/switch-model", summary="Hot-Swap Ollama LLM in VRAM", tags=["Orchestration"]) async def api_switch_model(req: SwitchRequest): - """Hot-swap the active Ollama model in VRAM and measure exact load duration and token evaluation speed.""" + """Hot-swap the active Ollama model, measuring real load bandwidth and token throughput.""" + if req.free_comfy_first: + await vram_arbitrator.arbitrator.request_vram_for_ollama() res = await vram_arbitrator.switch_ollama_model(req.model, keep_alive=req.keep_alive or "30m") if not res.get("success"): raise HTTPException(status_code=500, detail=res.get("error")) return res @app.post("/api/free-vram", summary="Soft-Yield Ollama VRAM", tags=["Orchestration"]) -async def api_free_vram(): - """Instruct Ollama to instantly yield VRAM to 0MB in ~15ms while preserving model weights in the 64GB host RAM page cache.""" - return await vram_arbitrator.instant_free_ollama_vram() +async def api_free_vram(confirm: bool = Query(True, description="Wait for the driver to actually release the allocation")): + """Yield Ollama's VRAM and wait for the release to be confirmed by NVML.""" + return await vram_arbitrator.instant_free_ollama_vram(confirm=confirm) @app.post("/api/comfy-free", summary="Purge ComfyUI VRAM Cache", tags=["Orchestration"]) async def api_comfy_free(): """Purge loaded diffusion models and VRAM cache from the ComfyUI pipeline.""" return await vram_arbitrator.instant_free_comfyui_vram() -@app.post("/api/warm-all", summary="Pre-warm All Models into RAM Cache", tags=["Memory Optimization"]) -async def api_warm_all(): - """Pre-fault and read all installed Ollama GGUF models and ComfyUI Safetensors checkpoints into the Linux OS Page Cache.""" - return await ram_optimizer.warm_all_models() +@app.post("/api/request-vram", summary="Ask for VRAM on Ollama's behalf", tags=["Orchestration"]) +async def api_request_vram(req: RequestVramRequest): + """Force an immediate ComfyUI purge if there is not enough free VRAM for Ollama.""" + return await vram_arbitrator.arbitrator.request_vram_for_ollama(req.needed_gb) + +@app.post("/api/warm-all", summary="Pre-warm Models into RAM Cache", tags=["Memory Optimization"]) +async def api_warm_all(req: Optional[WarmAllRequest] = None): + """Warm the highest-value models into the page cache within a byte budget.""" + return await ram_optimizer.warm_all_models(budget_gb=req.budget_gb if req else None) + +@app.get("/api/warm-plan", summary="Preview the Warm Plan", tags=["Memory Optimization"]) +async def api_warm_plan(budget_gb: Optional[float] = Query(None, description="Override the byte budget")): + """Show what warming would read, in what order, and what it would skip โ€” without doing it.""" + return ram_optimizer.build_warm_plan(budget_gb) @app.post("/api/warm-model", summary="Pre-warm Single Model or File", tags=["Memory Optimization"]) async def api_warm_model(req: WarmRequest): - """Pre-warm a specific Ollama model or individual file path into Linux RAM cache.""" + """Pre-warm a specific Ollama model or file path into the Linux page cache.""" + if req.model_name and req.blob_only: + return ram_optimizer.warm_ollama_blob(req.model_name) if req.model_name: return await ram_optimizer.warm_ollama_model(req.model_name, keep_alive="1m") - elif req.filepath: + if req.filepath: return ram_optimizer.warm_file_to_ram(req.filepath) - else: - raise HTTPException(status_code=400, detail="model_name or filepath required") + raise HTTPException(status_code=400, detail="model_name or filepath required") + +@app.get("/api/cache/report", summary="Measured Page-Cache Residency", tags=["Memory Optimization"]) +async def api_cache_report(files: bool = Query(True), refresh: bool = Query(False)): + """Measured (not assumed) page-cache residency for every model on disk.""" + report = ram_optimizer.get_cache_report(include_files=files, force_refresh=refresh) + report["capability"] = ram_optimizer.residency_capability() + return report @app.get("/api/models", summary="List All Installed Models", tags=["Catalog"]) -async def api_get_models(): - """List all installed Ollama models and discovered ComfyUI model checkpoints/safetensors on disk with sizes and quantization levels.""" +async def api_get_models(refresh: bool = Query(False)): + """All installed Ollama models (with their on-disk blobs) and ComfyUI checkpoints.""" + catalog = ram_optimizer.get_model_catalog(force_refresh=refresh) ollama_state = await vram_arbitrator.get_ollama_live_state() - comfy_models = ram_optimizer.find_comfy_model_files() return { "ollama_models": ollama_state.get("installed_models", []), - "comfy_models": comfy_models, + "ollama_blobs": catalog["ollama"], + "comfy_models": catalog["comfy"], + "cached_at": catalog["cached_at"], } @app.get("/api/history", summary="Model Switch History Log", tags=["Analytics"]) -async def api_get_history(limit: int = Query(20, description="Max history items to return")): - """Get the recent history log of model switch events, swap durations (in ms), and RAM cache hit status.""" - history = vram_arbitrator.get_switch_history() - return history[:limit] +async def api_get_history(limit: int = Query(20, description="Max history items to return"), + durable: bool = Query(False, description="Read from the persistent store instead of the in-memory ring")): + """Recent swap events, durations, achieved bandwidth and cache status.""" + if durable: + return telemetry_store.recent_events(limit) + return vram_arbitrator.get_switch_history()[:limit] @app.post("/api/benchmark", summary="Run Latency Benchmark", tags=["Analytics"]) async def api_run_benchmark(req: BenchmarkRequest): - """Run an automated benchmark swapping between available models to measure round-trip latency and RAM cache effectiveness.""" + """Automated round-trip switch benchmark measuring latency and cache effectiveness.""" from mcp_server import run_model_switch_benchmark res_str = await run_model_switch_benchmark(iterations=req.iterations or 2) return json.loads(res_str) + +# ========================================== +# ANALYTICS (persisted) +# ========================================== + +@app.get("/api/analytics/profiles", summary="Which Overclock Profile Is Actually Faster", tags=["Analytics"]) +async def api_analytics_profiles(days: float = Query(7.0)): + """Decode throughput and thermals grouped by the profile that was active at the time.""" + return {"window_days": days, "profiles": telemetry_store.profile_comparison(days)} + +@app.get("/api/analytics/swaps", summary="Swap Statistics", tags=["Analytics"]) +async def api_analytics_swaps(days: float = Query(7.0)): + """Aggregated swap/yield/purge latencies, cache-hit split and per-model throughput.""" + return telemetry_store.swap_stats(days) + +@app.get("/api/analytics/timeseries", summary="Downsampled Telemetry History", tags=["Analytics"]) +async def api_analytics_timeseries(hours: float = Query(6.0), buckets: int = Query(240)): + """Long-range history for charts that outlive a page refresh.""" + return {"hours": hours, "points": telemetry_store.timeseries(hours, buckets)} + +@app.get("/api/analytics/models", summary="Model Usage Ranking", tags=["Analytics"]) +async def api_analytics_models(days: float = Query(30.0)): + """Recency/frequency ranking used to prioritise the RAM warm budget.""" + return {"window_days": days, "models": telemetry_store.model_usage_ranking(days)} + +@app.get("/api/db", summary="Telemetry Store Info", tags=["Analytics"]) +async def api_db_info(): + """Where the persistent store lives and how much history it holds.""" + return telemetry_store.db_info() + + # ========================================== # OVERCLOCK MANAGEMENT # ========================================== @app.get("/api/overclock", summary="Overclock Status & Profiles", tags=["Overclock"]) async def api_overclock_status(): - """Get live GPU overclock state, active profile, and all per-app profiles.""" - return overclock_manager.get_status() + """Live GPU overclock state, active profile, governor state and all per-app profiles.""" + status = overclock_manager.get_status() + status["governor"] = thermal_governor.governor.get_status() + return status @app.post("/api/overclock/apply", summary="Apply Overclock Profile", tags=["Overclock"]) async def api_overclock_apply(req: OverclockApplyRequest): - """Apply a named overclock profile (ollama | comfy | balanced) to the GPU immediately.""" + """Apply a named overclock profile (ollama | comfy | balanced) immediately.""" res = overclock_manager.apply_profile(req.profile) if not res.get("success"): raise HTTPException(status_code=400, detail=res.get("error")) return res +@app.post("/api/overclock/restore", summary="Restore Stock GPU State", tags=["Overclock"]) +async def api_overclock_restore(): + """Drop all clock locks and offsets, restore default power limit and automatic fans.""" + return overclock_manager.restore_safe("manual request") + @app.get("/api/overclock/profiles", summary="List Overclock Profiles", tags=["Overclock"]) async def api_overclock_profiles(): - """List all overclock profiles with their current settings.""" + """All overclock profiles with their current settings.""" return overclock_manager.get_profiles() @app.post("/api/overclock/profiles/{name}", summary="Update Overclock Profile", tags=["Overclock"]) @@ -216,7 +427,7 @@ async def api_overclock_update_profile(name: str, req: OverclockProfileUpdate): @app.get("/api/overclock/fan", summary="Get GPU Fan Status", tags=["Overclock"]) @app.get("/api/gpu/fan", summary="Get GPU Fan Status", tags=["Overclock"]) async def api_get_fan_status(): - """Get current GPU fan control mode and speed.""" + """Current GPU fan control mode and speed.""" return overclock_manager.get_fan_status() @app.post("/api/overclock/fan", summary="Set GPU Fan Speed", tags=["Overclock"]) @@ -228,12 +439,59 @@ async def api_set_fan(req: FanRequest): return overclock_manager.set_fan_speed(pct) return overclock_manager.set_fan_auto() + +# ========================================== +# THERMAL GOVERNOR +# ========================================== + +@app.get("/api/governor", summary="Thermal Governor State", tags=["Governor"]) +async def api_governor_status(): + """Current derate level, why it was applied, and the escalation history.""" + return thermal_governor.governor.get_status() + +@app.post("/api/governor", summary="Control the Thermal Governor", tags=["Governor"]) +async def api_governor_control(req: GovernorRequest): + """Enable/disable the governor, or clear an active derate.""" + if req.enabled is not None: + thermal_governor.governor.set_enabled(req.enabled) + if req.reset: + thermal_governor.governor.reset() + return thermal_governor.governor.get_status() + + +# ========================================== +# AUTOTUNE +# ========================================== + +@app.get("/api/autotune", summary="Autotune Status & History", tags=["Autotune"]) +async def api_autotune_status(): + """Sweep progress, the last result, and every recorded autotune step.""" + return autotune.get_status() + +@app.post("/api/autotune/sweep", summary="Run an Overclock Sweep", tags=["Autotune"]) +async def api_autotune_sweep(req: SweepRequest): + """Walk a clock offset upward, measuring tok/s and watching for instability at each step.""" + res = await autotune.sweep( + knob=req.knob, profile=req.profile, model=req.model, + start=req.start, stop=req.stop, step=req.step, + repeats=req.repeats, apply_best=req.apply_best, + ) + if not res.get("success"): + raise HTTPException(status_code=400, detail=res.get("error")) + return res + +@app.post("/api/autotune/cancel", summary="Cancel a Running Sweep", tags=["Autotune"]) +async def api_autotune_cancel(): + """Stop the current sweep after the step in flight; the profile is restored either way.""" + return autotune.cancel() + + # Mount static web UI files -app.mount("/static", StaticFiles(directory="/home/drjones/unified-model-manager/static"), name="static") +app.mount("/static", StaticFiles(directory=f"{BASE_DIR}/static"), name="static") @app.get("/", summary="Dashboard Web UI", tags=["UI"]) async def root_index(): - with open("/home/drjones/unified-model-manager/static/index.html", "r") as f: + with open(f"{BASE_DIR}/static/index.html", "r") as f: content = f.read() return HTMLResponse(content=content) diff --git a/static/app.js b/static/app.js index 686932a..b388f61 100644 --- a/static/app.js +++ b/static/app.js @@ -33,6 +33,9 @@ function initSSE() { function updateDashboard(data) { if (!data) return; + // Governor state rides along in the shared snapshot โ€” no extra polling needed. + if (data.governor) renderGovernor(data.governor); + // 1. GPU VRAM Stats const gpu = data.gpu || {}; const ram = data.ram || {}; @@ -646,3 +649,231 @@ async function saveOverclockProfile() { alert(`Error: ${err}`); } } + +// ============================================================================ +// THERMAL GOVERNOR / RESIDENCY / ANALYTICS / AUTOTUNE +// ============================================================================ + +let governorEnabled = true; + +function renderGovernor(gov) { + if (!gov) return; + governorEnabled = gov.enabled; + const levels = 3; + const el = (id) => document.getElementById(id); + if (!el('gov-label')) return; + + el('gov-label').textContent = gov.label || 'โ€”'; + el('gov-label').className = 'text-2xl font-bold ' + + (gov.level === 0 ? 'text-emerald-400' : gov.level < 3 ? 'text-amber-400' : 'text-rose-400'); + el('gov-level').textContent = `level ${gov.level} / ${levels}`; + el('gov-bar').style.width = `${(gov.level / levels) * 100}%`; + el('gov-esc').textContent = `${gov.escalate_at_c}ยฐC`; + el('gov-rec').textContent = `${gov.recover_below_c}ยฐC`; + el('gov-scale').textContent = `${Math.round((gov.offset_scale ?? 1) * 100)}%`; + el('gov-reason').textContent = gov.last_reason || 'โ€”'; + + const toggle = el('gov-toggle'); + toggle.textContent = gov.enabled ? 'Enabled' : 'Disabled'; + toggle.className = 'px-2.5 py-1 text-xs font-semibold rounded-lg border transition ' + + (gov.enabled ? 'bg-emerald-950/70 border-emerald-800 text-emerald-300 hover:bg-emerald-900' + : 'bg-slate-800 border-slate-700 text-slate-400 hover:bg-slate-700'); + + const hist = el('gov-history'); + hist.innerHTML = (gov.history || []).map(h => { + const t = new Date(h.ts * 1000).toLocaleTimeString(); + const up = h.to_level > h.from_level; + return `
+ ${t} ${up ? 'โ–ฒ' : 'โ–ผ'} L${h.to_level} ${h.label} + ${h.reason || ''}
`; + }).join('') || '
No derates recorded โ€” the card has stayed happy.
'; +} + +async function toggleGovernor() { + const res = await fetch('/api/governor', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ enabled: !governorEnabled }) + }); + renderGovernor(await res.json()); +} + +async function resetGovernor() { + const res = await fetch('/api/governor', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ reset: true }) + }); + renderGovernor(await res.json()); +} + +// ---------------------------------------------------------------- residency + +async function fetchCacheReport(refresh = false) { + const body = document.getElementById('cache-files'); + if (!body) return; + if (refresh) body.innerHTML = '
Scanningโ€ฆ
'; + try { + const res = await fetch(`/api/cache/report?refresh=${refresh ? 'true' : 'false'}`); + const d = await res.json(); + + document.getElementById('cache-total').textContent = + `${d.resident_total_gb} / ${d.catalog_total_gb} GB`; + document.getElementById('cache-meta').textContent = + `${d.residency_pct}% resident ยท ${d.warm_files}/${d.files_scanned} warm ยท ${d.scan_ms}ms`; + + const cap = d.capability || {}; + document.getElementById('cache-capability').innerHTML = cap.exact_everywhere + ? 'Exact measurement available for every model file.' + : `Ollama weights measured by read-rate probe โ€” ${cap.reason || ''}`; + + body.innerHTML = (d.files || []).slice(0, 40).map(f => { + const pct = f.resident_pct ?? 0; + const color = pct >= 90 ? 'bg-emerald-500' : pct >= 25 ? 'bg-amber-500' : 'bg-slate-700'; + const kind = f.kind === 'ollama' ? 'text-cyan-400' : 'text-fuchsia-400'; + return `
+
+ ${f.name} + ${pct}% ยท ${f.size_gb}GB +
+
+
+
+
`; + }).join(''); + } catch (e) { + body.innerHTML = `
Residency scan failed: ${e}
`; + } +} + +async function fetchWarmPlan() { + const body = document.getElementById('cache-files'); + body.innerHTML = '
Building planโ€ฆ
'; + try { + const d = await (await fetch('/api/warm-plan')).json(); + document.getElementById('cache-total').textContent = `${d.planned_gb} GB planned`; + document.getElementById('cache-meta').textContent = + `budget ${d.budget_gb} GB ยท ${d.warm_count} to warm ยท ${d.skipped_count} skipped`; + document.getElementById('cache-capability').innerHTML = + 'Ranked by recency/frequency; already-resident files cost nothing.'; + body.innerHTML = d.plan.map((p, i) => `
+ ${i + 1}. ${p.name} + ${p.size_gb}GB ยท ${p.resident_pct}% ยท s=${p.score} +
`).join('') || '
Everything worth warming is already resident.
'; + } catch (e) { + body.innerHTML = `
${e}
`; + } +} + +// ---------------------------------------------------------------- analytics + +async function fetchAnalytics() { + const el = document.getElementById('analytics-body'); + if (!el) return; + const days = document.getElementById('analytics-days').value; + try { + const d = await (await fetch(`/api/analytics/profiles?days=${days}`)).json(); + if (!d.profiles || !d.profiles.length) { + el.innerHTML = '
No swaps recorded yet in this window. Switch a model to start building the comparison.
'; + return; + } + const best = Math.max(...d.profiles.map(p => p.avg_tok_s || 0)); + el.innerHTML = ` + + + ` + + d.profiles.map(p => { + const win = (p.avg_tok_s || 0) === best; + return ` + + + + `; + }).join('') + '
profiletok/sload GB/sยฐC avgW avgSM MHzn
${win ? 'โ˜… ' : ''}${p.profile ?? 'โ€”'}${p.avg_tok_s ?? 'โ€”'}${p.avg_load_gbps ?? 'โ€”'}${p.avg_temp_c ?? 'โ€”'}${p.avg_power_w ?? 'โ€”'}${p.avg_clock_sm ?? 'โ€”'}${p.swaps}
'; + } catch (e) { + el.innerHTML = `
${e}
`; + } +} + +// ---------------------------------------------------------------- autotune + +let sweepPoll = null; + +async function runSweep() { + const btn = document.getElementById('at-run'); + const body = document.getElementById('autotune-body'); + const payload = { + knob: document.getElementById('at-knob').value, + profile: document.getElementById('at-profile').value, + start: parseInt(document.getElementById('at-start').value, 10), + stop: parseInt(document.getElementById('at-stop').value, 10), + step: parseInt(document.getElementById('at-step').value, 10), + apply_best: document.getElementById('at-apply').checked, + }; + btn.disabled = true; + btn.classList.add('opacity-50'); + body.innerHTML = '
Sweepingโ€ฆ each step loads the model and benchmarks decode. This takes a few minutes.
'; + sweepPoll = setInterval(pollSweep, 2000); + try { + const res = await fetch('/api/autotune/sweep', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(payload) + }); + const d = await res.json(); + renderSweep(res.ok ? d : { error: d.detail }); + } catch (e) { + body.innerHTML = `
${e}
`; + } finally { + clearInterval(sweepPoll); + btn.disabled = false; + btn.classList.remove('opacity-50'); + document.getElementById('autotune-state').textContent = 'idle'; + } +} + +async function pollSweep() { + try { + const d = await (await fetch('/api/autotune')).json(); + const s = document.getElementById('autotune-state'); + if (d.running && d.current) { + s.textContent = `${d.current.knob}=${d.current.value} โ†’ ${d.current.tokens_per_sec} tok/s`; + s.className = 'text-xs font-mono text-fuchsia-400'; + } + } catch (e) { /* transient */ } +} + +async function cancelSweep() { + await fetch('/api/autotune/cancel', { method: 'POST' }); + document.getElementById('autotune-state').textContent = 'cancellingโ€ฆ'; +} + +function renderSweep(d) { + const body = document.getElementById('autotune-body'); + if (d.error) { body.innerHTML = `
${d.error}
`; return; } + const rows = (d.table || []).map(r => ` + ${r.value}${r.tokens_per_sec}${r.temp_c} + ${r.power_w ?? 'โ€”'}${r.clock_mem_mhz ?? 'โ€”'} + ${r.instability || 'stable'}`).join(''); + const best = d.best + ? `
โ˜… Best stable: ${d.knob}=${d.best.value} + โ†’ ${d.best.tokens_per_sec} tok/s at ${d.best.temp_c}ยฐC + ${d.gain_pct != null ? `(${d.gain_pct > 0 ? '+' : ''}${d.gain_pct}% vs baseline)` : ''} + ${d.applied_to_profile ? 'ยท saved to profile' : ''}
` + : '
No stable step produced throughput.
'; + const unstable = d.first_unstable + ? `
Stopped climbing at ${d.first_unstable.value}: ${d.first_unstable.why}
` : ''; + body.innerHTML = best + unstable + ` + + + ${rows}
offsettok/sยฐCWmem MHzstatus
`; +} + +// ---------------------------------------------------------------- wiring + +document.addEventListener('DOMContentLoaded', () => { + fetchCacheReport(false); + fetchAnalytics(); + setInterval(fetchAnalytics, 60000); + setInterval(() => fetchCacheReport(false), 30000); + fetch('/api/autotune').then(r => r.json()).then(d => { + if (d.last_result) renderSweep(d.last_result); + }).catch(() => {}); +}); diff --git a/static/index.html b/static/index.html index 7d6e7ae..3d9fb4b 100644 --- a/static/index.html +++ b/static/index.html @@ -665,6 +665,144 @@ + +
+ + +
+
+
+
+ +
+
+

Thermal Governor

+

Walks the overclock back when the card complains

+
+
+
+ + +
+
+
+
+ full + level 0 / 3 +
+
+
+
+
+
+
Escalate
+
83ยฐC
+
+
+
Recover
+
72ยฐC
+
+
+
Offset Scale
+
100%
+
+
+

Last action: cold start

+
+
+
+ + +
+
+
+
+ +
+
+

Measured Page-Cache Residency

+

What is genuinely in RAM, not what we hope is

+
+
+
+ + +
+
+
+
+ โ€” GB + โ€” +
+
+
+
+
+ + +
+
+
+
+ +
+
+

Is the Overclock Actually Working?

+

Decode throughput per profile, from persisted history

+
+
+ +
+
Collecting dataโ€ฆ
+
+ + +
+
+
+
+ +
+
+

Overclock Autotune

+

Sweep a clock offset, measure tok/s, stop at instability

+
+
+ idle +
+
+ + + + + + +
+
+ + + Restores the profile when done, even on error. +
+
+
+ +
+ diff --git a/telemetry_store.py b/telemetry_store.py new file mode 100644 index 0000000..e846250 --- /dev/null +++ b/telemetry_store.py @@ -0,0 +1,400 @@ +"""SQLite time-series persistence for HyperSwap telemetry, swap events and autotune runs. + +Everything the arbitrator learns used to die with the process (SWITCH_HISTORY was an +in-memory deque of 50). This module keeps it on disk so we can answer the question the +whole app exists to answer: does a given overclock profile actually deliver more tok/s? + +Design notes: + * WAL mode + a single writer thread -> the 1Hz sampler never blocks the event loop. + * Telemetry rows are batched and flushed every FLUSH_INTERVAL_S. + * Retention pruning runs opportunistically, keeping the DB bounded and small. +""" +import logging +import os +import queue +import sqlite3 +import threading +import time +from typing import Any, Dict, List, Optional + +logger = logging.getLogger("telemetry_store") + +_BASE = os.path.dirname(os.path.abspath(__file__)) +DB_PATH = os.environ.get("HYPERSWAP_DB", os.path.join(_BASE, "hyperswap.db")) + +TELEMETRY_RETENTION_DAYS = float(os.environ.get("HYPERSWAP_TELEMETRY_RETENTION_DAYS", "14")) +EVENT_RETENTION_DAYS = float(os.environ.get("HYPERSWAP_EVENT_RETENTION_DAYS", "180")) +FLUSH_INTERVAL_S = 2.0 +PRUNE_INTERVAL_S = 3600.0 + +SCHEMA = """ +PRAGMA journal_mode=WAL; +PRAGMA synchronous=NORMAL; + +CREATE TABLE IF NOT EXISTS telemetry ( + ts REAL NOT NULL, + profile TEXT, + gpu_util_pct REAL, + mem_util_pct REAL, + temp_c REAL, + power_w REAL, + power_limit_w REAL, + fan_pct REAL, + clock_sm_mhz REAL, + clock_mem_mhz REAL, + vram_used_bytes INTEGER, + ollama_bytes INTEGER, + comfy_bytes INTEGER, + system_bytes INTEGER, + ram_used_bytes INTEGER, + ram_cached_bytes INTEGER, + pcie_tx_kbps INTEGER, + pcie_rx_kbps INTEGER, + throttle_reasons TEXT +); +CREATE INDEX IF NOT EXISTS idx_telemetry_ts ON telemetry(ts); + +CREATE TABLE IF NOT EXISTS events ( + ts REAL NOT NULL, + event_type TEXT, + source TEXT, + target TEXT, + profile TEXT, + duration_ms REAL, + load_duration_ms REAL, + yield_confirm_ms REAL, + tokens_per_sec REAL, + bytes_loaded INTEGER, + load_gbps REAL, + cache_status TEXT, + detail TEXT +); +CREATE INDEX IF NOT EXISTS idx_events_ts ON events(ts); +CREATE INDEX IF NOT EXISTS idx_events_type ON events(event_type); + +CREATE TABLE IF NOT EXISTS autotune_runs ( + ts REAL NOT NULL, + profile TEXT, + knob TEXT, + core_offset_mhz INTEGER, + mem_offset_mhz INTEGER, + tokens_per_sec REAL, + load_gbps REAL, + temp_c REAL, + power_w REAL, + stable INTEGER, + instability TEXT, + note TEXT +); +CREATE INDEX IF NOT EXISTS idx_autotune_ts ON autotune_runs(ts); +""" + + +class _Writer(threading.Thread): + """Single background writer: batches telemetry, commits events immediately.""" + + daemon = True + + def __init__(self) -> None: + super().__init__(name="telemetry-writer") + self.q: "queue.Queue[Optional[tuple]]" = queue.Queue(maxsize=10000) + self._stop = threading.Event() + self._last_prune = 0.0 + + def run(self) -> None: + conn = sqlite3.connect(DB_PATH) + conn.executescript(SCHEMA) + conn.commit() + pending: List[tuple] = [] + last_flush = time.time() + while not self._stop.is_set(): + try: + item = self.q.get(timeout=0.5) + except queue.Empty: + item = None + if item is not None: + kind, sql, params = item + if kind == "telemetry": + pending.append((sql, params)) + else: + try: + conn.execute(sql, params) + conn.commit() + except Exception as e: + logger.warning("event write failed: %s", e) + now = time.time() + if pending and (now - last_flush) >= FLUSH_INTERVAL_S: + try: + for sql, params in pending: + conn.execute(sql, params) + conn.commit() + except Exception as e: + logger.warning("telemetry flush failed: %s", e) + pending.clear() + last_flush = now + if now - self._last_prune > PRUNE_INTERVAL_S: + self._last_prune = now + try: + conn.execute("DELETE FROM telemetry WHERE ts < ?", + (now - TELEMETRY_RETENTION_DAYS * 86400,)) + conn.execute("DELETE FROM events WHERE ts < ?", + (now - EVENT_RETENTION_DAYS * 86400,)) + conn.commit() + except Exception as e: + logger.debug("prune failed: %s", e) + try: + for sql, params in pending: + conn.execute(sql, params) + conn.commit() + conn.close() + except Exception: + pass + + def stop(self) -> None: + self._stop.set() + + +_writer: Optional[_Writer] = None +_writer_lock = threading.Lock() + + +def start() -> None: + global _writer + with _writer_lock: + if _writer is None or not _writer.is_alive(): + _writer = _Writer() + _writer.start() + logger.info("telemetry store started at %s", DB_PATH) + + +def stop() -> None: + global _writer + with _writer_lock: + if _writer is not None: + _writer.stop() + _writer.join(timeout=3.0) + _writer = None + + +def _submit(kind: str, sql: str, params: tuple) -> None: + w = _writer + if w is None: + return + try: + w.q.put_nowait((kind, sql, params)) + except queue.Full: + logger.debug("telemetry queue full, dropping sample") + + +_TELEMETRY_SQL = """ +INSERT INTO telemetry (ts, profile, gpu_util_pct, mem_util_pct, temp_c, power_w, power_limit_w, + fan_pct, clock_sm_mhz, clock_mem_mhz, vram_used_bytes, ollama_bytes, comfy_bytes, system_bytes, + ram_used_bytes, ram_cached_bytes, pcie_tx_kbps, pcie_rx_kbps, throttle_reasons) +VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) +""" + + +def record_telemetry(gpu: Dict[str, Any], ram: Dict[str, Any], profile: Optional[str] = None, + throttle_reasons: Optional[str] = None) -> None: + """Persist a single 1Hz telemetry sample. Never raises.""" + try: + if not gpu.get("available"): + return + bd = gpu.get("breakdown", {}) + _submit("telemetry", _TELEMETRY_SQL, ( + time.time(), profile, + gpu.get("gpu_util_pct"), gpu.get("mem_util_pct"), gpu.get("temperature_c"), + gpu.get("power_w"), gpu.get("power_limit_w"), gpu.get("fan_pct"), + gpu.get("clock_graphics_mhz"), gpu.get("clock_mem_mhz"), + gpu.get("vram_used_bytes"), + int(bd.get("ollama_gb", 0) * (1024 ** 3)), + int(bd.get("comfyui_gb", 0) * (1024 ** 3)), + int(bd.get("system_gb", 0) * (1024 ** 3)), + ram.get("used_bytes"), ram.get("cached_bytes"), + gpu.get("pcie_tx_kbps"), gpu.get("pcie_rx_kbps"), + throttle_reasons, + )) + except Exception as e: + logger.debug("record_telemetry failed: %s", e) + + +_EVENT_SQL = """ +INSERT INTO events (ts, event_type, source, target, profile, duration_ms, load_duration_ms, + yield_confirm_ms, tokens_per_sec, bytes_loaded, load_gbps, cache_status, detail) +VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?) +""" + + +def record_event(event: Dict[str, Any], profile: Optional[str] = None) -> None: + """Persist a swap/yield/purge event. Never raises.""" + try: + _submit("event", _EVENT_SQL, ( + event.get("ts", time.time()), + event.get("event_type"), event.get("source"), event.get("target"), + profile or event.get("profile"), + event.get("duration_ms"), event.get("load_duration_ms"), + event.get("yield_confirm_ms"), event.get("tokens_per_sec"), + event.get("bytes_loaded"), event.get("load_gbps"), + event.get("cache_status"), event.get("detail"), + )) + except Exception as e: + logger.debug("record_event failed: %s", e) + + +_AUTOTUNE_SQL = """ +INSERT INTO autotune_runs (ts, profile, knob, core_offset_mhz, mem_offset_mhz, tokens_per_sec, + load_gbps, temp_c, power_w, stable, instability, note) +VALUES (?,?,?,?,?,?,?,?,?,?,?,?) +""" + + +def record_autotune(row: Dict[str, Any]) -> None: + try: + _submit("event", _AUTOTUNE_SQL, ( + row.get("ts", time.time()), row.get("profile"), row.get("knob"), + row.get("core_offset_mhz"), row.get("mem_offset_mhz"), + row.get("tokens_per_sec"), row.get("load_gbps"), + row.get("temp_c"), row.get("power_w"), + 1 if row.get("stable") else 0, + row.get("instability"), row.get("note"), + )) + except Exception as e: + logger.debug("record_autotune failed: %s", e) + + +# ------------------------------------------------------------------ queries + +def _read_conn() -> sqlite3.Connection: + conn = sqlite3.connect(f"file:{DB_PATH}?mode=ro", uri=True, timeout=5.0) + conn.row_factory = sqlite3.Row + return conn + + +def _rows(sql: str, params: tuple = ()) -> List[Dict[str, Any]]: + if not os.path.exists(DB_PATH): + return [] + try: + with _read_conn() as conn: + return [dict(r) for r in conn.execute(sql, params).fetchall()] + except Exception as e: + logger.debug("query failed: %s", e) + return [] + + +def profile_comparison(days: float = 7.0) -> List[Dict[str, Any]]: + """The headline question: which overclock profile actually produces more tok/s? + + Joins decode throughput from switch events against thermals sampled while that + profile was active. + """ + since = time.time() - days * 86400 + perf = _rows(""" + SELECT profile, + COUNT(*) AS swaps, + AVG(tokens_per_sec) AS avg_tok_s, + MAX(tokens_per_sec) AS max_tok_s, + AVG(load_gbps) AS avg_load_gbps, + AVG(load_duration_ms) AS avg_load_ms + FROM events + WHERE ts > ? AND event_type = 'LLM Model Switch' AND tokens_per_sec > 0 + GROUP BY profile + """, (since,)) + thermals = {r["profile"]: r for r in _rows(""" + SELECT profile, + AVG(temp_c) AS avg_temp_c, + MAX(temp_c) AS max_temp_c, + AVG(power_w) AS avg_power_w, + AVG(clock_sm_mhz) AS avg_clock_sm, + AVG(clock_mem_mhz) AS avg_clock_mem, + COUNT(*) AS samples + FROM telemetry + WHERE ts > ? AND gpu_util_pct > 5 + GROUP BY profile + """, (since,))} + out = [] + for row in perf: + merged = dict(row) + merged.update(thermals.get(row["profile"], {})) + for k, v in list(merged.items()): + if isinstance(v, float): + merged[k] = round(v, 2) + out.append(merged) + out.sort(key=lambda r: r.get("avg_tok_s") or 0, reverse=True) + return out + + +def swap_stats(days: float = 7.0) -> Dict[str, Any]: + since = time.time() - days * 86400 + by_type = _rows(""" + SELECT event_type, COUNT(*) AS n, + AVG(duration_ms) AS avg_ms, MIN(duration_ms) AS min_ms, MAX(duration_ms) AS max_ms, + AVG(yield_confirm_ms) AS avg_confirm_ms + FROM events WHERE ts > ? GROUP BY event_type ORDER BY n DESC + """, (since,)) + cache = _rows(""" + SELECT cache_status, COUNT(*) AS n, AVG(load_gbps) AS avg_gbps + FROM events WHERE ts > ? AND cache_status IS NOT NULL GROUP BY cache_status + """, (since,)) + models = _rows(""" + SELECT target AS model, COUNT(*) AS loads, AVG(tokens_per_sec) AS avg_tok_s, + AVG(load_gbps) AS avg_gbps, MAX(ts) AS last_used + FROM events WHERE ts > ? AND event_type = 'LLM Model Switch' + GROUP BY target ORDER BY loads DESC LIMIT 25 + """, (since,)) + return {"by_type": by_type, "by_cache_status": cache, "by_model": models, "window_days": days} + + +def model_usage_ranking(days: float = 30.0) -> List[Dict[str, Any]]: + """Recency+frequency score per model, used to prioritise the RAM warm budget.""" + since = time.time() - days * 86400 + now = time.time() + rows = _rows(""" + SELECT target AS model, COUNT(*) AS loads, MAX(ts) AS last_used + FROM events WHERE ts > ? AND target IS NOT NULL AND event_type IN + ('LLM Model Switch','Model Warm') GROUP BY target + """, (since,)) + for r in rows: + age_h = max((now - (r["last_used"] or since)) / 3600.0, 0.01) + # frequency, decayed by recency (half-life ~24h) + r["score"] = round(r["loads"] * (0.5 ** (age_h / 24.0)) + 1.0 / age_h, 4) + r["age_hours"] = round(age_h, 2) + rows.sort(key=lambda r: r["score"], reverse=True) + return rows + + +def timeseries(hours: float = 6.0, buckets: int = 240) -> List[Dict[str, Any]]: + """Downsampled history for long-range dashboard charts.""" + since = time.time() - hours * 3600 + width = max((hours * 3600) / max(buckets, 1), 1.0) + return _rows(""" + SELECT CAST(ts / ? AS INTEGER) * ? AS bucket_ts, + AVG(gpu_util_pct) AS gpu_util_pct, AVG(temp_c) AS temp_c, + AVG(power_w) AS power_w, AVG(fan_pct) AS fan_pct, + AVG(vram_used_bytes) AS vram_used_bytes, + AVG(ollama_bytes) AS ollama_bytes, AVG(comfy_bytes) AS comfy_bytes, + AVG(ram_cached_bytes) AS ram_cached_bytes, + AVG(clock_sm_mhz) AS clock_sm_mhz, AVG(clock_mem_mhz) AS clock_mem_mhz + FROM telemetry WHERE ts > ? + GROUP BY bucket_ts ORDER BY bucket_ts + """, (width, width, since)) + + +def recent_events(limit: int = 50) -> List[Dict[str, Any]]: + return _rows("SELECT * FROM events ORDER BY ts DESC LIMIT ?", (limit,)) + + +def autotune_history(limit: int = 200) -> List[Dict[str, Any]]: + return _rows("SELECT * FROM autotune_runs ORDER BY ts DESC LIMIT ?", (limit,)) + + +def db_info() -> Dict[str, Any]: + info = {"path": DB_PATH, "exists": os.path.exists(DB_PATH)} + if info["exists"]: + info["size_mb"] = round(os.path.getsize(DB_PATH) / (1024 ** 2), 2) + for tbl in ("telemetry", "events", "autotune_runs"): + r = _rows(f"SELECT COUNT(*) AS n FROM {tbl}") + info[f"{tbl}_rows"] = r[0]["n"] if r else 0 + r = _rows("SELECT MIN(ts) AS a, MAX(ts) AS b FROM telemetry") + if r and r[0]["a"]: + info["coverage_hours"] = round((r[0]["b"] - r[0]["a"]) / 3600.0, 2) + return info diff --git a/thermal_governor.py b/thermal_governor.py new file mode 100644 index 0000000..1d8b1b6 --- /dev/null +++ b/thermal_governor.py @@ -0,0 +1,175 @@ +"""Thermal governor: walks the overclock back when the card says it is unhappy. + +Nothing in HyperSwap used to de-escalate. A profile pinned the fans to a manual PWM and +locked the clocks, and those settings stayed exactly where they were put no matter what +the card reported. This module watches the telemetry the sampler already collects (so it +costs no extra NVML calls) and derates the active profile when the GPU is hot or +throttling, then restores it once things settle. + +Hysteresis is deliberate: escalation needs HOT_SAMPLES consecutive bad samples and +recovery needs COOL_SAMPLES consecutive good ones, so a single spike during a diffusion +step does not cause profile thrash. +""" +import logging +import threading +import time +from typing import Any, Dict, List, Optional + +import overclock_manager + +logger = logging.getLogger("thermal_governor") + +# Derate ladder: each step scales the profile's clock offsets and raises the fan floor. +DERATE_LADDER = [ + {"level": 0, "offset_scale": 1.00, "fan_floor": 0, "label": "full"}, + {"level": 1, "offset_scale": 0.60, "fan_floor": 80, "label": "derated -40%"}, + {"level": 2, "offset_scale": 0.25, "fan_floor": 90, "label": "derated -75%"}, + {"level": 3, "offset_scale": 0.00, "fan_floor": 100, "label": "stock clocks, fans max"}, +] + +TEMP_ESCALATE_C = 83.0 +TEMP_RECOVER_C = 72.0 +HOT_SAMPLES = 5 # ~5 s at 1 Hz before we act +COOL_SAMPLES = 30 # ~30 s of calm before we give the clocks back +REAPPLY_COOLDOWN_S = 20.0 + +# Throttle reasons that mean the hardware is protecting itself, not just hitting a cap. +HARD_THROTTLES = {"hw_thermal_slowdown", "sw_thermal_slowdown", "hw_slowdown", + "hw_power_brake_slowdown"} + + +class ThermalGovernor: + def __init__(self) -> None: + self.enabled = True + self.level = 0 + self.hot_streak = 0 + self.cool_streak = 0 + self.last_change = 0.0 + self.last_reason = "cold start" + self.history: List[Dict[str, Any]] = [] + self._lock = threading.Lock() + self._base_profile: Optional[str] = None + + # ---------------------------------------------------------------- observation + + def observe(self, gpu: Dict[str, Any], active_profile: Optional[str] = None) -> None: + """Feed one telemetry sample. Cheap and non-blocking; actuation runs in a thread.""" + if not self.enabled or not gpu.get("available"): + return + temp = gpu.get("temperature_c") or 0 + reasons = set(gpu.get("throttle_reasons") or []) + hard = bool(reasons & HARD_THROTTLES) + + hot = temp >= TEMP_ESCALATE_C or hard + cool = temp <= TEMP_RECOVER_C and not hard + + with self._lock: + if hot: + self.hot_streak += 1 + self.cool_streak = 0 + elif cool: + self.cool_streak += 1 + self.hot_streak = 0 + else: + self.hot_streak = 0 + self.cool_streak = 0 + + now = time.time() + if now - self.last_change < REAPPLY_COOLDOWN_S: + return + + if self.hot_streak >= HOT_SAMPLES and self.level < len(DERATE_LADDER) - 1: + why = (f"{temp:.0f}ยฐC" if temp >= TEMP_ESCALATE_C else "") + \ + (f" throttling: {','.join(sorted(reasons & HARD_THROTTLES))}" if hard else "") + self._step(self.level + 1, why.strip(), active_profile, temp) + elif self.cool_streak >= COOL_SAMPLES and self.level > 0: + self._step(self.level - 1, f"stable at {temp:.0f}ยฐC", active_profile, temp) + + def _step(self, new_level: int, reason: str, active_profile: Optional[str], + temp: float) -> None: + old = self.level + self.level = new_level + self.hot_streak = self.cool_streak = 0 + self.last_change = time.time() + self.last_reason = reason + profile = active_profile or overclock_manager.ACTIVE_PROFILE + self._base_profile = profile + entry = { + "ts": time.time(), "from_level": old, "to_level": new_level, + "label": DERATE_LADDER[new_level]["label"], "reason": reason, + "temp_c": temp, "profile": profile, + } + self.history.insert(0, entry) + del self.history[50:] + direction = "escalating" if new_level > old else "recovering" + logger.warning(f"Thermal governor {direction} to level {new_level} " + f"({DERATE_LADDER[new_level]['label']}) โ€” {reason}") + threading.Thread(target=self._actuate, args=(profile,), daemon=True).start() + + def _actuate(self, profile: str) -> None: + try: + overclock_manager.apply_profile(profile, overrides=self.overrides_for(profile)) + except Exception as e: + logger.error(f"Governor failed to apply derate: {e}") + + # ---------------------------------------------------------------- overrides + + def overrides_for(self, profile: str) -> Dict[str, Any]: + """Translate the current derate level into profile overrides.""" + step = DERATE_LADDER[self.level] + if self.level == 0: + return {} + cfg = overclock_manager.load_profiles().get(profile, {}) + scale = step["offset_scale"] + overrides: Dict[str, Any] = { + "core_offset_mhz": int(cfg.get("core_offset_mhz", 0) * scale), + "mem_offset_mhz": int(cfg.get("mem_offset_mhz", 0) * scale), + } + if step["level"] >= 2: + # Stop pinning the core clock high when the card is already backing off. + overrides["lock_core_min"] = 0 + overrides["lock_core_max"] = 0 + floor = step["fan_floor"] + if floor: + overrides["fan_mode"] = "manual" + overrides["fan_speed_pct"] = max(int(cfg.get("fan_speed_pct", 0)), floor) + return overrides + + # ---------------------------------------------------------------- control + + def reset(self) -> Dict[str, Any]: + with self._lock: + self.level = 0 + self.hot_streak = self.cool_streak = 0 + self.last_change = time.time() + self.last_reason = "manual reset" + profile = self._base_profile or overclock_manager.ACTIVE_PROFILE + if profile and profile != "stock": + overclock_manager.apply_profile(profile) + return self.get_status() + + def set_enabled(self, enabled: bool) -> Dict[str, Any]: + self.enabled = bool(enabled) + if not enabled and self.level > 0: + self.reset() + return self.get_status() + + def get_status(self) -> Dict[str, Any]: + step = DERATE_LADDER[self.level] + return { + "enabled": self.enabled, + "level": self.level, + "label": step["label"], + "offset_scale": step["offset_scale"], + "fan_floor": step["fan_floor"], + "last_reason": self.last_reason, + "hot_streak": self.hot_streak, + "cool_streak": self.cool_streak, + "escalate_at_c": TEMP_ESCALATE_C, + "recover_below_c": TEMP_RECOVER_C, + "seconds_since_change": round(time.time() - self.last_change, 1) if self.last_change else None, + "history": self.history[:10], + } + + +governor = ThermalGovernor() diff --git a/vram_arbitrator.py b/vram_arbitrator.py index f1fcea2..6473fcb 100644 --- a/vram_arbitrator.py +++ b/vram_arbitrator.py @@ -11,6 +11,8 @@ import json import websockets import overclock_manager +import ram_optimizer +import telemetry_store try: import pynvml @@ -24,9 +26,116 @@ logger = logging.getLogger("vram_arbitrator") OLLAMA_API_BASE = "http://localhost:11434" COMFY_API_BASE = "http://127.0.0.1:8188" -# Circular buffer for transition events +# Circular buffer for transition events (the durable log lives in telemetry_store) SWITCH_HISTORY = deque(maxlen=50) +# Bandwidth thresholds used to classify how a model actually got into VRAM. +# PCIe 4.0 x16 tops out near 31.5 GB/s; this NVMe sustains well under 2 GB/s. +RAM_HIT_GBPS = 5.0 +PARTIAL_HIT_GBPS = 1.5 + +# How long Ollama's VRAM may take to actually drain before we stop waiting. +YIELD_CONFIRM_TIMEOUT_S = 3.0 +YIELD_CONFIRM_POLL_S = 0.02 +YIELD_RESIDUAL_BYTES = 256 * 1024 ** 2 # treat <256 MB as "released" + +# Connection-pooled clients. Re-creating an AsyncClient per call meant a fresh TCP +# handshake on every one of the watchdog's polls. +_clients: Dict[str, httpx.AsyncClient] = {} + + +def _client(base_url: str, timeout: float) -> httpx.AsyncClient: + key = f"{base_url}|{timeout}" + c = _clients.get(key) + if c is None or c.is_closed: + c = httpx.AsyncClient( + base_url=base_url, + timeout=timeout, + limits=httpx.Limits(max_keepalive_connections=4, max_connections=8), + ) + _clients[key] = c + return c + + +async def close_clients() -> None: + for c in list(_clients.values()): + try: + await c.aclose() + except Exception: + pass + _clients.clear() + + +# Bit flags from nvmlDeviceGetCurrentClocksThrottleReasons, decoded for the governor. +THROTTLE_REASONS = { + 0x0000000000000001: "gpu_idle", + 0x0000000000000002: "applications_clocks_setting", + 0x0000000000000004: "sw_power_cap", + 0x0000000000000008: "hw_slowdown", + 0x0000000000000010: "sync_boost", + 0x0000000000000020: "sw_thermal_slowdown", + 0x0000000000000040: "hw_thermal_slowdown", + 0x0000000000000080: "hw_power_brake_slowdown", + 0x0000000000000100: "display_clock_setting", +} + + +def decode_throttle_reasons(bits: int) -> List[str]: + return [name for mask, name in THROTTLE_REASONS.items() if bits & mask] + + +def get_process_vram_bytes() -> Dict[str, int]: + """Fast NVML-only VRAM attribution, used by the yield barrier's tight poll loop. + + Deliberately avoids psutil lookups: this runs every 20 ms while we wait for VRAM + to actually drain. + """ + out = {"ollama_bytes": 0, "comfyui_bytes": 0, "other_bytes": 0, "free_bytes": 0} + if not NVML_AVAILABLE: + return out + try: + handle = pynvml.nvmlDeviceGetHandleByIndex(0) + out["free_bytes"] = pynvml.nvmlDeviceGetMemoryInfo(handle).free + procs = list(pynvml.nvmlDeviceGetComputeRunningProcesses(handle)) + try: + procs += list(pynvml.nvmlDeviceGetGraphicsRunningProcesses(handle)) + except Exception: + pass + merged: Dict[int, int] = {} + for p in procs: + merged[p.pid] = max(merged.get(p.pid, 0), p.usedGpuMemory or 0) + for pid, used in merged.items(): + kind = _PID_KIND_CACHE.get(pid) + if kind is None: + kind = _classify_pid(pid) + _PID_KIND_CACHE[pid] = kind + if kind == "ollama": + out["ollama_bytes"] += used + elif kind == "comfy": + out["comfyui_bytes"] += used + else: + out["other_bytes"] += used + except Exception as e: + logger.debug(f"get_process_vram_bytes failed: {e}") + return out + + +_PID_KIND_CACHE: Dict[int, str] = {} + + +def _classify_pid(pid: int) -> str: + try: + proc = psutil.Process(pid) + pname = proc.name().lower() + cmdline = " ".join(proc.cmdline()).lower() + except Exception: + return "other" + if "ollama" in pname or "llama-server" in cmdline: + return "ollama" + if "comfy" in cmdline or "main.py" in cmdline: + return "comfy" + return "other" + def get_gpu_hardware_stats() -> Dict[str, Any]: """Retrieve comprehensive GPU hardware and process metrics via NVML.""" @@ -77,6 +186,49 @@ def get_gpu_hardware_stats() -> Dict[str, Any]: clock_graphics = 0 clock_mem = 0 + # PCIe throughput (KB/s) โ€” TX + RX. Key metric for the RAM-cache + # PCIe-speed swap thesis (assimilated from pmady/gpu-mcp-server). + pcie_tx_kbps = 0 + pcie_rx_kbps = 0 + try: + pcie_tx_kbps = pynvml.nvmlDeviceGetPcieThroughput(handle, pynvml.NVML_PCIE_UTIL_TX_BYTES) + except Exception: + pass + try: + pcie_rx_kbps = pynvml.nvmlDeviceGetPcieThroughput(handle, pynvml.NVML_PCIE_UTIL_RX_BYTES) + except Exception: + pass + + # Why the GPU is not running at full clocks โ€” the thermal governor reads this. + throttle_bits = 0 + throttle_reasons: List[str] = [] + try: + throttle_bits = pynvml.nvmlDeviceGetCurrentClocksThrottleReasons(handle) + throttle_reasons = decode_throttle_reasons(throttle_bits) + except Exception: + pass + + # Power management limit (watts) โ€” the OC ceiling. + power_limit_w = 0.0 + try: + power_limit_w = round(pynvml.nvmlDeviceGetPowerManagementLimit(handle) / 1000.0, 1) + except Exception: + pass + + # Driver + CUDA version (completeness). + driver_version = "" + cuda_version = "" + try: + dv = pynvml.nvmlSystemGetDriverVersion() + driver_version = dv.decode("utf-8") if isinstance(dv, bytes) else str(dv) + except Exception: + pass + try: + cv = pynvml.nvmlSystemGetCudaDriverVersion() + cuda_version = cv # int like 12030 == CUDA 12.3 + except Exception: + pass + # Discover processes on GPU proc_breakdown = { "ollama_bytes": 0, @@ -142,6 +294,13 @@ def get_gpu_hardware_stats() -> Dict[str, Any]: "mem_util_pct": util_rates.memory, "temperature_c": temp_c, "power_w": power_w, + "power_limit_w": power_limit_w, + "pcie_tx_kbps": pcie_tx_kbps, + "pcie_rx_kbps": pcie_rx_kbps, + "throttle_bits": throttle_bits, + "throttle_reasons": throttle_reasons, + "driver_version": driver_version, + "cuda_version": cuda_version, "fan_pct": fan_pct, "fans": fans, "num_fans": len(fans), @@ -174,25 +333,25 @@ async def get_ollama_live_state() -> Dict[str, Any]: "installed_models": [] } try: - async with httpx.AsyncClient(timeout=3.0) as client: - # Check running models (ps) - ps_resp = await client.get(f"{OLLAMA_API_BASE}/api/ps") - if ps_resp.status_code == 200: - state["online"] = True - models = ps_resp.json().get("models", []) - state["loaded_models"] = models - if models: - first = models[0] - state["active_model_name"] = first.get("name") - vram_bytes = first.get("size_vram", first.get("size", 0)) - state["active_model_vram_gb"] = round(vram_bytes / (1024**3), 2) - state["active_context"] = first.get("context_length", 0) - state["expires_at"] = first.get("expires_at") - - # Check all tags - tags_resp = await client.get(f"{OLLAMA_API_BASE}/api/tags") - if tags_resp.status_code == 200: - state["installed_models"] = tags_resp.json().get("models", []) + client = _client(OLLAMA_API_BASE, 3.0) + # Check running models (ps) + ps_resp = await client.get("/api/ps") + if ps_resp.status_code == 200: + state["online"] = True + models = ps_resp.json().get("models", []) + state["loaded_models"] = models + if models: + first = models[0] + state["active_model_name"] = first.get("name") + vram_bytes = first.get("size_vram", first.get("size", 0)) + state["active_model_vram_gb"] = round(vram_bytes / (1024**3), 2) + state["active_context"] = first.get("context_length", 0) + state["expires_at"] = first.get("expires_at") + + # Check all tags + tags_resp = await client.get("/api/tags") + if tags_resp.status_code == 200: + state["installed_models"] = tags_resp.json().get("models", []) except Exception as e: logger.debug(f"Ollama check error: {e}") @@ -211,156 +370,286 @@ async def get_comfyui_live_state() -> Dict[str, Any]: "vram_total_mb": 0, } try: - async with httpx.AsyncClient(timeout=3.0) as client: - # Check system stats - stats_resp = await client.get(f"{COMFY_API_BASE}/system_stats") - if stats_resp.status_code == 200: - state["online"] = True - data = stats_resp.json() - devices = data.get("devices", []) - if devices: - dev = devices[0] - state["vram_free_mb"] = round(dev.get("vram_free", 0) / (1024**2), 1) - state["vram_total_mb"] = round(dev.get("vram_total", 0) / (1024**2), 1) - - # Check queue - queue_resp = await client.get(f"{COMFY_API_BASE}/queue") - if queue_resp.status_code == 200: - qdata = queue_resp.json() - running = qdata.get("queue_running", []) - pending = qdata.get("queue_pending", []) - state["queue_running"] = len(running) - state["queue_remaining"] = len(pending) - state["executing"] = len(running) > 0 - if running: - state["current_prompt_id"] = running[0][1] if len(running[0]) > 1 else str(running[0]) + client = _client(COMFY_API_BASE, 3.0) + # Check system stats + stats_resp = await client.get("/system_stats") + if stats_resp.status_code == 200: + state["online"] = True + data = stats_resp.json() + devices = data.get("devices", []) + if devices: + dev = devices[0] + state["vram_free_mb"] = round(dev.get("vram_free", 0) / (1024**2), 1) + state["vram_total_mb"] = round(dev.get("vram_total", 0) / (1024**2), 1) + + # Check queue + queue_resp = await client.get("/queue") + if queue_resp.status_code == 200: + qdata = queue_resp.json() + running = qdata.get("queue_running", []) + pending = qdata.get("queue_pending", []) + state["queue_running"] = len(running) + state["queue_remaining"] = len(pending) + state["executing"] = len(running) > 0 + if running: + state["current_prompt_id"] = running[0][1] if len(running[0]) > 1 else str(running[0]) except Exception as e: logger.debug(f"ComfyUI check error: {e}") return state -async def instant_free_ollama_vram(model_name: Optional[str] = None) -> Dict[str, Any]: - """Tell Ollama to instantly yield VRAM without evicting from OS page cache.""" +async def _await_vram_release(baseline_bytes: int, + timeout_s: float = YIELD_CONFIRM_TIMEOUT_S) -> Dict[str, Any]: + """Block until Ollama's VRAM has actually drained, or we give up. + + Posting keep_alive:0 only *asks* Ollama to unload; the driver frees the allocation + some milliseconds later. Returning before that happens is how ComfyUI ends up + allocating into VRAM that is still occupied, which shows up as a CUDA OOM mid-graph. + """ + t0 = time.perf_counter() + last = baseline_bytes + while True: + snap = get_process_vram_bytes() + last = snap["ollama_bytes"] + if last <= YIELD_RESIDUAL_BYTES: + return { + "confirmed": True, + "confirm_ms": round((time.perf_counter() - t0) * 1000, 2), + "residual_bytes": last, + "free_bytes": snap["free_bytes"], + } + if (time.perf_counter() - t0) >= timeout_s: + return { + "confirmed": False, + "confirm_ms": round((time.perf_counter() - t0) * 1000, 2), + "residual_bytes": last, + "free_bytes": snap["free_bytes"], + "error": f"Ollama still holding {round(last / (1024**3), 2)} GB after {timeout_s}s", + } + await asyncio.sleep(YIELD_CONFIRM_POLL_S) + + +def _record(event: Dict[str, Any]) -> None: + """Push an event to both the in-memory ring and the durable store.""" + event.setdefault("ts", time.time()) + event.setdefault("timestamp", time.strftime("%H:%M:%S")) + SWITCH_HISTORY.appendleft(event) + telemetry_store.record_event(event, profile=overclock_manager.ACTIVE_PROFILE) + + +async def instant_free_ollama_vram(model_name: Optional[str] = None, + confirm: bool = True) -> Dict[str, Any]: + """Yield Ollama's VRAM and wait for the driver to actually release it. + + The returned duration_ms is now the real end-to-end release time, not just how long + the HTTP POST took. + """ t0 = time.perf_counter() if not model_name: ollama_state = await get_ollama_live_state() model_name = ollama_state.get("active_model_name") - + if not model_name: - return {"success": True, "message": "No active Ollama model in VRAM", "duration_ms": 0} - + return {"success": True, "message": "No active Ollama model in VRAM", + "duration_ms": 0, "confirmed": True} + + baseline = get_process_vram_bytes()["ollama_bytes"] try: - async with httpx.AsyncClient(timeout=5.0) as client: - resp = await client.post( - f"{OLLAMA_API_BASE}/api/generate", - json={"model": model_name, "keep_alive": 0}, - ) - duration_ms = round((time.perf_counter() - t0) * 1000, 2) - event = { - "timestamp": time.strftime("%H:%M:%S"), - "event_type": "Ollama VRAM Yield", - "source": model_name, - "target": "VRAM 0MB (Kept in RAM)", - "duration_ms": duration_ms, - "cache_status": "RAM-Cached", - } - SWITCH_HISTORY.appendleft(event) - return {"success": True, "model": model_name, "duration_ms": duration_ms} + client = _client(OLLAMA_API_BASE, 5.0) + await client.post("/api/generate", json={"model": model_name, "keep_alive": 0}) + request_ms = round((time.perf_counter() - t0) * 1000, 2) + + barrier = {"confirmed": None, "confirm_ms": 0.0, "residual_bytes": baseline} + if confirm: + barrier = await _await_vram_release(baseline) + + duration_ms = round((time.perf_counter() - t0) * 1000, 2) + freed_gb = round(max(baseline - barrier.get("residual_bytes", 0), 0) / (1024**3), 2) + + _record({ + "event_type": "Ollama VRAM Yield", + "source": model_name, + "target": "VRAM 0MB (Kept in RAM)", + "duration_ms": duration_ms, + "yield_confirm_ms": barrier.get("confirm_ms"), + "cache_status": "RAM-Cached" if barrier.get("confirmed") else "Yield Timeout", + "detail": barrier.get("error"), + }) + return { + "success": True, + "model": model_name, + "duration_ms": duration_ms, + "request_ms": request_ms, + "confirm_ms": barrier.get("confirm_ms"), + "confirmed": barrier.get("confirmed"), + "freed_gb": freed_gb, + "residual_gb": round(barrier.get("residual_bytes", 0) / (1024**3), 2), + "free_vram_gb": round(barrier.get("free_bytes", 0) / (1024**3), 2), + "error": barrier.get("error"), + } except Exception as e: - return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + return {"success": False, "error": str(e), + "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + async def instant_free_comfyui_vram() -> Dict[str, Any]: """Tell ComfyUI to purge loaded diffusion models from VRAM.""" t0 = time.perf_counter() try: - async with httpx.AsyncClient(timeout=5.0) as client: - resp = await client.post( - f"{COMFY_API_BASE}/free", - json={"unload_models": True, "free_memory": True}, - ) - duration_ms = round((time.perf_counter() - t0) * 1000, 2) - event = { - "timestamp": time.strftime("%H:%M:%S"), - "event_type": "ComfyUI VRAM Purge", - "source": "ComfyUI Pipeline", - "target": "VRAM Free", - "duration_ms": duration_ms, - "cache_status": "Cleaned", - } - SWITCH_HISTORY.appendleft(event) - return {"success": True, "duration_ms": duration_ms} + client = _client(COMFY_API_BASE, 5.0) + await client.post("/free", json={"unload_models": True, "free_memory": True}) + duration_ms = round((time.perf_counter() - t0) * 1000, 2) + snap = get_process_vram_bytes() + _record({ + "event_type": "ComfyUI VRAM Purge", + "source": "ComfyUI Pipeline", + "target": "VRAM Free", + "duration_ms": duration_ms, + "cache_status": "Cleaned", + }) + return {"success": True, "duration_ms": duration_ms, + "free_vram_gb": round(snap["free_bytes"] / (1024**3), 2)} except Exception as e: - return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + return {"success": False, "error": str(e), + "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + + +_MODEL_SIZE_CACHE: Dict[str, int] = {} + + +def _model_size_bytes(model_name: str) -> int: + """On-disk weight size for an Ollama model, used to turn load time into bandwidth.""" + if model_name in _MODEL_SIZE_CACHE: + return _MODEL_SIZE_CACHE[model_name] + try: + for f in ram_optimizer.find_ollama_model_files(): + _MODEL_SIZE_CACHE[f["model"]] = f["size_bytes"] + except Exception as e: + logger.debug(f"model size lookup failed: {e}") + return _MODEL_SIZE_CACHE.get(model_name, 0) + + +def classify_load(size_bytes: int, load_duration_ms: float) -> Dict[str, Any]: + """Classify how a model reached VRAM, from achieved bandwidth rather than a constant. + + The old rule was `load_duration_ms < 2500`, which called a 27B Q2_K read from NVMe a + cache hit and a small model read from RAM a cold load. Bandwidth separates them + cleanly: page cache feeds PCIe at many GB/s, this NVMe does not. + """ + if load_duration_ms <= 1.0: + return {"cache_status": "Already in VRAM", "load_gbps": None, "is_ram_hit": True} + if not size_bytes: + # No size on record โ€” fall back to the old heuristic, but say so. + return { + "cache_status": "RAM Cache Hit โšก" if load_duration_ms < 2500 else "Cold Disk Load ๐Ÿ’พ", + "load_gbps": None, + "is_ram_hit": load_duration_ms < 2500, + "detail": "size unknown, fell back to duration heuristic", + } + gbps = (size_bytes / (1024**3)) / (load_duration_ms / 1000.0) + if gbps >= RAM_HIT_GBPS: + status = "RAM Cache Hit โšก" + elif gbps >= PARTIAL_HIT_GBPS: + status = "Partial Cache ๐ŸŒค" + else: + status = "Cold Disk Load ๐Ÿ’พ" + return {"cache_status": status, "load_gbps": round(gbps, 2), "is_ram_hit": gbps >= RAM_HIT_GBPS} + async def switch_ollama_model(target_model: str, keep_alive: str = "30m") -> Dict[str, Any]: """High-speed hot-swap to target Ollama model, tracking swap metrics.""" t0 = time.perf_counter() cur_state = await get_ollama_live_state() prev_model = cur_state.get("active_model_name") or "None" - + try: - async with httpx.AsyncClient(timeout=180.0) as client: - resp = await client.post( - f"{OLLAMA_API_BASE}/api/generate", - json={"model": target_model, "prompt": "Ready check", "stream": False, "keep_alive": keep_alive}, - ) - total_duration = time.perf_counter() - t0 - total_duration_ms = round(total_duration * 1000, 2) - - if resp.status_code == 200: - data = resp.json() - load_dur_ms = round(data.get("load_duration", 0) / 1e6, 2) - eval_dur_ms = round(data.get("eval_duration", 0) / 1e6, 2) - eval_count = data.get("eval_count", 0) - tokens_per_sec = round((eval_count / (eval_dur_ms / 1000)) if eval_dur_ms > 0 else 0, 1) - - # Check if it was a RAM cache hit (load duration < 1500ms for large model indicates RAM hit) - is_ram_hit = load_dur_ms < 2500 - - event = { - "timestamp": time.strftime("%H:%M:%S"), - "event_type": "LLM Model Switch", - "source": prev_model, - "target": target_model, - "duration_ms": total_duration_ms, - "load_duration_ms": load_dur_ms, - "tokens_per_sec": tokens_per_sec, - "cache_status": "RAM Cache Hit โšก" if is_ram_hit else "Cold Disk Load ๐Ÿ’พ", - } - SWITCH_HISTORY.appendleft(event) - - return { - "success": True, - "prev_model": prev_model, - "target_model": target_model, - "total_duration_ms": total_duration_ms, - "load_duration_ms": load_dur_ms, - "tokens_per_sec": tokens_per_sec, - "is_ram_hit": is_ram_hit, - "response": data.get("response", ""), - } - else: - return {"success": False, "error": f"HTTP {resp.status_code}: {resp.text}", "duration_ms": total_duration_ms} + client = _client(OLLAMA_API_BASE, 180.0) + resp = await client.post( + "/api/generate", + json={"model": target_model, "prompt": "Ready check", "stream": False, + "keep_alive": keep_alive}, + ) + total_duration_ms = round((time.perf_counter() - t0) * 1000, 2) + + if resp.status_code == 200: + data = resp.json() + load_dur_ms = round(data.get("load_duration", 0) / 1e6, 2) + eval_dur_ms = round(data.get("eval_duration", 0) / 1e6, 2) + eval_count = data.get("eval_count", 0) + tokens_per_sec = round((eval_count / (eval_dur_ms / 1000)) if eval_dur_ms > 0 else 0, 1) + + size_bytes = _model_size_bytes(target_model) + cls = classify_load(size_bytes, load_dur_ms) + + _record({ + "event_type": "LLM Model Switch", + "source": prev_model, + "target": target_model, + "duration_ms": total_duration_ms, + "load_duration_ms": load_dur_ms, + "tokens_per_sec": tokens_per_sec, + "bytes_loaded": size_bytes, + "load_gbps": cls["load_gbps"], + "cache_status": cls["cache_status"], + "detail": cls.get("detail"), + }) + + return { + "success": True, + "prev_model": prev_model, + "target_model": target_model, + "total_duration_ms": total_duration_ms, + "load_duration_ms": load_dur_ms, + "tokens_per_sec": tokens_per_sec, + "model_size_gb": round(size_bytes / (1024**3), 2) if size_bytes else None, + "load_gbps": cls["load_gbps"], + "cache_status": cls["cache_status"], + "is_ram_hit": cls["is_ram_hit"], + "response": data.get("response", ""), + } + return {"success": False, "error": f"HTTP {resp.status_code}: {resp.text}", + "duration_ms": total_duration_ms} except Exception as e: - return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + return {"success": False, "error": str(e), + "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} + def get_switch_history() -> List[Dict[str, Any]]: return list(SWITCH_HISTORY) class AutoArbitrator: - """Real-time bidirectional background arbitrator for seamless Ollama <-> ComfyUI hot-swapping.""" + """Real-time bidirectional background arbitrator for seamless Ollama <-> ComfyUI hot-swapping. + + Two behavioural changes worth knowing about: + + * ComfyUI's VRAM is no longer purged 1.5 s after every finished prompt. Iterating on + a workflow is the common case, and purging between runs forced a full checkpoint + reload each time. The purge now waits for COMFY_IDLE_PURGE_S of genuinely empty + queue, and happens immediately only when Ollama actually needs the VRAM. + * The watchdog no longer polls two ComfyUI endpoints every 300 ms. The WebSocket is + the primary signal; polling is a fallback that runs at 1 Hz and only hits /queue, + backing off further while the socket is healthy. + """ + + COMFY_IDLE_PURGE_S = 30.0 + WATCHDOG_INTERVAL_S = 1.0 + WATCHDOG_INTERVAL_WS_OK_S = 3.0 def __init__(self): self.running = False self.ws_task: Optional[asyncio.Task] = None self.poll_task: Optional[asyncio.Task] = None + self.idle_task: Optional[asyncio.Task] = None self.last_yield_time = 0.0 self.last_comfy_free_time = 0.0 self.connected_ws = False self.last_action = "Idle" self.comfy_was_active = False + self.comfy_idle_since: Optional[float] = None self.oc_profile = None + self.pending_purge = False + self.stats = {"yields": 0, "purges": 0, "yield_timeouts": 0, "deferred_purges": 0} async def start(self): if self.running: @@ -368,24 +657,27 @@ class AutoArbitrator: self.running = True self.ws_task = asyncio.create_task(self._ws_listener()) self.poll_task = asyncio.create_task(self._poll_watchdog()) + self.idle_task = asyncio.create_task(self._idle_purge_loop()) logger.info("AutoArbitrator background engine started (Bidirectional).") - # Apply the default (balanced) overclock profile on startup. try: - await asyncio.get_event_loop().run_in_executor(None, overclock_manager.apply_profile, "balanced") + await asyncio.get_running_loop().run_in_executor( + None, overclock_manager.apply_profile, "balanced") + self.oc_profile = "balanced" except Exception as e: logger.warning(f"Startup overclock apply failed: {e}") async def stop(self): self.running = False - if self.ws_task: - self.ws_task.cancel() - if self.poll_task: - self.poll_task.cancel() + for task in (self.ws_task, self.poll_task, self.idle_task): + if task: + task.cancel() + await close_clients() logger.info("AutoArbitrator background engine stopped.") async def trigger_comfy_priority(self, reason: str = "ComfyUI prompt detected"): - """Instantly yield Ollama VRAM to 0MB when ComfyUI needs to run diffusion models.""" + """Yield Ollama's VRAM โ€” and confirm it is gone โ€” before diffusion allocates.""" self.comfy_was_active = True + self.comfy_idle_since = None self._apply_oc_profile("comfy") now = time.time() if now - self.last_yield_time < 1.0: @@ -394,89 +686,143 @@ class AutoArbitrator: ollama_state = await get_ollama_live_state() if ollama_state.get("active_model_name"): model = ollama_state["active_model_name"] - logger.info(f"โšก ComfyUI active ({reason}) -> Auto-yielding Ollama model '{model}' from VRAM...") + logger.info(f"โšก ComfyUI active ({reason}) -> Auto-yielding Ollama model '{model}'...") self.last_yield_time = time.time() - res = await instant_free_ollama_vram(model) - dur = res.get("duration_ms", 0) - self.last_action = f"Auto-yielded '{model}' for ComfyUI ({dur}ms)" + res = await instant_free_ollama_vram(model, confirm=True) + self.stats["yields"] += 1 + if res.get("confirmed"): + self.last_action = (f"Yielded '{model}' for ComfyUI in " + f"{res.get('confirm_ms')}ms (confirmed {res.get('freed_gb')}GB free)") + else: + self.stats["yield_timeouts"] += 1 + self.last_action = (f"โš  Yield of '{model}' NOT confirmed: " + f"{res.get('residual_gb')}GB still held") + logger.warning(f"VRAM yield barrier timed out: {res.get('error')}") logger.info(f"Ollama auto-yield completed: {res}") - async def trigger_comfy_completed(self): - """Purge ComfyUI VRAM cache when generation finishes, keeping VRAM 100% free for Ollama.""" + async def trigger_comfy_completed(self, immediate: bool = False): + """Mark the end of a generation. The actual purge is deferred unless forced.""" + self.comfy_was_active = False + if self.comfy_idle_since is None: + self.comfy_idle_since = time.time() + self._apply_oc_profile("ollama") + if immediate: + await self._purge_comfy_now("Ollama needs VRAM") + else: + self.pending_purge = True + self.stats["deferred_purges"] += 1 + self.last_action = (f"ComfyUI idle โ€” holding its checkpoints for " + f"{int(self.COMFY_IDLE_PURGE_S)}s in case you iterate") + + async def _purge_comfy_now(self, reason: str): now = time.time() if now - self.last_comfy_free_time < 3.0: return self.last_comfy_free_time = now - self.comfy_was_active = False - self._apply_oc_profile("ollama") - logger.info("โšก ComfyUI finished generation -> Auto-purging ComfyUI VRAM cache for Ollama...") + self.pending_purge = False + logger.info(f"โšก Purging ComfyUI VRAM cache ({reason})...") res = await instant_free_comfyui_vram() - dur = res.get("duration_ms", 0) - self.last_action = f"Auto-purged ComfyUI VRAM ({dur}ms) - Ready for Ollama" - logger.info(f"ComfyUI auto-purge completed: {res}") + self.stats["purges"] += 1 + self.last_action = f"Purged ComfyUI VRAM ({res.get('duration_ms')}ms) โ€” {reason}" + logger.info(f"ComfyUI purge completed: {res}") + + async def _idle_purge_loop(self): + """Purge ComfyUI's VRAM only after a real idle gap, not between iterations.""" + while self.running: + try: + if self.pending_purge and self.comfy_idle_since and not self.comfy_was_active: + idle_for = time.time() - self.comfy_idle_since + if idle_for >= self.COMFY_IDLE_PURGE_S: + await self._purge_comfy_now( + f"idle {int(idle_for)}s") + except Exception as e: + logger.debug(f"idle purge loop error: {e}") + await asyncio.sleep(2.0) + + async def request_vram_for_ollama(self, needed_gb: float = 0.0) -> Dict[str, Any]: + """Called when Ollama needs VRAM now: purge ComfyUI immediately rather than waiting.""" + snap = get_process_vram_bytes() + free_gb = snap["free_bytes"] / (1024**3) + if needed_gb and free_gb >= needed_gb: + return {"purged": False, "free_gb": round(free_gb, 2), "reason": "enough free VRAM"} + if snap["comfyui_bytes"] > YIELD_RESIDUAL_BYTES: + await self._purge_comfy_now(f"Ollama requested {needed_gb or '?'}GB") + snap = get_process_vram_bytes() + return {"purged": True, "free_gb": round(snap["free_bytes"] / (1024**3), 2)} + return {"purged": False, "free_gb": round(free_gb, 2), "reason": "ComfyUI holds no VRAM"} async def _ws_listener(self): client_id = "hyperswap-arbitrator" ws_url = f"ws://127.0.0.1:8188/ws?clientId={client_id}" + backoff = 2.0 while self.running: try: async with websockets.connect(ws_url, ping_interval=10, ping_timeout=10) as ws: self.connected_ws = True + backoff = 2.0 logger.info("AutoArbitrator connected to ComfyUI WebSocket.") while self.running: msg = await ws.recv() - if isinstance(msg, str): - try: - data = json.loads(msg) - msg_type = data.get("type") - msg_data = data.get("data", {}) + if not isinstance(msg, str): + continue + try: + data = json.loads(msg) + msg_type = data.get("type") + msg_data = data.get("data", {}) - if msg_type == "status": - queue_rem = msg_data.get("status", {}).get("exec_info", {}).get("queue_remaining", 0) - if queue_rem > 0: - await self.trigger_comfy_priority(f"Queue remaining: {queue_rem}") - elif queue_rem == 0 and self.comfy_was_active: - # Prompt queue finished - await asyncio.sleep(1.5) - await self.trigger_comfy_completed() - elif msg_type in ("execution_start", "execution_cached"): - await self.trigger_comfy_priority(f"Event: {msg_type}") - elif msg_type == "executing": - node = msg_data.get("node") - if node is not None: - await self.trigger_comfy_priority(f"Executing node: {node}") - elif node is None and self.comfy_was_active: - # Finished executing graph - await asyncio.sleep(1.5) - await self.trigger_comfy_completed() - elif msg_type == "execution_success": - await asyncio.sleep(1.5) + if msg_type == "status": + queue_rem = (msg_data.get("status", {}) + .get("exec_info", {}).get("queue_remaining", 0)) + if queue_rem > 0: + await self.trigger_comfy_priority(f"Queue remaining: {queue_rem}") + elif queue_rem == 0 and self.comfy_was_active: await self.trigger_comfy_completed() - except Exception as e: - logger.debug(f"WS parse error: {e}") + elif msg_type in ("execution_start", "execution_cached"): + await self.trigger_comfy_priority(f"Event: {msg_type}") + elif msg_type == "executing": + node = msg_data.get("node") + if node is not None: + await self.trigger_comfy_priority(f"Executing node: {node}") + elif self.comfy_was_active: + await self.trigger_comfy_completed() + elif msg_type == "execution_success": + await self.trigger_comfy_completed() + elif msg_type == "execution_error": + logger.warning(f"ComfyUI execution error: {msg_data}") + await self.trigger_comfy_completed() + except Exception as e: + logger.debug(f"WS parse error: {e}") except (websockets.exceptions.ConnectionClosed, OSError, asyncio.CancelledError): self.connected_ws = False except Exception as e: self.connected_ws = False logger.debug(f"WS connection error: {e}") - await asyncio.sleep(2.0) + await asyncio.sleep(backoff) + backoff = min(backoff * 1.5, 15.0) async def _poll_watchdog(self): - """Watchdog polling /queue every 300ms for robust bidirectional arbitration.""" + """Fallback for when the WebSocket is down. One cheap /queue call, 1 Hz. + + The previous version hit /system_stats and /queue every 300 ms on fresh TCP + connections โ€” roughly 6.6 requests/second against ComfyUI, forever. + """ while self.running: + interval = self.WATCHDOG_INTERVAL_WS_OK_S if self.connected_ws else self.WATCHDOG_INTERVAL_S try: - comfy_state = await get_comfyui_live_state() - is_executing = comfy_state.get("queue_running", 0) > 0 or comfy_state.get("queue_remaining", 0) > 0 or comfy_state.get("executing", False) - if is_executing: - await self.trigger_comfy_priority("Polling detected active queue/execution") - elif self.comfy_was_active and not is_executing: - await asyncio.sleep(1.5) - await self.trigger_comfy_completed() + client = _client(COMFY_API_BASE, 3.0) + resp = await client.get("/queue") + if resp.status_code == 200: + q = resp.json() + busy = len(q.get("queue_running", [])) > 0 or len(q.get("queue_pending", [])) > 0 + if busy: + await self.trigger_comfy_priority("Watchdog saw an active queue") + elif self.comfy_was_active: + await self.trigger_comfy_completed() except Exception: pass - await asyncio.sleep(0.3) + await asyncio.sleep(interval) def _apply_oc_profile(self, profile: str): """Apply an overclock profile in a background thread; only fire on transition.""" @@ -484,18 +830,25 @@ class AutoArbitrator: return self.oc_profile = profile try: - loop = asyncio.get_event_loop() + loop = asyncio.get_running_loop() loop.run_in_executor(None, overclock_manager.apply_profile, profile) logger.info(f"๐ŸŽ›๏ธ Overclock profile switched -> '{profile}'") except Exception as e: logger.warning(f"Overclock profile switch failed ({profile}): {e}") def get_status(self) -> Dict[str, Any]: + idle_for = (time.time() - self.comfy_idle_since) if self.comfy_idle_since else None return { "running": self.running, "connected_ws": self.connected_ws, "last_action": self.last_action, "mode": "Bidirectional Hot-Swap (ComfyUI <-> Ollama)", + "comfy_active": self.comfy_was_active, + "pending_purge": self.pending_purge, + "comfy_idle_s": round(idle_for, 1) if idle_for is not None else None, + "idle_purge_after_s": self.COMFY_IDLE_PURGE_S, + "oc_profile": self.oc_profile, + "counters": dict(self.stats), } From 63297dc49d596a50d4fd2613807ce15d0a5e1948 Mon Sep 17 00:00:00 2001 From: drjones Date: Fri, 28 Aug 2026 09:10:06 -0700 Subject: [PATCH 2/6] Verify autotune knobs against hardware before sweeping Sweeping a knob the driver ignores measures nothing but benchmark noise, and the tuner would then confidently report 'best = the highest value tried'. On this box (driver 595.84) nvidia-settings accepts GPUGraphicsClockOffset/GPUMemoryTransferRate Offset and silently discards them: assigning 0 reports success and reads back 250. The ollama profile's core_offset_mhz=35 and mem_offset_mhz=200 have therefore been doing nothing. - _knob_effective() applies a probe value and confirms the hardware actually moved before any sweep starts, choosing the candidate furthest from the current reading (probing with the maximum fails when the card already sits at its top clock). - Adds discrete clock-lock knobs (lock_mem_mhz, lock_core_max) driven by the card's own supported-clock list, since -lmc/-lgc do work where offsets do not. - Reasoning models return their output in 'thinking' with an empty 'response', which the degeneracy check was flagging as corruption. Token count is now the primary signal. - Separates gain-vs-current-setting from gain-vs-slowest-value-tried. Reporting the latter as 'gain vs baseline' implied a +102% speedup that nobody would observe. Co-Authored-By: Claude Opus 5 --- autotune.py | 172 ++++++++++++++++++++++++++++++++++++++++++++------ static/app.js | 6 +- 2 files changed, 157 insertions(+), 21 deletions(-) diff --git a/autotune.py b/autotune.py index c819096..8dd6cf5 100644 --- a/autotune.py +++ b/autotune.py @@ -32,11 +32,96 @@ SETTLE_S = 2.5 TEMP_CEILING_C = 84.0 KNOBS = { - "mem_offset_mhz": {"default_start": 0, "default_stop": 1000, "default_step": 100}, - "core_offset_mhz": {"default_start": 0, "default_stop": 300, "default_step": 25}, + # Clock offsets go through nvidia-settings. On some drivers (595.84 here) the + # attribute is accepted and then silently ignored -- assigning 0 reports success and + # reads back 250 -- so a sweep of these can measure pure noise. _knob_effective() + # checks before any sweep runs. + "mem_offset_mhz": {"default_start": 0, "default_stop": 1000, "default_step": 100, + "kind": "offset", "verify": "mem_offset"}, + "core_offset_mhz": {"default_start": 0, "default_stop": 300, "default_step": 25, + "kind": "offset", "verify": "core_offset"}, + # Clock locks go through nvidia-smi and do work on the open/proprietary module alike. + # Memory clock is the lever that matters for LLM decode, which is bandwidth bound. + "lock_mem_mhz": {"kind": "discrete", "verify": "clock_mem", + "values": None}, # filled from the card's supported clock list + "lock_core_max": {"kind": "discrete", "verify": "clock_sm", + "values": None}, } +def _supported_clocks(which: str = "mem") -> List[int]: + """Discrete clock values the card will actually accept for -lmc / -lgc.""" + try: + proc = subprocess.run( + ["nvidia-smi", f"--query-supported-clocks={'mem' if which == 'mem' else 'gr'}", + "--format=csv,noheader,nounits"], + capture_output=True, text=True, timeout=10) + col = 0 if which == "mem" else 1 + vals = set() + for line in proc.stdout.splitlines(): + parts = [p.strip() for p in line.split(",")] + if len(parts) > col and parts[col].isdigit(): + vals.add(int(parts[col])) + return sorted(vals) + except Exception as e: + logger.debug(f"supported clock query failed: {e}") + return [] + + +def _read_hw(field: str) -> Optional[float]: + """Read back the hardware state a knob is supposed to move.""" + gpu = vram_arbitrator.get_gpu_hardware_stats() + if field == "clock_mem": + return gpu.get("clock_mem_mhz") + if field == "clock_sm": + return gpu.get("clock_graphics_mhz") + if field in ("mem_offset", "core_offset"): + r = overclock_manager._nvidia_settings( + "-q", f"[gpu:0]/{'GPUMemoryTransferRateOffset' if field == 'mem_offset' else 'GPUGraphicsClockOffset'}[3]") + for line in (r.get("out") or "").splitlines(): + if "Attribute" in line and "):" in line: + try: + return float(line.split("):")[-1].split(".")[0].strip()) + except Exception: + pass + return None + + +def _knob_effective(knob: str, profile: str, candidates: List[int], + baseline_value: int) -> Dict[str, Any]: + """Verify a knob actually moves the hardware before we sweep it. + + Without this the tuner happily reports "best = the highest value tried" from run-to-run + benchmark noise on a knob the driver is ignoring. + + The probe value is chosen as the candidate *furthest* from where the hardware currently + sits. Probing with the maximum is not good enough: if the card already happens to be at + its top clock, setting it there again moves nothing and a perfectly good knob looks + broken. + """ + field = KNOBS[knob].get("verify") + before = _read_hw(field) + if before is not None and candidates: + probe_value = max(candidates, key=lambda v: abs(v - before)) + else: + probe_value = candidates[-1] if candidates else 0 + overclock_manager.apply_profile(profile, overrides={knob: probe_value}) + time.sleep(2.0) + after = _read_hw(field) + overclock_manager.apply_profile(profile, overrides={knob: baseline_value}) + moved = (before is not None and after is not None and abs(after - before) > 1e-6) + return { + "effective": bool(moved), + "field": field, + "before": before, + "after": after, + "probe_value": probe_value, + "detail": (f"{field} moved {before} -> {after}" if moved else + f"{field} stayed at {after} after setting {knob}={probe_value}; " + f"this driver accepts the setting and ignores it"), + } + + def _xid_since(since_ts: float) -> List[str]: """Look for NVIDIA Xid errors in the kernel log โ€” the clearest instability signal.""" try: @@ -70,7 +155,9 @@ async def _decode_benchmark(model: str) -> Dict[str, Any]: data = resp.json() eval_ms = data.get("eval_duration", 0) / 1e6 eval_count = data.get("eval_count", 0) - text = data.get("response", "") or "" + # Reasoning models put their output in `thinking` and leave `response` empty, so a + # check that only looked at `response` flagged every one of them as degenerate. + text = ((data.get("response") or "") + " " + (data.get("thinking") or "")).strip() return { "ok": True, "tokens_per_sec": round(eval_count / (eval_ms / 1000), 2) if eval_ms > 0 else 0.0, @@ -79,8 +166,11 @@ async def _decode_benchmark(model: str) -> Dict[str, Any]: "prompt_eval_ms": round(data.get("prompt_eval_duration", 0) / 1e6, 2), "wall_ms": wall_ms, "response_chars": len(text), - # A model producing almost nothing, or pure repetition, is a corruption signal. - "degenerate": eval_count < BENCH_TOKENS * 0.5 or len(set(text.split())) < 8, + # Corruption signals: the model stopped far short of the token budget, or it + # produced text that is pure repetition. Token count is the primary signal -- + # empty text alone is not enough, since output can arrive in other fields. + "degenerate": (eval_count < BENCH_TOKENS * 0.5 + or (len(text) > 0 and len(set(text.split())) < 8)), } @@ -123,13 +213,36 @@ async def sweep(knob: str = "mem_offset_mhz", model = installed[0].get("name") defaults = KNOBS[knob] - start = defaults["default_start"] if start is None else start - stop = defaults["default_stop"] if stop is None else stop - step = defaults["default_step"] if step is None else step - if step <= 0 or stop < start: - return {"success": False, "error": "invalid sweep range"} + if defaults.get("kind") == "discrete": + supported = defaults.get("values") or _supported_clocks( + "mem" if knob == "lock_mem_mhz" else "gr") + if not supported: + return {"success": False, "error": f"card reported no supported clocks for {knob}"} + values = [v for v in supported + if (start is None or v >= start) and (stop is None or v <= stop)] + if not values: + return {"success": False, "error": f"no supported values in range; card offers {supported}"} + start, stop, step = values[0], values[-1], None + else: + start = defaults["default_start"] if start is None else start + stop = defaults["default_stop"] if stop is None else stop + step = defaults["default_step"] if step is None else step + if step <= 0 or stop < start: + return {"success": False, "error": "invalid sweep range"} + values = list(range(start, stop + 1, step)) baseline_cfg = overclock_manager.load_profiles().get(profile, {}) + baseline_value = int(baseline_cfg.get(knob, 0) or 0) + + # Refuse to sweep a knob the driver is going to ignore. + effectiveness = _knob_effective(knob, profile, values, baseline_value) + if not effectiveness["effective"]: + return { + "success": False, + "error": f"'{knob}' does not move this GPU: {effectiveness['detail']}", + "effectiveness": effectiveness, + } + state.running = True state.cancel = False results: List[Dict[str, Any]] = [] @@ -139,8 +252,9 @@ async def sweep(knob: str = "mem_offset_mhz", # Load the model once up front so the first step does not pay the load cost. await _decode_benchmark(model) - value = start - while value <= stop and not state.cancel: + for value in values: + if state.cancel: + break overclock_manager.apply_profile(profile, overrides={knob: value}) await asyncio.sleep(SETTLE_S) step_started = time.time() @@ -197,14 +311,24 @@ async def sweep(knob: str = "mem_offset_mhz", if not row["stable"]: logger.warning(f"autotune stopping climb at {knob}={value}: {row['instability']}") break - value += step stable = [r for r in results if r["stable"] and r["tokens_per_sec"] > 0] best = max(stable, key=lambda r: r["tokens_per_sec"]) if stable else None - baseline = next((r for r in results if r["value"] == start), None) - gain_pct = None - if best and baseline and baseline["tokens_per_sec"] > 0: - gain_pct = round((best["tokens_per_sec"] / baseline["tokens_per_sec"] - 1) * 100, 2) + + # Two different comparisons, and conflating them is how a sweep oversells itself. + # The first step of the range is not "baseline" unless it happens to be what the + # profile is already set to -- reporting "+102%" against the slowest value tried + # implies a speedup the user would not actually observe. + first_step = next((r for r in results if r["value"] == values[0]), None) + current = next((r for r in results if r["value"] == baseline_value), None) + gain_vs_first_step_pct = None + if best and first_step and first_step["tokens_per_sec"] > 0: + gain_vs_first_step_pct = round( + (best["tokens_per_sec"] / first_step["tokens_per_sec"] - 1) * 100, 2) + gain_vs_current_pct = None + if best and current and current["tokens_per_sec"] > 0: + gain_vs_current_pct = round( + (best["tokens_per_sec"] / current["tokens_per_sec"] - 1) * 100, 2) applied = None if apply_best and best: @@ -217,14 +341,22 @@ async def sweep(knob: str = "mem_offset_mhz", "knob": knob, "profile": profile, "model": model, - "range": {"start": start, "stop": stop, "step": step}, + "range": {"start": start, "stop": stop, "step": step, "values": values}, + "effectiveness": effectiveness, "steps_run": len(results), "duration_s": round(time.time() - t_start, 1), "cancelled": state.cancel, "best": {k: best[k] for k in ("value", "tokens_per_sec", "temp_c", "clock_mem_mhz", "clock_sm_mhz")} if best else None, - "baseline_tokens_per_sec": baseline["tokens_per_sec"] if baseline else None, - "gain_pct": gain_pct, + "current_profile_value": baseline_value, + "first_step_tokens_per_sec": first_step["tokens_per_sec"] if first_step else None, + "current_tokens_per_sec": current["tokens_per_sec"] if current else None, + "gain_vs_first_step_pct": gain_vs_first_step_pct, + "gain_vs_current_pct": gain_vs_current_pct, + "gain_note": ("compared against the profile's current setting" + if current is not None else + f"the profile's current value ({baseline_value}) was not in the " + f"swept range, so only the spread across tried values is shown"), "applied_to_profile": applied, "first_unstable": next(({"value": r["value"], "why": r["instability"]} for r in results if not r["stable"]), None), diff --git a/static/app.js b/static/app.js index b388f61..98a7604 100644 --- a/static/app.js +++ b/static/app.js @@ -855,7 +855,11 @@ function renderSweep(d) { const best = d.best ? `
โ˜… Best stable: ${d.knob}=${d.best.value} โ†’ ${d.best.tokens_per_sec} tok/s at ${d.best.temp_c}ยฐC - ${d.gain_pct != null ? `(${d.gain_pct > 0 ? '+' : ''}${d.gain_pct}% vs baseline)` : ''} + ${d.gain_vs_current_pct != null + ? `(${d.gain_vs_current_pct > 0 ? '+' : ''}${d.gain_vs_current_pct}% vs your current setting)` + : d.gain_vs_first_step_pct != null + ? `(spread across tried values: ${d.gain_vs_first_step_pct > 0 ? '+' : ''}${d.gain_vs_first_step_pct}% vs slowest)` + : ''} ${d.applied_to_profile ? 'ยท saved to profile' : ''}
` : '
No stable step produced throughput.
'; const unstable = d.first_unstable From 27dcfd0ada221d3fe14901cd2f77af9e6bf98962 Mon Sep 17 00:00:00 2001 From: drjones Date: Fri, 28 Aug 2026 09:14:58 -0700 Subject: [PATCH 3/6] Unload every resident Ollama model on yield; widen the confirm window The confirm barrier surfaced two yields that left 8.2 GB allocated after 3s. Two separate issues behind that class of failure: - The yield only unloaded loaded_models[0]. Ollama can hold several models resident (OLLAMA_MAX_LOADED_MODELS), so releasing the first left the rest allocated. It now unloads every resident model concurrently. This box runs with the limit at 1, so the change is defensive here rather than a fix for the observed case. - The observed 8.2 GB stalls happened while ComfyUI was starting and Ollama had a generation in flight; Ollama will not unload mid-request. A 3s ceiling reported a timeout for a model that was simply busy finishing. Raised to 10s -- waiting longer is the safer failure mode, since the alternative is diffusion allocating into VRAM that is still occupied. Co-Authored-By: Claude Opus 5 --- vram_arbitrator.py | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/vram_arbitrator.py b/vram_arbitrator.py index 6473fcb..071fd59 100644 --- a/vram_arbitrator.py +++ b/vram_arbitrator.py @@ -35,7 +35,12 @@ RAM_HIT_GBPS = 5.0 PARTIAL_HIT_GBPS = 1.5 # How long Ollama's VRAM may take to actually drain before we stop waiting. -YIELD_CONFIRM_TIMEOUT_S = 3.0 +# Ollama will not unload a model while a generation is in flight, so a short ceiling +# reports a timeout for what is really just a busy model finishing its request. Observed +# here: two yields hit the old 3 s limit with 8.2 GB still held while ComfyUI was starting. +# Waiting longer is the safer failure mode -- the alternative is diffusion allocating into +# VRAM that is still occupied. +YIELD_CONFIRM_TIMEOUT_S = 10.0 YIELD_CONFIRM_POLL_S = 0.02 YIELD_RESIDUAL_BYTES = 256 * 1024 ** 2 # treat <256 MB as "released" @@ -445,18 +450,29 @@ async def instant_free_ollama_vram(model_name: Optional[str] = None, the HTTP POST took. """ t0 = time.perf_counter() - if not model_name: + if model_name: + targets = [model_name] + else: + # Unload *every* resident model, not just loaded_models[0]. Ollama will happily + # keep several models in VRAM at once; releasing only the first left the rest + # allocated, which the confirm barrier caught as "still holding 8.2 GB after 3s". ollama_state = await get_ollama_live_state() - model_name = ollama_state.get("active_model_name") + targets = [m.get("name") for m in ollama_state.get("loaded_models", []) if m.get("name")] + if not targets and ollama_state.get("active_model_name"): + targets = [ollama_state["active_model_name"]] - if not model_name: + if not targets: return {"success": True, "message": "No active Ollama model in VRAM", "duration_ms": 0, "confirmed": True} + model_name = targets[0] if len(targets) == 1 else f"{len(targets)} models" baseline = get_process_vram_bytes()["ollama_bytes"] try: client = _client(OLLAMA_API_BASE, 5.0) - await client.post("/api/generate", json={"model": model_name, "keep_alive": 0}) + await asyncio.gather(*[ + client.post("/api/generate", json={"model": t, "keep_alive": 0}) + for t in targets + ], return_exceptions=True) request_ms = round((time.perf_counter() - t0) * 1000, 2) barrier = {"confirmed": None, "confirm_ms": 0.0, "residual_bytes": baseline} @@ -468,7 +484,7 @@ async def instant_free_ollama_vram(model_name: Optional[str] = None, _record({ "event_type": "Ollama VRAM Yield", - "source": model_name, + "source": ", ".join(targets)[:200], "target": "VRAM 0MB (Kept in RAM)", "duration_ms": duration_ms, "yield_confirm_ms": barrier.get("confirm_ms"), @@ -478,6 +494,7 @@ async def instant_free_ollama_vram(model_name: Optional[str] = None, return { "success": True, "model": model_name, + "models_unloaded": targets, "duration_ms": duration_ms, "request_ms": request_ms, "confirm_ms": barrier.get("confirm_ms"), From a30444ef8e698f4031ad58f61e6cbe92086cf417 Mon Sep 17 00:00:00 2001 From: drjones Date: Fri, 28 Aug 2026 09:16:36 -0700 Subject: [PATCH 4/6] Stop open SSE streams from blocking graceful shutdown Every connected dashboard holds a StreamingResponse open indefinitely, so uvicorn's graceful shutdown waited on them, systemd hit its 90s stop timeout and SIGKILLed the unit. That skipped the in-process restore hook entirely, leaving the GPU restore to ExecStopPost alone. - broker.stop() sets a closing flag and pushes a sentinel to every subscriber queue so the SSE generators return instead of parking on q.get(). - uvicorn gets timeout_graceful_shutdown=10 and the unit TimeoutStopSec=20, bounding the worst case rather than relying on the 90s default. Restart now completes in ~11s with 'Restoring GPU to safe stock state (server shutdown)' running in-process, and no SIGKILL. Co-Authored-By: Claude Opus 5 --- server.py | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/server.py b/server.py index c172792..bf6747b 100644 --- a/server.py +++ b/server.py @@ -55,6 +55,8 @@ class TelemetryBroker: self.running = False self.samples = 0 self.last_sample_ms = 0.0 + # Set on shutdown so open SSE generators finish instead of holding the server up. + self.closing = False async def start(self) -> None: if self.running: @@ -64,6 +66,13 @@ class TelemetryBroker: async def stop(self) -> None: self.running = False + self.closing = True + # Wake every subscriber so their generator can return. Without this, uvicorn waits + # on the open SSE responses during graceful shutdown and systemd eventually + # SIGKILLs the unit -- which skips the in-process GPU restore hook entirely. + for q in list(self.subscribers): + with contextlib.suppress(asyncio.QueueFull): + q.put_nowait(None) if self.task: self.task.cancel() with contextlib.suppress(asyncio.CancelledError): @@ -250,11 +259,13 @@ async def sse_telemetry_stream(request: Request): try: snap = await broker.get() yield f"data: {json.dumps(snap)}\n\n" - while True: + while not broker.closing: if await request.is_disconnected(): break try: snap = await asyncio.wait_for(q.get(), timeout=15.0) + if snap is None: # shutdown sentinel + break yield f"data: {json.dumps(snap)}\n\n" except asyncio.TimeoutError: yield ": keepalive\n\n" @@ -497,4 +508,5 @@ async def root_index(): if __name__ == "__main__": import uvicorn - uvicorn.run("server:app", host="0.0.0.0", port=9090, reload=False, log_level="info") + uvicorn.run("server:app", host="0.0.0.0", port=9090, reload=False, log_level="info", + timeout_graceful_shutdown=10) From c689ec871171600ab75e23fbbb131ec740990651 Mon Sep 17 00:00:00 2001 From: drjones Date: Fri, 28 Aug 2026 11:18:35 -0700 Subject: [PATCH 5/6] Tune profiles from measurement; add a diffusion benchmark to close the loop The profiles were hand-written and had never been checked against the hardware. Adding a ComfyUI benchmark alongside the existing decode one made the compute side measurable for the first time, and most of what the profiles configured turned out to do nothing. Measured on this card (RTX 4080 SUPER, driver 595.84): - LLM decode is not power-bound: 73.0-73.5 tok/s flat from 222W to 370W, with the card never drawing more than 224W at any limit. The ollama profile's 370W did nothing. - Diffusion is power-bound: 5.48 it/s @222W rising to 6.71 @370W, so comfy's 370W is worth a real +2.8% over the 320W stock default. - Clock locks did nothing for either workload: 72.6 tok/s locked at 11251MHz vs 72.7 unlocked; 6.77 it/s locked at 3105MHz vs 6.73 unlocked, and 6.78 at 2400MHz. - Memory bandwidth is still the decode bottleneck (5001MHz halves throughput to 35.9 tok/s), confirming the profile's premise -- the card just gets there unaided. - Fans: 48,435 samples show 81C all-time max and zero thermal throttle events, while the ollama profile held 49.6C average by running fans at 87%. All profiles now use automatic fans and let the thermal governor escalate on demand. Code changes supporting that: - _diffusion_benchmark() queues a fixed SDXL graph via ComfyUI's API. The seed must vary per run: ComfyUI caches by node inputs, so a fixed seed returned in ~1ms without executing. Implausibly fast results are now rejected as cache hits rather than recorded as record scores. - The arbitrator's automatic profile switching is suspended during a sweep. A diffusion benchmark trips trigger_comfy_priority, which reapplies the whole profile and would silently overwrite the clock being measured. - _supported_clocks() queries the mem,gr pair; asking for a single field returned one column and reading index 1 yielded an empty list rather than an error. Graphics clocks are subsampled (the card enumerates 194 of them) and lock sweeps include an explicit unlocked control step. - offsets_supported() probes once and apply_profile skips inert offset levers with an explanation instead of pretending they applied. - Profiles carry a 'measured' field recording the evidence behind each setting. Co-Authored-By: Claude Opus 5 --- README.md | 69 ++++++++++--- autotune.py | 209 ++++++++++++++++++++++++++++++++++++---- overclock_manager.py | 49 +++++++++- overclock_profiles.json | 41 ++++---- server.py | 45 ++++++++- vram_arbitrator.py | 16 ++- 6 files changed, 373 insertions(+), 56 deletions(-) diff --git a/README.md b/README.md index 67df288..1b23626 100644 --- a/README.md +++ b/README.md @@ -33,17 +33,55 @@ * **Budgeted, Ranked Warming**: This box has 64 GB of RAM and >270 GB of model files; reading everything simply evicts whatever was warmed first. Files are ranked by recency/frequency (from the persisted event log) and warmed until a byte budget is spent, skipping anything already resident. `GET /api/warm-plan` previews the decision without executing it. * **Memory Telemetry**: Real-time breakdown of Total Host RAM, Applications Memory, Active Model Page Cache, Free Memory, and measured Cache Residency Ratio. -### ๐ŸŽ›๏ธ Dynamic Overclocking & Thermal Management -* **Workload-Aware Overclock Profiles**: - * **`ollama` Profile (Memory-Bandwidth Bound)**: Max 370W power limit, +150 MHz Core Offset, +825 MHz Memory Offset, and 100% fan speed for maximum prompt eval / generation bandwidth. - * **`comfy` Profile (Compute Bound)**: Max 370W power limit, +100 MHz Core Offset, +500 MHz Memory Offset, Core Clock locked to 2900โ€“3105 MHz, and 75% fan speed for maximum diffusion compute. - * **`balanced` Profile (Stock/General Purpose)**: Unlocked 370W power limit with stock dynamic boost curves and automatic fan control. -* **Hardware Actuation Hierarchy**: - * Level 1: Power Limit Control (`nvidia-smi -pl 370`). - * Level 2: Core & Memory Clock Locking (`nvidia-smi -lgc` / `-lmc`). - * Level 3: Clock Offsets via headless X display (`:8`) with Coolbits support (`nvidia-settings`). -* **Hardware Fan Control**: Switch between `auto` and `manual` PWM control (30%โ€“100%) with synchronized dual-fan actuation (`[fan:0]` and `[fan:1]`). -* **Automated Lockstep Profile Switching**: AutoArbitrator automatically switches hardware profiles in lockstep with the active workload (`comfy` on generation start, `ollama` on completion). +### ๐ŸŽ›๏ธ Measured Overclock Profiles & Thermal Management + +Every profile setting in this repo is now backed by a measurement from `autotune.py` on +this specific card and driver. Several long-standing settings turned out to do nothing. + +**What this driver actually honours** (NVIDIA 595.84, RTX 4080 SUPER): + +| Lever | Mechanism | Works? | +| :--- | :--- | :--- | +| Power limit | `nvidia-smi -pl` | โœ… Yes โ€” and it is the only lever that changes anything measurable | +| Core / memory clock lock | `nvidia-smi -lgc` / `-lmc` | โœ… Applies correctly, but made no measurable difference to either workload | +| Core / memory clock offsets | `nvidia-settings -a ...Offset` | โŒ **Silently ignored.** The driver reports `assigned value 0` and the attribute still reads back `250`. Detected automatically by `offsets_supported()`; `apply_profile` now skips them and says so rather than pretending. | +| Fan control | `nvidia-settings GPUTargetFanSpeed` | โœ… Yes | + +**Measured results** (`POST /api/autotune/sweep`): + +*LLM decode is not power-bound.* Throughput is flat across the card's entire power range โ€” +the GPU never drew more than 224 W no matter what the limit allowed: + +| Power limit | 222 W | 259 W | 296 W | 320 W | 333 W | 370 W | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| tok/s (`qwen3.8long`) | 73.17 | 73.13 | 73.43 | 73.51 | 73.10 | 73.04 | + +*Diffusion is power-bound.* Here the watts genuinely buy throughput: + +| Power limit | 222 W | 259 W | 296 W | 320 W | 333 W | 370 W | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| it/s (SDXL 1024, 20 steps) | 5.48 | 6.22 | 6.50 | 6.52 | 6.63 | **6.71** | + +*Clock locks changed nothing for either workload.* Memory clock: 72.6 tok/s locked at +11251 MHz vs 72.7 unlocked. Core clock: 6.73 it/s unlocked vs 6.77 locked at 3105 MHz โ€” +and 6.78 at 2400 MHz, so diffusion here is not core-clock-bound at all. + +*Memory bandwidth is the decode bottleneck*, confirming the profile's original premise โ€” +dropping the memory clock to 5001 MHz halves throughput (35.9 tok/s vs 72.6). The card +simply reaches its top memory clock on its own; pinning it there adds nothing. + +**Resulting profiles**: +* **`ollama`** โ€” 320 W (stock), no locks, automatic fans. Decode draws ~224 W and is + bandwidth-bound, so the previous 370 W limit and 100% fan pinning bought nothing. +* **`comfy`** โ€” 370 W, no locks, automatic fans. The extra power is worth a measured + **+2.8%** over the 320 W stock default. +* **`balanced`** โ€” stock power and boost, automatic fans. + +**On fans**: all three profiles previously pinned the fans to manual 100%. Across 48,435 +telemetry samples this card has never exceeded **81 ยฐC** and has logged **zero** thermal +throttle events; the `ollama` profile was holding 49.6 ยฐC average by running the fans at +87%. Fans are now automatic in every profile, with the thermal governor escalating them +only if the card actually needs it. ### ๐ŸŒก๏ธ Thermal Governor (closed-loop de-escalation) * Every overclock lever here is sticky: a profile locks clocks and pins the fans to a manual PWM, and nothing used to undo that. The governor watches the telemetry the sampler already collects (so it costs no extra NVML calls) and walks the overclock back through a four-step derate ladder when the card runs hot or reports a hardware throttle. @@ -51,9 +89,12 @@ * **Guaranteed restore**: stock clocks, default power limit and automatic fans are restored by the server's shutdown hook *and* by a systemd `ExecStopPost=`, so a `SIGKILL` cannot leave the card with locked clocks and fans pinned at 100%. ### ๐Ÿ”ฌ Overclock Autotune (`autotune.py`) -* Walks a clock offset upward, running a fixed decode benchmark at each step, and reports the **fastest stable** value with its measured gain over baseline. +* Sweeps a knob (`power_limit_w`, `lock_mem_mhz`, `lock_core_max`, clock offsets) and reports the **fastest stable** value. +* **Both workloads are measurable.** `workload=ollama` benchmarks decode throughput in tok/s; `workload=comfy` queues a fixed SDXL 1024/20-step graph through ComfyUI's API and measures it/s. Without the second one there was no way to tell whether the compute-oriented `comfy` profile was doing anything at all โ€” and it was not. +* **Refuses to sweep a knob the driver ignores.** A preflight applies a probe value and confirms the hardware moved; the probe is chosen as the candidate furthest from the current reading, since probing with the maximum proves nothing when the card already sits there. This is what caught the silently-discarded clock offsets. * **Instability detection**: kernel `Xid`/`NVRM` messages via `journalctl -k`, benchmark failure, degenerate output, and a temperature ceiling. The sweep stops climbing the moment a step looks unstable. -* **Safety**: refuses to start while ComfyUI is executing, and restores the original profile in a `finally` block โ€” including on exception or cancellation. +* **Honest gain reporting**: gain against the profile's *current* setting is reported separately from the spread across values tried. Conflating them turns a flat result into a headline "+102%". +* **Safety**: refuses to start while ComfyUI is executing, suspends the arbitrator's automatic profile switching for the duration (otherwise a diffusion benchmark trips `trigger_comfy_priority`, which reapplies the whole profile and overwrites the clock being measured), and restores the original profile in a `finally` block โ€” including on exception or cancellation. ### ๐Ÿ—„๏ธ Persistent Telemetry Store (`telemetry_store.py`) * Swap history used to be an in-memory `deque(maxlen=50)` that evaporated on every restart. Telemetry and events now persist to SQLite (WAL, single writer thread, batched 1 Hz inserts, automatic retention pruning) at roughly **0.4 MB per hour**. @@ -161,7 +202,7 @@ The HyperSwap server runs on port `9090` by default. Interactive OpenAPI/Swagger | `/api/governor` | `GET` / `POST` | Current derate level and why; enable/disable, or clear an active derate. | | `/api/overclock/restore` | `POST` | Drop all clock locks and offsets, restore default power limit and automatic fans. | | `/api/autotune` | `GET` | Sweep progress, last result, and every recorded autotune step. | -| `/api/autotune/sweep` | `POST` | Walk a clock offset upward, measuring tok/s and watching for instability at each step. | +| `/api/autotune/sweep` | `POST` | Sweep a knob against a real workload (`workload`: `ollama` decode tok/s, `comfy` SDXL it/s), verifying the knob moves the hardware first. | | `/api/autotune/cancel` | `POST` | Stop the current sweep after the step in flight; the profile is restored either way. | --- diff --git a/autotune.py b/autotune.py index 8dd6cf5..15e68fb 100644 --- a/autotune.py +++ b/autotune.py @@ -15,6 +15,7 @@ Safety properties: """ import asyncio import logging +import random import subprocess import time from typing import Any, Dict, List, Optional @@ -46,28 +47,74 @@ KNOBS = { "values": None}, # filled from the card's supported clock list "lock_core_max": {"kind": "discrete", "verify": "clock_sm", "values": None}, + # The one lever this driver definitely honours. Worth knowing whether the extra + # watts actually buy throughput, or just heat and fan noise. + "power_limit_w": {"kind": "discrete", "verify": "power_limit", + "values": None, "no_unlocked": True}, } def _supported_clocks(which: str = "mem") -> List[int]: - """Discrete clock values the card will actually accept for -lmc / -lgc.""" + """Discrete clock values the card will actually accept for -lmc / -lgc. + + Always queried as the mem,gr pair: asking for a single field returns one column, and + reading index 1 from it silently yields an empty list rather than an error. + """ try: proc = subprocess.run( - ["nvidia-smi", f"--query-supported-clocks={'mem' if which == 'mem' else 'gr'}", - "--format=csv,noheader,nounits"], - capture_output=True, text=True, timeout=10) - col = 0 if which == "mem" else 1 - vals = set() + ["nvidia-smi", "--query-supported-clocks=mem,gr", "--format=csv,noheader,nounits"], + capture_output=True, text=True, timeout=15) + rows = [] for line in proc.stdout.splitlines(): parts = [p.strip() for p in line.split(",")] - if len(parts) > col and parts[col].isdigit(): - vals.add(int(parts[col])) - return sorted(vals) + if len(parts) >= 2 and parts[0].isdigit() and parts[1].isdigit(): + rows.append((int(parts[0]), int(parts[1]))) + if not rows: + return [] + if which == "mem": + return sorted({m for m, _ in rows}) + # Graphics clocks are enumerated per memory clock; take the list for the highest + # memory clock, which is the one any real workload runs at. + top_mem = max(m for m, _ in rows) + return sorted({g for m, g in rows if m == top_mem}) except Exception as e: logger.debug(f"supported clock query failed: {e}") return [] +def _supported_power_limits(steps: int = 5) -> List[int]: + """Power limits between the card's minimum and maximum, in even increments.""" + try: + proc = subprocess.run( + ["nvidia-smi", "--query-gpu=power.min_limit,power.max_limit,power.default_limit", + "--format=csv,noheader,nounits"], + capture_output=True, text=True, timeout=10) + parts = [p.strip() for p in proc.stdout.strip().split(",")] + lo, hi, default = (int(float(parts[0])), int(float(parts[1])), int(float(parts[2]))) + except Exception as e: + logger.debug(f"power limit query failed: {e}") + return [] + # Start at 60% of max -- below that the card is not doing useful work for these + # workloads -- and always include the stock default as a reference point. + lo = max(lo, int(hi * 0.6)) + span = hi - lo + vals = {lo + round(i * span / (steps - 1)) for i in range(steps)} + vals.add(default) + return sorted(v for v in vals if lo <= v <= hi) + + +def _subsample(values: List[int], max_steps: int) -> List[int]: + """Evenly spaced subset, always keeping the endpoints. + + The card enumerates ~194 graphics clocks in 15 MHz increments; benchmarking every one + would take hours and tell us nothing that a handful of well-spread points does not. + """ + if len(values) <= max_steps: + return values + idx = [round(i * (len(values) - 1) / (max_steps - 1)) for i in range(max_steps)] + return sorted({values[i] for i in idx}) + + def _read_hw(field: str) -> Optional[float]: """Read back the hardware state a knob is supposed to move.""" gpu = vram_arbitrator.get_gpu_hardware_stats() @@ -75,6 +122,8 @@ def _read_hw(field: str) -> Optional[float]: return gpu.get("clock_mem_mhz") if field == "clock_sm": return gpu.get("clock_graphics_mhz") + if field == "power_limit": + return gpu.get("power_limit_w") if field in ("mem_offset", "core_offset"): r = overclock_manager._nvidia_settings( "-q", f"[gpu:0]/{'GPUMemoryTransferRateOffset' if field == 'mem_offset' else 'GPUGraphicsClockOffset'}[3]") @@ -174,6 +223,97 @@ async def _decode_benchmark(model: str) -> Dict[str, Any]: } +# A fixed SDXL txt2img graph. Deterministic seed/steps/resolution so every step of a +# sweep does identical work and the only variable is the clock. PreviewImage rather than +# SaveImage keeps benchmark runs out of the user's output gallery. +COMFY_BENCH_CKPT = "sd_xl_base_1.0.safetensors" +COMFY_BENCH_STEPS = 20 +COMFY_BENCH_SIZE = 1024 + + +def _comfy_workflow(ckpt: str = COMFY_BENCH_CKPT, seed: Optional[int] = None) -> Dict[str, Any]: + # The seed must vary per run. ComfyUI caches by node inputs, so a fixed seed makes the + # second and later benchmarks return in ~1ms without executing anything at all. The + # cost of the graph is identical regardless of seed, so this costs no comparability. + seed = random.randint(1, 2**31) if seed is None else seed + return { + "1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": ckpt}}, + "2": {"class_type": "CLIPTextEncode", + "inputs": {"clip": ["1", 1], + "text": "a detailed photograph of a mountain range at sunrise"}}, + "3": {"class_type": "CLIPTextEncode", + "inputs": {"clip": ["1", 1], "text": "blurry, low quality"}}, + "4": {"class_type": "EmptyLatentImage", + "inputs": {"width": COMFY_BENCH_SIZE, "height": COMFY_BENCH_SIZE, "batch_size": 1}}, + "5": {"class_type": "KSampler", + "inputs": {"model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0], + "latent_image": ["4", 0], "seed": seed, "steps": COMFY_BENCH_STEPS, + "cfg": 7.0, "sampler_name": "euler", "scheduler": "normal", + "denoise": 1.0}}, + "6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}}, + "7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}, + } + + +async def _diffusion_benchmark(ckpt: str = COMFY_BENCH_CKPT, + timeout_s: float = 300.0) -> Dict[str, Any]: + """Queue one fixed SDXL graph and time it. This is the compute-bound counterpart to + the decode benchmark, and the only way to tell whether the 'comfy' profile helps.""" + client = vram_arbitrator._client(vram_arbitrator.COMFY_API_BASE, 30.0) + t0 = time.perf_counter() + try: + resp = await client.post("/prompt", json={"prompt": _comfy_workflow(ckpt), + "client_id": "hyperswap-autotune"}) + if resp.status_code != 200: + return {"ok": False, "error": f"queue failed HTTP {resp.status_code}: {resp.text[:200]}"} + prompt_id = resp.json().get("prompt_id") + except Exception as e: + return {"ok": False, "error": f"queue failed: {e}"} + + while (time.perf_counter() - t0) < timeout_s: + await asyncio.sleep(0.25) + try: + h = await client.get(f"/history/{prompt_id}") + if h.status_code != 200: + continue + entry = (h.json() or {}).get(prompt_id) + if not entry: + continue + status = entry.get("status", {}) + if status.get("status_str") == "error" or not status.get("completed", True): + if status.get("status_str") == "error": + return {"ok": False, "error": "ComfyUI reported an execution error", + "wall_ms": round((time.perf_counter() - t0) * 1000, 2)} + if status.get("completed"): + wall = time.perf_counter() - t0 + # ComfyUI stamps execution_start/success in the status messages; the delta + # between them excludes our polling overhead and the queue wait. + stamps = {} + for msg in status.get("messages", []): + if isinstance(msg, list) and len(msg) >= 2 and isinstance(msg[1], dict): + if "timestamp" in msg[1]: + stamps[msg[0]] = msg[1]["timestamp"] + exec_ms = None + if "execution_start" in stamps and "execution_success" in stamps: + exec_ms = round(stamps["execution_success"] - stamps["execution_start"], 2) + effective_ms = exec_ms or wall * 1000 + # A graph that "finished" implausibly fast was served from ComfyUI's cache + # rather than executed; treat it as an invalid sample, not a record score. + cached = effective_ms < 250 + return { + "ok": not cached, + "error": "result served from ComfyUI cache, not executed" if cached else None, + "wall_ms": round(wall * 1000, 2), + "exec_ms": exec_ms, + "steps": COMFY_BENCH_STEPS, + "it_per_sec": round(COMFY_BENCH_STEPS / (effective_ms / 1000), 3), + "degenerate": cached, + } + except Exception: + continue + return {"ok": False, "error": f"diffusion benchmark timed out after {timeout_s}s"} + + class SweepState: def __init__(self) -> None: self.running = False @@ -187,11 +327,14 @@ state = SweepState() async def sweep(knob: str = "mem_offset_mhz", profile: str = "ollama", + workload: str = "auto", model: Optional[str] = None, start: Optional[int] = None, stop: Optional[int] = None, step: Optional[int] = None, repeats: int = 1, + max_steps: int = 6, + include_unlocked: bool = True, apply_best: bool = False) -> Dict[str, Any]: """Sweep one clock offset and return the fastest stable value.""" if knob not in KNOBS: @@ -203,7 +346,23 @@ async def sweep(knob: str = "mem_offset_mhz", if comfy.get("executing") or comfy.get("queue_remaining"): return {"success": False, "error": "ComfyUI is busy; refusing to change clocks mid-render"} - if not model: + # 'auto': tune the workload the profile is actually for. + if workload == "auto": + workload = "comfy" if profile == "comfy" else "ollama" + if workload not in ("ollama", "comfy"): + return {"success": False, "error": "workload must be 'ollama', 'comfy' or 'auto'"} + if workload == "comfy" and not comfy.get("online"): + return {"success": False, "error": "ComfyUI is not reachable; cannot run a diffusion sweep"} + + if workload == "comfy": + model = model or COMFY_BENCH_CKPT + benchmark = lambda: _diffusion_benchmark(model) + metric = "it_per_sec" + else: + benchmark = lambda: _decode_benchmark(model) + metric = "tokens_per_sec" + + if workload == "ollama" and not model: ollama = await vram_arbitrator.get_ollama_live_state() model = ollama.get("active_model_name") if not model: @@ -211,17 +370,26 @@ async def sweep(knob: str = "mem_offset_mhz", if not installed: return {"success": False, "error": "no Ollama model available to benchmark"} model = installed[0].get("name") + benchmark = lambda: _decode_benchmark(model) defaults = KNOBS[knob] if defaults.get("kind") == "discrete": - supported = defaults.get("values") or _supported_clocks( - "mem" if knob == "lock_mem_mhz" else "gr") + if knob == "power_limit_w": + supported = _supported_power_limits() + else: + supported = defaults.get("values") or _supported_clocks( + "mem" if knob == "lock_mem_mhz" else "gr") if not supported: return {"success": False, "error": f"card reported no supported clocks for {knob}"} values = [v for v in supported if (start is None or v >= start) and (stop is None or v <= stop)] if not values: return {"success": False, "error": f"no supported values in range; card offers {supported}"} + values = _subsample(values, max_steps or 6) + # 0 means "no lock at all". That is the honest control for a profile whose whole + # premise is that locking the clock beats letting the card boost on its own. + if include_unlocked and not defaults.get("no_unlocked"): + values = [0] + values start, stop, step = values[0], values[-1], None else: start = defaults["default_start"] if start is None else start @@ -235,7 +403,8 @@ async def sweep(knob: str = "mem_offset_mhz", baseline_value = int(baseline_cfg.get(knob, 0) or 0) # Refuse to sweep a knob the driver is going to ignore. - effectiveness = _knob_effective(knob, profile, values, baseline_value) + effectiveness = _knob_effective(knob, profile, [v for v in values if v] or values, + baseline_value) if not effectiveness["effective"]: return { "success": False, @@ -249,8 +418,9 @@ async def sweep(knob: str = "mem_offset_mhz", t_start = time.time() try: - # Load the model once up front so the first step does not pay the load cost. - await _decode_benchmark(model) + # Warm-up: load weights once up front so the first step does not pay the load cost. + vram_arbitrator.arbitrator.suspend_oc("autotune sweep") + await benchmark() for value in values: if state.cancel: @@ -261,7 +431,7 @@ async def sweep(knob: str = "mem_offset_mhz", samples = [] for _ in range(max(repeats, 1)): - samples.append(await _decode_benchmark(model)) + samples.append(await benchmark()) if state.cancel: break @@ -278,11 +448,13 @@ async def sweep(knob: str = "mem_offset_mhz", if temp >= TEMP_CEILING_C: instability.append(f"temperature ceiling hit ({temp}ยฐC)") - tok_s = round(max((s["tokens_per_sec"] for s in ok_samples), default=0.0), 2) + tok_s = round(max((s.get(metric, 0.0) for s in ok_samples), default=0.0), 2) row = { "knob": knob, "value": value, "profile": profile, + "workload": workload, + "metric": metric, "model": model, "tokens_per_sec": tok_s, "temp_c": temp, @@ -340,6 +512,8 @@ async def sweep(knob: str = "mem_offset_mhz", "success": True, "knob": knob, "profile": profile, + "workload": workload, + "metric": metric, "model": model, "range": {"start": start, "stop": stop, "step": step, "values": values}, "effectiveness": effectiveness, @@ -370,6 +544,7 @@ async def sweep(knob: str = "mem_offset_mhz", # Always hand the card back exactly as we found it. state.running = False state.current = None + vram_arbitrator.arbitrator.resume_oc(profile) try: overclock_manager.apply_profile(profile) logger.info(f"autotune restored profile '{profile}'") diff --git a/overclock_manager.py b/overclock_manager.py index 1fb7368..14f6a41 100644 --- a/overclock_manager.py +++ b/overclock_manager.py @@ -67,6 +67,46 @@ DEFAULT_PROFILES: Dict[str, Dict[str, Any]] = { }, } +_OFFSETS_SUPPORTED: Optional[bool] = None + + +def offsets_supported(recheck: bool = False) -> bool: + """Whether nvidia-settings clock offsets actually take effect on this driver. + + Driver 595.84 accepts GPUGraphicsClockOffset/GPUMemoryTransferRateOffset and silently + discards them: assigning 0 returns success and the attribute still reads back its old + value. Profiles carrying core_offset_mhz/mem_offset_mhz were therefore configuring + nothing. Probed once and cached. + """ + global _OFFSETS_SUPPORTED + if _OFFSETS_SUPPORTED is not None and not recheck: + return _OFFSETS_SUPPORTED + + def _read() -> Optional[int]: + q = _nvidia_settings("-q", "[gpu:0]/GPUGraphicsClockOffset[3]") + for line in (q.get("out") or "").splitlines(): + if "Attribute" in line and "):" in line: + try: + return int(line.split("):")[-1].split(".")[0].strip()) + except Exception: + pass + return None + + before = _read() + if before is None: + _OFFSETS_SUPPORTED = False + return False + probe = before + 25 + _nvidia_settings("-a", f"[gpu:0]/GPUGraphicsClockOffset[3]={probe}") + after = _read() + _nvidia_settings("-a", f"[gpu:0]/GPUGraphicsClockOffset[3]={before}") + _OFFSETS_SUPPORTED = (after is not None and after != before) + if not _OFFSETS_SUPPORTED: + logger.warning("Clock offsets are not honoured by this driver " + f"(set {probe}, read back {after}); profile offset fields are inert.") + return _OFFSETS_SUPPORTED + + ACTIVE_PROFILE = "balanced" _LAST_RESULT: Dict[str, Any] = {} FAN_MANUAL = False @@ -231,7 +271,11 @@ def apply_profile(name: str, overrides: Optional[Dict[str, Any]] = None) -> Dict "power_limit": _apply_power_limit(int(cfg.get("power_limit_w", 370))), "clock_lock": _apply_clock_lock(int(cfg.get("lock_core_min", 0)), int(cfg.get("lock_core_max", 0))), "mem_lock": _apply_mem_lock(int(cfg.get("lock_mem_mhz", 0))), - "offsets": _apply_offsets(int(cfg.get("core_offset_mhz", 0)), int(cfg.get("mem_offset_mhz", 0))), + "offsets": (_apply_offsets(int(cfg.get("core_offset_mhz", 0)), + int(cfg.get("mem_offset_mhz", 0))) + if offsets_supported() else + {"applied": False, "supported": False, + "detail": "skipped: this driver accepts clock offsets and ignores them"}), "fan": apply_fan_control(fan_mode, fan_speed), } result["gpu"] = get_gpu_state() @@ -382,6 +426,9 @@ def get_status() -> Dict[str, Any]: """Full overclock status for the dashboard.""" return { "active_profile": ACTIVE_PROFILE, + "offsets_supported": offsets_supported(), + "effective_levers": (["power_limit", "clock_lock", "mem_lock", "fan"] + + (["offsets"] if offsets_supported() else [])), "profiles": load_profiles(), "gpu": get_gpu_state(), "fan": get_fan_status(), diff --git a/overclock_profiles.json b/overclock_profiles.json index d871404..5f14754 100644 --- a/overclock_profiles.json +++ b/overclock_profiles.json @@ -1,28 +1,19 @@ { "ollama": { - "label": "Ollama \u2014 LLM decode (memory-bandwidth bound)", - "power_limit_w": 370, - "core_offset_mhz": 35, - "mem_offset_mhz": 200, + "label": "Ollama โ€” LLM decode (memory-bandwidth bound; measured insensitive to power and clocks)", + "measured": "73.0-73.5 tok/s flat from 222W to 370W (qwen3.8long, 2026-08-28). Actual draw never exceeded 224W at any limit. Memory clock lock made no difference (72.6 locked vs 72.7 unlocked).", + "power_limit_w": 320, + "core_offset_mhz": 0, + "mem_offset_mhz": 0, "lock_core_min": 0, "lock_core_max": 0, "lock_mem_mhz": 0, - "fan_mode": "manual", - "fan_speed_pct": 100 + "fan_mode": "auto", + "fan_speed_pct": 0 }, "comfy": { - "label": "ComfyUI \u2014 diffusion (core-compute bound)", - "power_limit_w": 370, - "core_offset_mhz": 100, - "mem_offset_mhz": 150, - "lock_core_min": 2900, - "lock_core_max": 3105, - "lock_mem_mhz": 0, - "fan_mode": "manual", - "fan_speed_pct": 100 - }, - "balanced": { - "label": "Balanced \u2014 stock boost, power unlocked", + "label": "ComfyUI โ€” diffusion (compute bound; genuinely power-scaling)", + "measured": "SDXL 1024/20-step: 5.48 it/s @222W, 6.22 @259W, 6.50 @296W, 6.52 @320W, 6.63 @333W, 6.71 @370W (2026-08-28). Worth +2.8% over the 320W stock default. Core clock lock made no difference across 2400-3105 MHz.", "power_limit_w": 370, "core_offset_mhz": 0, "mem_offset_mhz": 0, @@ -31,5 +22,17 @@ "lock_mem_mhz": 0, "fan_mode": "auto", "fan_speed_pct": 0 + }, + "balanced": { + "label": "Balanced โ€” stock power and boost, automatic fans", + "measured": "Card's own design point. 48k telemetry samples show 67.8C average under load at 39.5% auto fan, 81C all-time max, zero thermal throttle events.", + "power_limit_w": 320, + "core_offset_mhz": 0, + "mem_offset_mhz": 0, + "lock_core_min": 0, + "lock_core_max": 0, + "lock_mem_mhz": 0, + "fan_mode": "auto", + "fan_speed_pct": 0 } -} \ No newline at end of file +} diff --git a/server.py b/server.py index bf6747b..07eab1e 100644 --- a/server.py +++ b/server.py @@ -3,6 +3,7 @@ import asyncio import contextlib import json import logging +import signal import time from contextlib import asynccontextmanager from typing import Dict, Any, Optional, List, Set @@ -64,6 +65,13 @@ class TelemetryBroker: self.running = True self.task = asyncio.create_task(self._loop()) + def begin_shutdown(self) -> None: + """Release every SSE subscriber. Safe to call from a signal handler.""" + self.closing = True + for q in list(self.subscribers): + with contextlib.suppress(asyncio.QueueFull): + q.put_nowait(None) + async def stop(self) -> None: self.running = False self.closing = True @@ -150,11 +158,36 @@ class TelemetryBroker: broker = TelemetryBroker() +def _install_shutdown_hook() -> None: + """Close SSE streams the moment a shutdown signal arrives. + + uvicorn runs the lifespan shutdown only after it has finished waiting on open + connections, so releasing subscribers from there is too late: the streams keep the + server busy until the graceful timeout expires and every one of them is force + cancelled, which logs a CancelledError traceback apiece. Chaining onto the existing + signal handler lets us drain them first and leaves uvicorn's own shutdown intact. + """ + loop = asyncio.get_running_loop() + for sig in (signal.SIGTERM, signal.SIGINT): + previous = signal.getsignal(sig) + + def handler(signum, frame, _prev=previous): + broker.begin_shutdown() + if callable(_prev): + _prev(signum, frame) + + try: + signal.signal(sig, handler) + except (ValueError, OSError): + pass # not on the main thread; the lifespan path still cleans up + + @asynccontextmanager async def lifespan(app: FastAPI): telemetry_store.start() await broker.start() await vram_arbitrator.arbitrator.start() + _install_shutdown_hook() yield await vram_arbitrator.arbitrator.stop() await broker.stop() @@ -218,13 +251,16 @@ class GovernorRequest(BaseModel): reset: bool = Field(False, description="Clear any active derate and reapply the full profile") class SweepRequest(BaseModel): - knob: str = Field("mem_offset_mhz", description="mem_offset_mhz | core_offset_mhz") + knob: str = Field("mem_offset_mhz", description="mem_offset_mhz | core_offset_mhz | lock_mem_mhz | lock_core_max") profile: str = Field("ollama", description="Profile to tune") + workload: str = Field("auto", description="ollama (decode tok/s) | comfy (diffusion it/s) | auto") model: Optional[str] = Field(None, description="Model to benchmark with; defaults to the loaded one") start: Optional[int] = Field(None, description="First offset value") stop: Optional[int] = Field(None, description="Last offset value") step: Optional[int] = Field(None, description="Offset increment") repeats: int = Field(1, description="Benchmark runs per step") + max_steps: int = Field(6, description="Cap on swept values for discrete clock knobs") + include_unlocked: bool = Field(True, description="Include an unlocked (0) control step") apply_best: bool = Field(False, description="Write the winning value into the profile") class RequestVramRequest(BaseModel): @@ -263,7 +299,7 @@ async def sse_telemetry_stream(request: Request): if await request.is_disconnected(): break try: - snap = await asyncio.wait_for(q.get(), timeout=15.0) + snap = await asyncio.wait_for(q.get(), timeout=5.0) if snap is None: # shutdown sentinel break yield f"data: {json.dumps(snap)}\n\n" @@ -483,9 +519,10 @@ async def api_autotune_status(): async def api_autotune_sweep(req: SweepRequest): """Walk a clock offset upward, measuring tok/s and watching for instability at each step.""" res = await autotune.sweep( - knob=req.knob, profile=req.profile, model=req.model, + knob=req.knob, profile=req.profile, workload=req.workload, model=req.model, start=req.start, stop=req.stop, step=req.step, - repeats=req.repeats, apply_best=req.apply_best, + repeats=req.repeats, max_steps=req.max_steps, + include_unlocked=req.include_unlocked, apply_best=req.apply_best, ) if not res.get("success"): raise HTTPException(status_code=400, detail=res.get("error")) diff --git a/vram_arbitrator.py b/vram_arbitrator.py index 071fd59..74752d6 100644 --- a/vram_arbitrator.py +++ b/vram_arbitrator.py @@ -666,6 +666,10 @@ class AutoArbitrator: self.comfy_idle_since: Optional[float] = None self.oc_profile = None self.pending_purge = False + # While a tuning sweep is running, the arbitrator must not fight it: a ComfyUI + # benchmark would otherwise trip trigger_comfy_priority, which reapplies the whole + # 'comfy' profile and silently overwrites the clock the sweep is measuring. + self.oc_suspended = False self.stats = {"yields": 0, "purges": 0, "yield_timeouts": 0, "deferred_purges": 0} async def start(self): @@ -841,9 +845,19 @@ class AutoArbitrator: pass await asyncio.sleep(interval) + def suspend_oc(self, reason: str = "tuning sweep") -> None: + self.oc_suspended = True + logger.info(f"Overclock auto-switching suspended ({reason})") + + def resume_oc(self, profile: Optional[str] = None) -> None: + self.oc_suspended = False + # Forget the cached profile so the next transition actually reapplies. + self.oc_profile = profile + logger.info("Overclock auto-switching resumed") + def _apply_oc_profile(self, profile: str): """Apply an overclock profile in a background thread; only fire on transition.""" - if self.oc_profile == profile: + if self.oc_suspended or self.oc_profile == profile: return self.oc_profile = profile try: From 01d2f4cfddd35fba76833458067eb2a912b46874 Mon Sep 17 00:00:00 2001 From: drjones Date: Fri, 28 Aug 2026 14:40:05 -0700 Subject: [PATCH 6/6] Recalibrate cache-hit thresholds against measured loads; bring MCP to parity Calibration. The same 12.87GB model loaded through Ollama on this box: 3.1% resident (FADV_DONTNEED) -> 34.3s -> 0.38 GB/s 100% resident (force-warmed) -> 4.9s -> 2.63 GB/s The thresholds had been guessed from PCIe bus bandwidth: cache hit at >=5 GB/s. A fully warm load only reaches 2.63 GB/s, because load_duration covers host-to-device transfer and model init as well as the file read -- the page cache itself reads at 6.4 GB/s. The 5 GB/s bar was therefore unreachable, and every warm load was being reported as a partial hit. Now 2.0 / 0.8 GB/s, either side of the measured 6.9x separation. Warm-skip was also unsafe. A 12.87GB blob was skipped as already resident on the strength of twelve 2MB probe windows, then loaded at 2.44 GB/s. Skipping now requires warm_confident: an exact cachestat reading, or a probe finding every one of 32 denser samples resident. warm_file_to_ram/warm_ollama_blob take force=True, exposed on the warm-model endpoint, whose Pydantic model was missing the field entirely. MCP parity: the server had drifted well behind the REST API. Adds tools for measured residency, warm planning, VRAM requests, per-profile analytics, thermal governor control, overclock status/apply/restore, and autotune sweeps plus status -- 23 tools and 6 resources, up from 12 and 3. The telemetry store now starts in __main__ rather than at import scope, since server.py imports this module for the benchmark tool. README: replaced the remaining theoretical claims (31.5 GB/s bus rate, sub-1.5s loads, 15ms yields) with the measured numbers. Co-Authored-By: Claude Opus 5 --- README.md | 41 ++++++++++++--- mcp_server.py | 128 +++++++++++++++++++++++++++++++++++++++++++-- ram_optimizer.py | 35 ++++++++----- server.py | 5 +- vram_arbitrator.py | 27 +++++++--- 5 files changed, 204 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 1b23626..8adc719 100644 --- a/README.md +++ b/README.md @@ -107,8 +107,8 @@ only if the card actually needs it. * **Server-Sent Events (SSE)**: A single background sampler produces one 1 Hz snapshot and fans it out to every subscriber via `GET /api/stream`. Previously each connected client independently re-ran the whole snapshot โ€” NVML, `/proc/meminfo`, an HTTP round-trip each to Ollama and ComfyUI, and a recursive walk of the ComfyUI models tree with a `stat()` per checkpoint โ€” once per second, so opening the dashboard in three tabs tripled the load on the thing it was measuring. Slow clients drop stale frames instead of stalling the sampler. ### ๐Ÿค– Model Context Protocol (MCP 2.0) Server -* **12 Native Agentic Tools**: Allows AI agents (Antigravity CLI, Claude Desktop, Cursor) to manage GPU resources, trigger model hot-swaps, tune fan curves, and inspect telemetry. -* **3 Live MCP Resources**: Exposes live metrics, model catalogs, and switch logs as streamable resources (`gpu://metrics/live`, `gpu://models/catalog`, `gpu://history/switches`). +* **23 Native Agentic Tools**: Allows AI agents (Antigravity CLI, Claude Desktop, Cursor) to manage GPU resources, trigger model hot-swaps, measure page-cache residency, read persisted performance analytics, drive the thermal governor, and run overclock sweeps. +* **6 Live MCP Resources**: Live metrics, model catalog, switch log, measured cache residency, per-profile analytics, and the overclock profiles with the evidence behind each setting. * **Dual Transport Support**: Run via standard input/output (`--stdio`) or network Server-Sent Events (`--sse --port 8001`). ### โฑ๏ธ Automated Latency & Throughput Benchmark Engine @@ -135,19 +135,32 @@ flowchart TD REST["REST API & OpenAPI Docs"] MCP["Model Context Protocol (MCP 2.0)"] SSE["1Hz Real-Time SSE Stream"] - Arbitrator["VRAM Arbitrator (15ms Soft-Yield)"] + Arbitrator["VRAM Arbitrator (confirmed yield)"] Overclock["Overclock & Fan Manager"] Warmer["Page Cache Pre-Warmer"] end - HostRAM <== "PCIe 4.0 x16 Bus (~31.5 GB/s Hot-Swap)" ==> GPU + HostRAM <== "PCIe 4.0 x16 Bus (measured 2.6 GB/s warm model load)" ==> GPU Orchestrator --> GPU Orchestrator --> HostRAM ``` ### The Physics of Sub-Second Switching * **Host RAM as Staging**: Active LLMs and diffusion checkpoints remain resident in the 64GB Linux Page Cache. -* **PCIe 4.0 x16 Hot-Swapping**: Transferring weights across PCIe 4.0 x16 achieves **~31.5 GB/s** bandwidth, reducing model loads from 30+ seconds (disk) to **under 1.5 seconds**. +* **Warm vs cold model loads, measured.** The same 12.87 GB model, loaded through Ollama on this box: + + | Page-cache residency | Load time | Effective rate | + | :--- | :--- | :--- | + | 3.1% (dropped with `FADV_DONTNEED`) | 34.3 s | 0.38 GB/s | + | 100% (force-warmed) | 4.9 s | 2.63 GB/s | + + A **6.9ร— speedup**, and the reason the page cache matters. Note the effective rate is + well below the PCIe 4.0 x16 bus rate and below the 6.4 GB/s the page cache itself + reads at: Ollama's `load_duration` also covers host-to-device transfer and model + initialisation, not just the file read. Classification thresholds are calibrated + against these measured numbers rather than the theoretical bus bandwidth โ€” an earlier + 5 GB/s cache-hit bar sat above what a fully warm load can even achieve, so every warm + load was misreported as a partial hit. * **Soft-Yielding**: Dropping Ollama's VRAM allocation via `keep_alive: 0` preserves the weights in host RAM. Measured on this box: the HTTP request returns in **~63 ms**, and the driver finishes releasing 14.9 GB **~77 ms after that**. HyperSwap waits for the second number before handing VRAM to ComfyUI โ€” the earlier "~15 ms" figure timed the request, not the release. --- @@ -220,19 +233,33 @@ HyperSwap includes a native **MCP 2.0 server** (`mcp_server.py`) exposing orches | **`set_gpu_fan_speed`** | `mode` (str), `percent` (optional int) | Sets fan speed mode (`auto`\|`manual`) and target PWM % (30โ€“100%). | | **`get_host_memory_status`** | *None* | 64GB host RAM breakdown, active page cache size, and cache ratio. | | **`switch_ollama_model`** | `model_name` (str), `keep_alive` (str) | Hot-swaps active LLM in VRAM, measures latency (ms) and tokens/sec. | -| **`soft_yield_ollama_vram`** | `model_name` (optional str) | Yields Ollama VRAM to 0 MB in ~15ms while keeping model weights in RAM cache. | +| **`soft_yield_ollama_vram`** | `model_name` (optional str) | Yields Ollama VRAM to 0 MB and waits for NVML to confirm the driver actually released it. Returns the request/confirm split. | | **`purge_comfyui_vram`** | *None* | Purges loaded diffusion models from ComfyUI pipeline VRAM. | -| **`prewarm_all_models_to_ram`** | *None* | Faults all local LLM and diffusion checkpoints into Linux OS page cache. | +| **`prewarm_all_models_to_ram`** | *None* | Warms the highest-value models into page cache within a byte budget, skipping what is already resident. | | **`prewarm_single_model`** | `model_name` (optional str), `filepath` (optional str) | Pre-warms a single GGUF or Safetensors file into RAM. | | **`list_available_models`** | *None* | Lists all installed Ollama models and discovered ComfyUI Safetensors on disk. | | **`get_switch_history`** | `limit` (int, default 20) | Retrieves recent switch events, millisecond latencies, and RAM hit status. | | **`run_model_switch_benchmark`**| `iterations` (int, default 2) | Automated round-trip latency benchmark between installed models. | +| **`get_page_cache_residency`** | `include_files` (bool) | Measured page-cache residency per model file, with the measurement method used for each. | +| **`get_warm_plan`** | `budget_gb` (optional float) | Previews what warming would read and skip, ranked by recency/frequency. Does not warm. | +| **`request_vram_for_ollama`** | `needed_gb` (float) | Purges ComfyUI's checkpoints immediately if VRAM headroom is short, bypassing the idle timer. | +| **`get_profile_performance`** | `days` (float, default 7) | Measured tok/s and thermals per overclock profile, from persisted history. | +| **`get_thermal_governor_status`** | *None* | Current derate level, the reason for it, and escalation history. | +| **`set_thermal_governor`** | `enabled` (optional bool), `reset` (bool) | Enable/disable the governor, or clear an active derate. | +| **`get_overclock_status`** | *None* | Active profile, all profiles with their evidence, and which levers this driver honours. | +| **`apply_overclock_profile`** | `profile` (str) | Apply `ollama` \| `comfy` \| `balanced`. | +| **`restore_stock_gpu_state`** | *None* | Drop clock locks and offsets, restore default power limit, return fans to automatic. | +| **`run_overclock_sweep`** | `knob`, `profile`, `workload`, `start`, `stop`, `repeats`, `apply_best` | Sweep a knob against a real workload and report the fastest stable value. Verifies the knob moves the hardware first. Takes minutes. | +| **`get_autotune_status`** | *None* | Sweep progress, the last result table, and all recorded autotune steps. | ### MCP Resources List * `gpu://metrics/live`: Real-time snapshot of GPU sensors and RAM page cache. * `gpu://models/catalog`: Catalog of all discovered GGUF and Safetensors models. * `gpu://history/switches`: Event log of recent model transitions and swap speeds. +* `gpu://cache/residency`: Measured page-cache residency across every model on disk. +* `gpu://analytics/profiles`: Measured throughput and thermals per overclock profile. +* `gpu://overclock/profiles`: Overclock profiles including the measurement behind each setting. --- diff --git a/mcp_server.py b/mcp_server.py index 7851760..47e037a 100644 --- a/mcp_server.py +++ b/mcp_server.py @@ -7,16 +7,19 @@ import logging from typing import Dict, List, Any, Optional from mcp.server import MCPServer -import ram_optimizer -import vram_arbitrator +import autotune import overclock_manager +import ram_optimizer +import telemetry_store +import thermal_governor +import vram_arbitrator logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s") logger = logging.getLogger("gpu_swapper_mcp") mcp = MCPServer( name="gpu-program-swapper", - version="1.0.0", + version="2.0.0", description="Orchestrates high-speed GPU VRAM hot-swaps between Ollama LLMs and ComfyUI with 64GB RAM cache telemetry." ) @@ -133,6 +136,105 @@ def set_gpu_fan_speed(mode: str = "auto", percent: Optional[int] = None) -> str: res = overclock_manager.set_fan_auto() return json.dumps(res, indent=2) +@mcp.tool() +def get_page_cache_residency(include_files: bool = True) -> str: + """Measure how much of each model on disk is genuinely resident in the Linux page cache. + + Uses cachestat(2) where the kernel permits it and a read-rate probe where it does not + (Ollama blobs are owned by another user). Reports which method was used per file, and + marks anything it cannot measure rather than guessing.""" + report = ram_optimizer.get_cache_report(include_files=include_files) + report["capability"] = ram_optimizer.residency_capability() + return json.dumps(report, indent=2, default=str) + + +@mcp.tool() +def get_warm_plan(budget_gb: Optional[float] = None) -> str: + """Preview which models pre-warming would load into RAM, in what order, and what it + would skip โ€” ranked by recency/frequency and capped by a byte budget. Does not warm.""" + return json.dumps(ram_optimizer.build_warm_plan(budget_gb), indent=2, default=str) + + +@mcp.tool() +async def request_vram_for_ollama(needed_gb: float = 0.0) -> str: + """Free VRAM for an LLM right now: purges ComfyUI's cached checkpoints immediately if + there is not enough headroom, instead of waiting for the normal idle timer.""" + res = await vram_arbitrator.arbitrator.request_vram_for_ollama(needed_gb) + return json.dumps(res, indent=2, default=str) + + +@mcp.tool() +def get_profile_performance(days: float = 7.0) -> str: + """Compare measured decode throughput and thermals per overclock profile, from + persisted history. Answers whether a given profile is actually delivering more tok/s.""" + return json.dumps({ + "window_days": days, + "profiles": telemetry_store.profile_comparison(days), + "swaps": telemetry_store.swap_stats(days), + }, indent=2, default=str) + + +@mcp.tool() +def get_thermal_governor_status() -> str: + """Current thermal derate level, why it was applied, and the escalation history.""" + return json.dumps(thermal_governor.governor.get_status(), indent=2, default=str) + + +@mcp.tool() +def set_thermal_governor(enabled: Optional[bool] = None, reset: bool = False) -> str: + """Enable or disable the thermal governor, or clear an active derate and reapply the + full profile.""" + if enabled is not None: + thermal_governor.governor.set_enabled(enabled) + if reset: + thermal_governor.governor.reset() + return json.dumps(thermal_governor.governor.get_status(), indent=2, default=str) + + +@mcp.tool() +def get_overclock_status() -> str: + """Active overclock profile, all profiles with the evidence behind their settings, and + which hardware levers this driver actually honours (clock offsets are ignored on some).""" + return json.dumps(overclock_manager.get_status(), indent=2, default=str) + + +@mcp.tool() +def apply_overclock_profile(profile: str) -> str: + """Apply an overclock profile by name: ollama | comfy | balanced.""" + return json.dumps(overclock_manager.apply_profile(profile), indent=2, default=str) + + +@mcp.tool() +def restore_stock_gpu_state() -> str: + """Drop all clock locks and offsets, restore the default power limit, and return the + fans to automatic control.""" + return json.dumps(overclock_manager.restore_safe("MCP request"), indent=2, default=str) + + +@mcp.tool() +async def run_overclock_sweep(knob: str = "power_limit_w", profile: str = "ollama", + workload: str = "auto", start: Optional[int] = None, + stop: Optional[int] = None, repeats: int = 1, + apply_best: bool = False) -> str: + """Sweep one GPU knob against a real workload and report the fastest stable value. + + knob: power_limit_w | lock_mem_mhz | lock_core_max | mem_offset_mhz | core_offset_mhz + workload: 'ollama' (decode tok/s), 'comfy' (SDXL it/s), or 'auto' to match the profile. + + Verifies the knob actually moves the hardware before sweeping, refuses to run while + ComfyUI is busy, and always restores the original profile. Takes minutes.""" + res = await autotune.sweep(knob=knob, profile=profile, workload=workload, + start=start, stop=stop, repeats=repeats, + apply_best=apply_best) + return json.dumps(res, indent=2, default=str) + + +@mcp.tool() +def get_autotune_status() -> str: + """Sweep progress, the last sweep's full result table, and every recorded autotune step.""" + return json.dumps(autotune.get_status(), indent=2, default=str) + + # ========================================== # MCP RESOURCES # ========================================== @@ -154,6 +256,21 @@ def get_switch_history_resource() -> str: """Recent model switch events and latencies.""" return json.dumps(vram_arbitrator.get_switch_history(), indent=2) +@mcp.resource("gpu://cache/residency") +def get_cache_residency_resource() -> str: + """Measured page-cache residency across every model on disk.""" + return json.dumps(ram_optimizer.get_cache_report(include_files=True), indent=2, default=str) + +@mcp.resource("gpu://analytics/profiles") +def get_profile_analytics_resource() -> str: + """Measured throughput and thermals per overclock profile, from persisted history.""" + return json.dumps(telemetry_store.profile_comparison(7.0), indent=2, default=str) + +@mcp.resource("gpu://overclock/profiles") +def get_overclock_profiles_resource() -> str: + """Overclock profiles, including the measurement recorded behind each setting.""" + return json.dumps(overclock_manager.get_status(), indent=2, default=str) + if __name__ == "__main__": import argparse @@ -163,6 +280,11 @@ if __name__ == "__main__": parser.add_argument("--port", type=int, default=8001, help="Port for SSE transport") args = parser.parse_args() + # Only when run as a standalone server. server.py imports this module for the + # benchmark tool, and starting the store at import scope would spin up a writer as a + # side effect of that import. + telemetry_store.start() + if args.sse: mcp.run(transport="sse", host="0.0.0.0", port=args.port) else: diff --git a/ram_optimizer.py b/ram_optimizer.py index bf0bff2..0a452ef 100644 --- a/ram_optimizer.py +++ b/ram_optimizer.py @@ -143,7 +143,7 @@ PROBE_WINDOW_BYTES = 2 * 1024 * 1024 PROBE_CACHED_GBPS = 1.5 -def _throughput_probe(fd: int, size: int) -> Dict[str, Any]: +def _throughput_probe(fd: int, size: int, windows_override: Optional[int] = None) -> Dict[str, Any]: """Infer residency by timing reads of small windows spread across the file. Used only where cachestat is not permitted (Ollama's blobs are owned by uid `ollama`). @@ -158,7 +158,7 @@ def _throughput_probe(fd: int, size: int) -> Dict[str, Any]: pollution the probe itself created, and leaving them behind would slowly warm the cache with data nobody asked for. """ - windows = min(PROBE_WINDOWS, max(int(size // PROBE_WINDOW_BYTES), 1)) + windows = min(windows_override or PROBE_WINDOWS, max(int(size // PROBE_WINDOW_BYTES), 1)) if windows <= 0: return {"resident_pct": 0.0, "windows": 0} @@ -194,7 +194,8 @@ def _throughput_probe(fd: int, size: int) -> Dict[str, Any]: } -def page_residency(filepath: str, allow_probe: bool = True) -> Dict[str, Any]: +def page_residency(filepath: str, allow_probe: bool = True, + probe_windows: Optional[int] = None) -> Dict[str, Any]: """Measure what fraction of a file is resident in the Linux page cache.""" try: size = os.path.getsize(filepath) @@ -216,7 +217,7 @@ def page_residency(filepath: str, allow_probe: bool = True) -> Dict[str, Any]: method, measurable = "cachestat", True extra = {"dirty_pages": cs.nr_dirty, "evicted_pages": cs.nr_evicted} elif allow_probe: - probe = _throughput_probe(fd, size) + probe = _throughput_probe(fd, size, probe_windows) pct = probe["resident_pct"] method, measurable = "probe", True extra = {"probe_windows": probe["windows"], "probe_median_gbps": probe.get("median_gbps")} @@ -236,6 +237,11 @@ def page_residency(filepath: str, allow_probe: bool = True) -> Dict[str, Any]: "method": method, "measurable": measurable, "warm": pct >= WARM_SKIP_THRESHOLD_PCT, + # Only an exact measurement is trustworthy enough to skip work on. A probe of a + # dozen 2 MB windows can clear 90% on a file that is mostly cold -- observed + # here as a 12.87 GB "already resident" blob that then loaded at 2.44 GB/s. + "warm_confident": (method == "cachestat" and pct >= WARM_SKIP_THRESHOLD_PCT) + or (method == "probe" and pct >= 100.0), **extra, } except Exception as e: @@ -468,16 +474,19 @@ def get_cache_report(include_files: bool = True, force_refresh: bool = False) -> # ---------------------------------------------------------------- warming def warm_file_to_ram(filepath: str, chunk_size: int = 16 * 1024 * 1024, - skip_if_warm: bool = True) -> Dict[str, Any]: - """Pre-fault a file into the Linux page cache, skipping it if already resident.""" + skip_if_warm: bool = True, force: bool = False) -> Dict[str, Any]: + """Pre-fault a file into the Linux page cache, skipping it only if confidently resident.""" if not os.path.exists(filepath): return {"success": False, "error": f"File not found: {filepath}", "duration_ms": 0} - before = page_residency(filepath) - if skip_if_warm and before.get("warm"): + # Probe densely here: this decision skips real work, so it is worth 32 samples + # rather than 12. + before = page_residency(filepath, probe_windows=32) + if skip_if_warm and not force and before.get("warm_confident"): return { "success": True, "filepath": filepath, "skipped": True, "reason": "already resident", "resident_pct": before.get("resident_pct"), + "method": before.get("method"), "size_mb": round(before.get("size_bytes", 0) / (1024**2), 2), "duration_ms": 0.0, "bytes_read": 0, } @@ -499,7 +508,7 @@ def warm_file_to_ram(filepath: str, chunk_size: int = 16 * 1024 * 1024, bytes_read += n duration = time.perf_counter() - t0 - after = page_residency(filepath) + after = page_residency(filepath, probe_windows=32) return { "success": True, "filepath": filepath, @@ -551,11 +560,11 @@ async def warm_ollama_model(model_name: str, keep_alive: str = "5m") -> Dict[str "duration_ms": round((time.perf_counter() - t0) * 1000, 2)} -def warm_ollama_blob(model_name: str) -> Dict[str, Any]: +def warm_ollama_blob(model_name: str, force: bool = False) -> Dict[str, Any]: """Warm a specific Ollama model's GGUF into page cache without touching VRAM.""" for f in find_ollama_model_files(): if f["model"] == model_name: - res = warm_file_to_ram(f["full_path"]) + res = warm_file_to_ram(f["full_path"], force=force) res["model"] = model_name return res return {"success": False, "error": f"no blob found for model '{model_name}'"} @@ -603,13 +612,13 @@ def build_warm_plan(budget_gb: Optional[float] = None) -> Dict[str, Any]: if c["full_path"] in seen_paths: continue seen_paths.add(c["full_path"]) - res = page_residency(c["full_path"]) + res = page_residency(c["full_path"], probe_windows=32) entry = { "name": c["name"], "kind": c["kind"], "full_path": c["full_path"], "size_gb": c.get("size_gb", 0), "score": round(c["score"], 4), "resident_pct": res.get("resident_pct", 0.0), } - if res.get("warm"): + if res.get("warm_confident"): entry["action"] = "already-warm" skipped.append(entry) continue diff --git a/server.py b/server.py index 07eab1e..e89d7d1 100644 --- a/server.py +++ b/server.py @@ -227,6 +227,7 @@ class WarmRequest(BaseModel): model_name: Optional[str] = Field(None, description="Ollama model name to warm", example="gemma4:26b") filepath: Optional[str] = Field(None, description="Absolute file path of Safetensors/GGUF to warm into RAM") blob_only: bool = Field(False, description="Warm the model's weights into page cache without loading VRAM") + force: bool = Field(False, description="Warm even if residency sampling thinks it is already resident") class WarmAllRequest(BaseModel): budget_gb: Optional[float] = Field(None, description="Byte budget for warming; defaults to 70% of MemAvailable", example=24.0) @@ -362,11 +363,11 @@ async def api_warm_plan(budget_gb: Optional[float] = Query(None, description="Ov async def api_warm_model(req: WarmRequest): """Pre-warm a specific Ollama model or file path into the Linux page cache.""" if req.model_name and req.blob_only: - return ram_optimizer.warm_ollama_blob(req.model_name) + return ram_optimizer.warm_ollama_blob(req.model_name, force=req.force) if req.model_name: return await ram_optimizer.warm_ollama_model(req.model_name, keep_alive="1m") if req.filepath: - return ram_optimizer.warm_file_to_ram(req.filepath) + return ram_optimizer.warm_file_to_ram(req.filepath, force=req.force) raise HTTPException(status_code=400, detail="model_name or filepath required") @app.get("/api/cache/report", summary="Measured Page-Cache Residency", tags=["Memory Optimization"]) diff --git a/vram_arbitrator.py b/vram_arbitrator.py index 74752d6..9610080 100644 --- a/vram_arbitrator.py +++ b/vram_arbitrator.py @@ -29,10 +29,20 @@ COMFY_API_BASE = "http://127.0.0.1:8188" # Circular buffer for transition events (the durable log lives in telemetry_store) SWITCH_HISTORY = deque(maxlen=50) -# Bandwidth thresholds used to classify how a model actually got into VRAM. -# PCIe 4.0 x16 tops out near 31.5 GB/s; this NVMe sustains well under 2 GB/s. -RAM_HIT_GBPS = 5.0 -PARTIAL_HIT_GBPS = 1.5 +# Bandwidth thresholds for classifying how a model reached VRAM, calibrated by measuring +# the same 12.87 GB model loaded cold and warm on this box (2026-08-28): +# +# 3.1% resident -> 34.3 s -> 0.38 GB/s +# 100% resident -> 4.9 s -> 2.63 GB/s +# +# The first cut at these numbers assumed a page-cache-fed load would approach the bus +# rate and set the cache-hit bar at 5 GB/s. It does not: Ollama's load_duration covers +# host-to-device transfer and model initialisation as well as the file read, so a fully +# resident model still reports ~2.6 GB/s while the page cache itself reads at 6.4 GB/s. +# A 5 GB/s bar could therefore never be met, and every warm load was being reported as +# a partial hit. Thresholds now sit either side of the measured 6.9x separation. +RAM_HIT_GBPS = 2.0 +PARTIAL_HIT_GBPS = 0.8 # How long Ollama's VRAM may take to actually drain before we stop waiting. # Ollama will not unload a model while a generation is in flight, so a short ceiling @@ -557,11 +567,14 @@ def classify_load(size_bytes: int, load_duration_ms: float) -> Dict[str, Any]: return {"cache_status": "Already in VRAM", "load_gbps": None, "is_ram_hit": True} if not size_bytes: # No size on record โ€” fall back to the old heuristic, but say so. + # Without a size we cannot compute bandwidth at all; this is a guess and is + # labelled as one. 8s roughly splits the measured warm (4.9s) and cold (34.3s) + # loads for a mid-size model, but it is meaningless for very small or large ones. return { - "cache_status": "RAM Cache Hit โšก" if load_duration_ms < 2500 else "Cold Disk Load ๐Ÿ’พ", + "cache_status": "RAM Cache Hit โšก" if load_duration_ms < 8000 else "Cold Disk Load ๐Ÿ’พ", "load_gbps": None, - "is_ram_hit": load_duration_ms < 2500, - "detail": "size unknown, fell back to duration heuristic", + "is_ram_hit": load_duration_ms < 8000, + "detail": "size unknown, fell back to a duration guess", } gbps = (size_bytes / (1024**3)) / (load_duration_ms / 1000.0) if gbps >= RAM_HIT_GBPS: