gpu-program-swapper/README.md

HyperSwap // GPU Program Swapper & Memory Orchestrator

HyperSwap is an ultra-low-latency VRAM arbitrator, host RAM cache pre-warmer, dynamic hardware overclocker, and real-time telemetry dashboard designed specifically for Linux deployment machines that simultaneously host Ollama LLM workloads and ComfyUI Diffusion pipelines on a single NVIDIA GPU.

---

Real-Time Telemetry & Control Dashboard

The HyperSwap live dashboard running on :9090, demonstrating real-time VRAM allocation tracking (Ollama 14.14 GB, ComfyUI 0.38 GB, Desktop 0.6 GB), 33.89 GB of models resident in 64GB host RAM cache, live dual-axis memory charts, hardware fan control, and sub-25ms model VRAM purges and soft-yields.

---

1. Feature Matrix

⚡ Bidirectional VRAM Hot-Swapping & Arbitration

• Both directions are now automatic. Yielding Ollama for ComfyUI always was; the reverse was not, despite "bidirectional" in this heading. Which way an LLM fails when it cannot fit depends on configuration: with n_gpu_layers left to Ollama it spills layers to the CPU and reports size_vram < size (roughly an order of magnitude slower, and silent). With n_gpu_layers pinned — 99 on this box — it refuses outright with cudaMalloc failed: out of memory. Both are handled: the spill triggers a reclaim from an idle ComfyUI, and the hard failure is caught by switch_ollama_model, which reclaims and retries once. Measured: a 12.87 GB model that returned HTTP 500 from Ollama directly now loads through HyperSwap after reclaiming 6.83 GB, at 3.85 GB/s.
• A busy LLM is not a failed yield. A model mid-generation cannot unload; the keep_alive: 0 request queues behind it and applies when it finishes. That is reported as busy (returning in ~610 ms) rather than blocking, with per-model backoff and a detached watcher that logs the eventual release. Only VRAM held while the GPU sits idle counts as a fault.
• Confirmed Soft-Yield (barrier, not fire-and-forget): Releases Ollama VRAM allocations (keep_alive: 0) down to 0 MB, then waits on NVML until the driver has actually freed the allocation before letting ComfyUI proceed. Posting keep_alive: 0 only asks Ollama to unload; on this box the HTTP call returns in ~63 ms while the driver takes a further ~77 ms to release 14.9 GB. Returning during that window is how diffusion ends up allocating into VRAM that is still occupied.
• Idle-Aware ComfyUI Purge: Diffusion checkpoints are held for COMFY_IDLE_PURGE_S (30 s) of genuinely empty queue rather than purged 1.5 s after every prompt — iterating on a workflow no longer pays a full checkpoint reload per run. An immediate purge still happens the moment Ollama actually asks for VRAM (POST /api/request-vram).
• Real-Time ComfyUI WebSocket & Watchdog Listener: Subscribes directly to ws://127.0.0.1:8188/ws. The WebSocket is the primary signal; a connection-pooled watchdog polls /queue at 1 Hz purely as a fallback, backing off to 3 s while the socket is healthy.
• Process-Level VRAM Attribution: Live NVML process inspection attributes exact GPU memory usage across Ollama (llama-server), ComfyUI (python), and Desktop display servers (gnome-shell, Xorg).
• Bandwidth-Classified Transition History: Every switch is classified by the bandwidth it actually achieved (model size ÷ load duration) rather than a fixed duration threshold: RAM Cache Hit ⚡ (≥5 GB/s), Partial Cache 🌤 (≥1.5 GB/s), Cold Disk Load 💾 (below that). The previous load_duration < 2500 ms rule called a 12.9 GB model read at 2.9 GB/s a "cold disk load" and a 0.5 GB model read from NVMe a "cache hit".

🧠 64GB Host RAM Cache & Page Pre-warmer

• Zero-Latency Model Discovery: Automatic cataloging of all local Ollama models (/usr/share/ollama/.ollama/models, ~/.ollama/models) and ComfyUI model directories (checkpoints, diffusion_models, unet, vae, clip, loras, controlnet).
• POSIX fadvise & Pinned Pre-warmer: Pre-faults multi-gigabyte GGUFs and Safetensors into the Linux OS Page Cache so that reloading models runs at page-cache speed rather than disk speed.
• Measured Residency via cachestat(2): Residency is measured, not assumed. cachestat(2) gives exact cached-page counts per file. Where the kernel refuses it — it only permits introspection of files you own, and Ollama's blobs are owned by uid ollama — HyperSwap falls back to a randomised read-rate probe and labels the result as such. Files it cannot measure are reported as unmeasurable rather than guessed at.
• Budgeted, Ranked Warming: This box has 64 GB of RAM and >270 GB of model files; reading everything simply evicts whatever was warmed first. Files are ranked by recency/frequency (from the persisted event log) and warmed until a byte budget is spent, skipping anything already resident. GET /api/warm-plan previews the decision without executing it.
• Memory Telemetry: Real-time breakdown of Total Host RAM, Applications Memory, Active Model Page Cache, Free Memory, and measured Cache Residency Ratio.

🎛️ Measured Overclock Profiles & Thermal Management

Every profile setting in this repo is now backed by a measurement from autotune.py on this specific card and driver. Several long-standing settings turned out to do nothing.

What this driver actually honours (NVIDIA 595.84, RTX 4080 SUPER):

Lever	Mechanism	Works?
Power limit	nvidia-smi -pl	✅ Yes — and it is the only lever that changes anything measurable
Core / memory clock lock	nvidia-smi -lgc / -lmc	✅ Applies correctly, but made no measurable difference to either workload
Core / memory clock offsets	nvidia-settings -a ...Offset	❌ Silently ignored. The driver reports assigned value 0 and the attribute still reads back 250. Detected automatically by offsets_supported(); apply_profile now skips them and says so rather than pretending.
Fan control	nvidia-settings GPUTargetFanSpeed	✅ Yes

Measured results (POST /api/autotune/sweep):

LLM decode is not power-bound. Throughput is flat across the card's entire power range — the GPU never drew more than 224 W no matter what the limit allowed:

Power limit	222 W	259 W	296 W	320 W	333 W	370 W
tok/s (qwen3.8long)	73.17	73.13	73.43	73.51	73.10	73.04

Diffusion is power-bound. Here the watts genuinely buy throughput:

Power limit	222 W	259 W	296 W	320 W	333 W	370 W
it/s (SDXL 1024, 20 steps)	5.48	6.22	6.50	6.52	6.63	6.71

Clock locks changed nothing for either workload. Memory clock: 72.6 tok/s locked at 11251 MHz vs 72.7 unlocked. Core clock: 6.73 it/s unlocked vs 6.77 locked at 3105 MHz — and 6.78 at 2400 MHz, so diffusion here is not core-clock-bound at all.

Memory bandwidth is the decode bottleneck, confirming the profile's original premise — dropping the memory clock to 5001 MHz halves throughput (35.9 tok/s vs 72.6). The card simply reaches its top memory clock on its own; pinning it there adds nothing.

Resulting profiles:

• ollama — 320 W (stock), no locks, automatic fans. Decode draws ~224 W and is bandwidth-bound, so the previous 370 W limit and 100% fan pinning bought nothing.
• comfy — 370 W, no locks, automatic fans. The extra power is worth a measured +2.8% over the 320 W stock default.
• balanced — stock power and boost, automatic fans.

On fans: all three profiles previously pinned the fans to manual 100%. Across 48,435 telemetry samples this card has never exceeded 81 °C and has logged zero thermal throttle events; the ollama profile was holding 49.6 °C average by running the fans at 87%. Fans are now automatic in every profile, with the thermal governor escalating them only if the card actually needs it.

🌡️ Thermal Governor (closed-loop de-escalation)

• Every overclock lever here is sticky: a profile locks clocks and pins the fans to a manual PWM, and nothing used to undo that. The governor watches the telemetry the sampler already collects (so it costs no extra NVML calls) and walks the overclock back through a four-step derate ladder when the card runs hot or reports a hardware throttle.
• Hysteresis by design: escalation needs 5 consecutive bad samples, recovery needs 30 consecutive good ones, with a 20 s cooldown between changes — a single spike during a diffusion step will not cause profile thrash.
• Guaranteed restore: stock clocks, default power limit and automatic fans are restored by the server's shutdown hook and by a systemd ExecStopPost=, so a SIGKILL cannot leave the card with locked clocks and fans pinned at 100%.

🔬 Overclock Autotune (autotune.py)

• Sweeps a knob (power_limit_w, lock_mem_mhz, lock_core_max, clock offsets) and reports the fastest stable value.
• Both workloads are measurable. workload=ollama benchmarks decode throughput in tok/s; workload=comfy queues a fixed SDXL 1024/20-step graph through ComfyUI's API and measures it/s. Without the second one there was no way to tell whether the compute-oriented comfy profile was doing anything at all — and it was not.
• Refuses to sweep a knob the driver ignores. A preflight applies a probe value and confirms the hardware moved; the probe is chosen as the candidate furthest from the current reading, since probing with the maximum proves nothing when the card already sits there. This is what caught the silently-discarded clock offsets.
• Instability detection: kernel Xid/NVRM messages via journalctl -k, benchmark failure, degenerate output, and a temperature ceiling. The sweep stops climbing the moment a step looks unstable.
• Honest gain reporting: gain against the profile's current setting is reported separately from the spread across values tried. Conflating them turns a flat result into a headline "+102%".
• Safety: refuses to start while ComfyUI is executing, suspends the arbitrator's automatic profile switching for the duration (otherwise a diffusion benchmark trips trigger_comfy_priority, which reapplies the whole profile and overwrites the clock being measured), and restores the original profile in a finally block — including on exception or cancellation.

🗄️ Persistent Telemetry Store (telemetry_store.py)

• Swap history used to be an in-memory deque(maxlen=50) that evaporated on every restart. Telemetry and events now persist to SQLite (WAL, single writer thread, batched 1 Hz inserts, automatic retention pruning) at roughly 0.4 MB per hour.
• This is what makes the app's central question answerable: GET /api/analytics/profiles compares decode throughput per overclock profile, joined against the thermals recorded while that profile was active.

📊 Real-Time Web Telemetry Dashboard (:9090)

• Live Hardware Telemetry: GPU utilization %, GPU temperature (°C), power draw (W), fan speeds (%), and graphics/memory clock frequencies (MHz).
• Live Dual-Axis Time-Series Chart: Real-time graphical visualization of VRAM usage (GB) and Host RAM Cache (GB) with zero frontend polling overhead.
• Interactive Control Center: Trigger model hot-swaps, soft-yields, cache pre-warms, fan adjustments, and benchmarks directly from the web interface.
• Server-Sent Events (SSE): A single background sampler produces one 1 Hz snapshot and fans it out to every subscriber via GET /api/stream. Previously each connected client independently re-ran the whole snapshot — NVML, /proc/meminfo, an HTTP round-trip each to Ollama and ComfyUI, and a recursive walk of the ComfyUI models tree with a stat() per checkpoint — once per second, so opening the dashboard in three tabs tripled the load on the thing it was measuring. Slow clients drop stale frames instead of stalling the sampler.

🤖 Model Context Protocol (MCP 2.0) Server

• 23 Native Agentic Tools: Allows AI agents (Antigravity CLI, Claude Desktop, Cursor) to manage GPU resources, trigger model hot-swaps, measure page-cache residency, read persisted performance analytics, drive the thermal governor, and run overclock sweeps.
• 6 Live MCP Resources: Live metrics, model catalog, switch log, measured cache residency, per-profile analytics, and the overclock profiles with the evidence behind each setting.
• Dual Transport Support: Run via standard input/output (--stdio) or network Server-Sent Events (--sse --port 8001).

⏱️ Automated Latency & Throughput Benchmark Engine

• Conducts automated round-trip model switching benchmarks to measure transition latency, model load time, tokens per second, and RAM cache effectiveness.

---

2. Architectural Overview

flowchart TD
    subgraph HostRAM["64 GB Host System RAM (Page Cache & Staging Buffer)"]
        OllamaGGUFs["Ollama GGUF Weights<br/>(Qwen, Gemma, Nemotron)"]
        ComfySafetensors["ComfyUI Safetensors & VAEs<br/>(Wan2.1, Flux, SDXL)"]
    end

    subgraph GPU["NVIDIA GeForce RTX 4080 SUPER (16 GB VRAM)"]
        direction LR
        ActiveLLM["Active LLM<br/>(0–15 GB VRAM)"]
        ActiveDiffusion["Active Diffusion Pipeline<br/>(0–15 GB VRAM)"]
    end

    subgraph Orchestrator["HyperSwap Control Plane (:9090)"]
        REST["REST API & OpenAPI Docs"]
        MCP["Model Context Protocol (MCP 2.0)"]
        SSE["1Hz Real-Time SSE Stream"]
        Arbitrator["VRAM Arbitrator (confirmed yield)"]
        Overclock["Overclock & Fan Manager"]
        Warmer["Page Cache Pre-Warmer"]
    end

    HostRAM <== "PCIe 4.0 x16 Bus (measured 2.6 GB/s warm model load)" ==> GPU
    Orchestrator --> GPU
    Orchestrator --> HostRAM

The Physics of Sub-Second Switching

• Host RAM as Staging: Active LLMs and diffusion checkpoints remain resident in the 64GB Linux Page Cache.

• Warm vs cold model loads, measured. The same 12.87 GB model, loaded through Ollama on this box:

  Page-cache residency	Load time	Effective rate
  3.1% (dropped with FADV_DONTNEED)	34.3 s	0.38 GB/s
  100% (force-warmed)	4.9 s	2.63 GB/s

  A 6.9× speedup, and the reason the page cache matters. Note the effective rate is well below the PCIe 4.0 x16 bus rate and below the 6.4 GB/s the page cache itself reads at: Ollama's load_duration also covers host-to-device transfer and model initialisation, not just the file read. Classification thresholds are calibrated against these measured numbers rather than the theoretical bus bandwidth — an earlier 5 GB/s cache-hit bar sat above what a fully warm load can even achieve, so every warm load was misreported as a partial hit.

• Soft-Yielding: Dropping Ollama's VRAM allocation via keep_alive: 0 preserves the weights in host RAM. Measured on this box: the HTTP request returns in ~63 ms, and the driver finishes releasing 14.9 GB ~77 ms after that. HyperSwap waits for the second number before handing VRAM to ComfyUI — the earlier "~15 ms" figure timed the request, not the release.

---

3. REST API Reference

The HyperSwap server runs on port 9090 by default. Interactive OpenAPI/Swagger docs are available at http://localhost:9090/docs.

Telemetry & Hardware Endpoints

Endpoint	Method	Description
/api/stats	GET	Complete unified JSON snapshot of hardware sensors, VRAM breakdown, host RAM, Ollama status, ComfyUI queue, and switch logs.
/api/gpu	GET	NVIDIA GPU sensors (utilization %, temperature, power draw in Watts, fan speeds, clocks, and active PIDs).
/api/memory	GET	Precise /proc/meminfo metrics (Total, Used, OS Page Cache containing models, Free memory).
/api/gpu/fan	GET	Current GPU fan mode (auto vs manual), target speed %, and live fan RPM/PWM status.
/api/gpu/fan	POST	Sets GPU fan speed mode (auto or manual) with target speed % (30–100%).
/api/overclock	GET	Active overclock profile, configured profiles, GPU clock limits, and fan status.
/api/overclock/apply	POST	Applies a named profile (ollama, comfy, balanced).
/api/overclock/profile	POST	Creates or updates an overclock profile configuration.
/api/stream	GET	Server-Sent Events (SSE) stream pushing full telemetry updates at 1Hz (text/event-stream).

Model Orchestration & Hot-Swap Endpoints

Endpoint	Method	Description
/api/switch-model	POST	Hot-swaps the active Ollama LLM in VRAM and tracks transition timing.
/api/free-vram	POST	Soft-yields Ollama VRAM to 0 MB and waits for NVML to confirm the release (?confirm=false to skip). Returns request_ms, confirm_ms and the GB actually freed.
/api/comfy-free	POST	Instructs ComfyUI to purge loaded diffusion weights and VRAM cache.
/api/request-vram	POST	Ollama-priority path: purges ComfyUI immediately if there is not enough free VRAM.
/api/warm-all	POST	Warms the highest-value models into page cache within a byte budget (budget_gb).
/api/warm-plan	GET	Previews what warming would read, in what order, and what it would skip — without doing it.
/api/warm-model	POST	Pre-warms a specific model or file into RAM (blob_only warms weights without touching VRAM).
/api/cache/report	GET	Measured page-cache residency per model file, with the measurement method used for each.
/api/benchmark	POST	Runs an automated back-and-forth model swap benchmark and calculates average latency.

Analytics Endpoints (persisted)

Endpoint	Method	Description
/api/analytics/profiles	GET	Decode throughput per overclock profile, joined with the thermals recorded under it.
/api/analytics/swaps	GET	Aggregated swap/yield/purge latencies, cache-hit split, and per-model throughput.
/api/analytics/timeseries	GET	Downsampled telemetry history for charts that outlive a page refresh.
/api/analytics/models	GET	Recency/frequency model ranking used to prioritise the warm budget.
/api/history?durable=true	GET	Swap history from the persistent store rather than the in-memory ring.
/api/db	GET	Store location, row counts and how many hours of history are held.

Governor & Autotune Endpoints

Endpoint	Method	Description
/api/governor	GET / POST	Current derate level and why; enable/disable, or clear an active derate.
/api/overclock/restore	POST	Drop all clock locks and offsets, restore default power limit and automatic fans.
/api/autotune	GET	Sweep progress, last result, and every recorded autotune step.
/api/autotune/sweep	POST	Sweep a knob against a real workload (workload: ollama decode tok/s, comfy SDXL it/s), verifying the knob moves the hardware first.
/api/autotune/cancel	POST	Stop the current sweep after the step in flight; the profile is restored either way.

---

4. Model Context Protocol (MCP 2.0) Reference

HyperSwap includes a native MCP 2.0 server (mcp_server.py) exposing orchestration and telemetry tools to AI agents.

MCP Tools List

Tool Name	Parameters	Description
get_gpu_status	None	Live NVIDIA GPU hardware telemetry, VRAM breakdown, temps, power, fan %, and PIDs.
get_gpu_fan_status	None	Current GPU fan mode (auto/manual) and target fan percentage.
set_gpu_fan_speed	mode (str), percent (optional int)	Sets fan speed mode (auto|manual) and target PWM % (30–100%).
get_host_memory_status	None	64GB host RAM breakdown, active page cache size, and cache ratio.
switch_ollama_model	model_name (str), keep_alive (str)	Hot-swaps active LLM in VRAM, measures latency (ms) and tokens/sec.
soft_yield_ollama_vram	model_name (optional str)	Yields Ollama VRAM to 0 MB and waits for NVML to confirm the driver actually released it. Returns the request/confirm split.
purge_comfyui_vram	None	Purges loaded diffusion models from ComfyUI pipeline VRAM.
prewarm_all_models_to_ram	None	Warms the highest-value models into page cache within a byte budget, skipping what is already resident.
prewarm_single_model	model_name (optional str), filepath (optional str)	Pre-warms a single GGUF or Safetensors file into RAM.
list_available_models	None	Lists all installed Ollama models and discovered ComfyUI Safetensors on disk.
get_switch_history	limit (int, default 20)	Retrieves recent switch events, millisecond latencies, and RAM hit status.
run_model_switch_benchmark	iterations (int, default 2)	Automated round-trip latency benchmark between installed models.
get_page_cache_residency	include_files (bool)	Measured page-cache residency per model file, with the measurement method used for each.
get_warm_plan	budget_gb (optional float)	Previews what warming would read and skip, ranked by recency/frequency. Does not warm.
request_vram_for_ollama	needed_gb (float)	Purges ComfyUI's checkpoints immediately if VRAM headroom is short, bypassing the idle timer.
get_profile_performance	days (float, default 7)	Measured tok/s and thermals per overclock profile, from persisted history.
get_thermal_governor_status	None	Current derate level, the reason for it, and escalation history.
set_thermal_governor	enabled (optional bool), reset (bool)	Enable/disable the governor, or clear an active derate.
get_overclock_status	None	Active profile, all profiles with their evidence, and which levers this driver honours.
apply_overclock_profile	profile (str)	Apply ollama | comfy | balanced.
restore_stock_gpu_state	None	Drop clock locks and offsets, restore default power limit, return fans to automatic.
run_overclock_sweep	knob, profile, workload, start, stop, repeats, apply_best	Sweep a knob against a real workload and report the fastest stable value. Verifies the knob moves the hardware first. Takes minutes.
get_autotune_status	None	Sweep progress, the last result table, and all recorded autotune steps.

MCP Resources List

• gpu://metrics/live: Real-time snapshot of GPU sensors and RAM page cache.
• gpu://models/catalog: Catalog of all discovered GGUF and Safetensors models.
• gpu://history/switches: Event log of recent model transitions and swap speeds.
• gpu://cache/residency: Measured page-cache residency across every model on disk.
• gpu://analytics/profiles: Measured throughput and thermals per overclock profile.
• gpu://overclock/profiles: Overclock profiles including the measurement behind each setting.

---

MCP Client Configurations

Antigravity Configuration (~/.gemini/antigravity-cli/mcp_config.json)

{
  "mcpServers": {
    "hyperswap": {
      "command": "/home/drjones/comfy-mcp-venv/bin/python",
      "args": ["/home/drjones/unified-model-manager/mcp_server.py", "--stdio"]
    }
  }
}

Claude Desktop Configuration (claude_desktop_config.json)

{
  "mcpServers": {
    "hyperswap": {
      "command": "/home/drjones/comfy-mcp-venv/bin/python",
      "args": ["/home/drjones/unified-model-manager/mcp_server.py", "--stdio"]
    }
  }
}

---

5. Linux Kernel & Host Tuning

To ensure that model weights remain permanently in RAM without kernel eviction:

# Set CPU scaling governor to performance
sudo cpupower frequency-set -g performance

# Configure sysctl optimizations in /etc/sysctl.d/99-hyperswap.conf
cat << 'EOF' | sudo tee /etc/sysctl.d/99-hyperswap.conf
# Retain model file cache aggressively in RAM
vm.vfs_cache_pressure = 50

# Prevent swapping cached models
vm.swappiness = 10

# Support large memory maps for high-parameter models
vm.max_map_count = 1048576

# Flush dirty pages quickly
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
EOF

# Apply sysctl settings immediately
sudo sysctl --system

---

6. Systemd Service Management

The HyperSwap server runs as a systemd service:

# Check service status
systemctl status hyperswap.service

# Restart service
sudo systemctl restart hyperswap.service

# View live telemetry and arbitration logs
journalctl -u hyperswap.service -f

---

7. License

MIT License. Developed for Google Antigravity & High-Throughput Linux AI Deployments.