gpu-program-swapper/README.md

HyperSwap // GPU Program Swapper & Memory Orchestrator

HyperSwap is an ultra-low-latency VRAM arbitrator, RAM cache pre-warmer, and real-time telemetry dashboard designed specifically for Linux deployment machines that simultaneously host Ollama LLM workloads and ComfyUI Diffusion pipelines on a single GPU.

---

Real-Time Telemetry & Control Dashboard

The HyperSwap live dashboard running on :9090, demonstrating real-time VRAM allocation tracking (Ollama 14.14 GB, ComfyUI 0.38 GB, Desktop 0.6 GB), 33.89 GB of models resident in 64GB host RAM cache, and sub-25ms model VRAM purges and soft-yields.

---

1. Architectural Overview & Physics of High-Speed Switching

flowchart TD
    subgraph HostRAM["64 GB DDR5 Host System RAM (Page Cache & Pinned Staging)"]
        OllamaGGUFs["Ollama GGUF Weights<br/>(Qwen, Gemma, Nemotron)"]
        ComfySafetensors["ComfyUI Safetensors & VAEs<br/>(53.6 GB Pinned Staging Buffer)"]
    end

    subgraph GPU["NVIDIA GeForce RTX 4080 SUPER (16 GB VRAM)"]
        direction LR
        ActiveLLM["Active LLM<br/>(0–14.5 GB VRAM)"]
        ActiveDiffusion["Active Diffusion Pipeline<br/>(0–14.5 GB VRAM)"]
    end

    subgraph Orchestrator["HyperSwap Control Plane (:9090)"]
        REST["REST API & OpenAPI Docs"]
        MCP["Model Context Protocol (MCP 2.0)"]
        SSE["1Hz Real-Time SSE Stream"]
        Arbitrator["VRAM Arbitrator (15ms Soft-Yield)"]
        Warmer["Page Cache Pre-Warmer"]
    end

    HostRAM <== "PCIe 4.0 x16 Bus (~31.5 GB/s Hot-Swap)" ==> GPU
    Orchestrator --> GPU
    Orchestrator --> HostRAM

The Problem: Disk Bottleneck & VRAM Contention

When running both Ollama and ComfyUI on a 16 GB GPU:

• An active LLM (e.g. 27B–30B parameter quantized model) uses 11–15 GB VRAM.
• A diffusion model (SDXL, Flux, SD 1.5) requires 4–14 GB VRAM during generation.
• If models are evicted to NVMe storage, reloading weights takes 10–40 seconds over disk I/O.

The Solution: 64 GB RAM Cache + PCIe x16 Hot-Swapping

• Host RAM as Staging: All active LLMs and diffusion checkpoints remain 100% resident in the 64 GB Linux OS Page Cache and pinned memory buffer.
• PCIe Bus Hot-Swap Speed: Reloading from host RAM over the PCIe 4.0 x16 bus achieves ~31.5 GB/s transfer bandwidth, bringing model swap times down to hundreds of milliseconds.
• 15ms Soft-Yield: When ComfyUI triggers an image generation, Ollama executes an instant soft-yield (keep_alive: 0), dropping VRAM allocation from 14.5 GB to 0 MB in ~15 milliseconds without discarding model pages from system RAM.

Hardware Optimization Tip: Offloading Display to iGPU

If your CPU has an integrated GPU (such as Intel UHD Graphics 750):

• Plugging your display monitor into the motherboard's HDMI/DisplayPort offloads the desktop display server (gnome-shell, firefox, Xwayland) to the iGPU (shared system RAM).
• This reclaims ~0.7 to 1.5 GB of dedicated GDDR6X VRAM on the RTX 4080 SUPER, giving AI models 100% dedicated access to the full 16.0 GB VRAM.

---

2. REST API Reference

The HyperSwap server runs on port 9090 by default. Interactive OpenAPI/Swagger docs are accessible at http://localhost:9090/docs.

Telemetry Endpoints

GET /api/stats

Returns a unified JSON snapshot of all system sensors, GPU processes, host RAM, Ollama status, ComfyUI queue, and transition logs.

GET /api/gpu

Returns hardware sensors (utilization %, temperature, power draw in Watts, fan %, GPU graphics/memory clocks, and active PIDs).

GET /api/memory

Returns precise /proc/meminfo metrics including Total, Used, OS Page Cache, and free memory.

GET /api/stream

Server-Sent Events (SSE) stream pushing full telemetry updates at 1Hz (Content-Type: text/event-stream).

---

Orchestration & Hot-Swap Endpoints

POST /api/switch-model

Hot-swaps the active Ollama LLM in VRAM and tracks transition timing.

Request Body:

{
  "model": "qwen3.8fast:latest",
  "keep_alive": "30m"
}

POST /api/free-vram

Instructs Ollama to soft-yield VRAM down to 0 MB in ~15 milliseconds while keeping model weights in 64GB RAM cache.

POST /api/comfy-free

Instructs ComfyUI to purge loaded diffusion weights and VRAM cache.

POST /api/warm-all

Pre-faults and reads all installed Ollama models and ComfyUI Safetensors into the Linux page cache.

POST /api/warm-model

Pre-warms a specific model or file into RAM.

POST /api/benchmark

Runs an automated back-and-forth model swap benchmark and computes average transition latency.

---

3. Model Context Protocol (MCP) Reference

HyperSwap includes a native MCP 2.0 server (mcp_server.py) that exposes all orchestration and telemetry functions as agentic tools.

MCP Tools List

Tool Name	Parameters	Description
get_gpu_status	None	Live NVIDIA GPU hardware telemetry, VRAM breakdown, temps, power, and PIDs.
get_host_memory_status	None	64GB host RAM breakdown, active page cache size, and cache ratio.
switch_ollama_model	model_name (str), keep_alive (str, default "30m")	Hot-swaps active LLM in VRAM, measures latency (ms) and tokens/sec.
soft_yield_ollama_vram	model_name (optional str)	Yields Ollama VRAM to 0 MB in ~15ms while keeping model weights in RAM cache.
purge_comfyui_vram	None	Purges loaded diffusion models from ComfyUI pipeline VRAM.
prewarm_all_models_to_ram	None	Faults all local LLM and diffusion checkpoints into Linux OS page cache.
prewarm_single_model	model_name (optional str), filepath (optional str)	Pre-warms a single GGUF or Safetensors file into RAM.
list_available_models	None	Lists all installed Ollama models and ComfyUI Safetensors on disk.
get_switch_history	limit (int, default 20)	Retrieves recent switch events, millisecond latencies, and RAM hit status.
run_model_switch_benchmark	iterations (int, default 2)	Automated round-trip latency benchmark between installed models.

MCP Resources List

• gpu://metrics/live - Real-time snapshot of GPU sensors and RAM page cache.
• gpu://models/catalog - Catalog of all discovered GGUF and Safetensors models.
• gpu://history/switches - Event log of recent model transitions and swap speeds.

---

MCP Client Configurations

Antigravity Configuration (~/.gemini/config/mcp_config.json)

{
  "mcpServers": {
    "hyperswap": {
      "command": "/home/drjones/comfy-mcp-venv/bin/python",
      "args": ["/home/drjones/unified-model-manager/mcp_server.py", "--stdio"]
    }
  }
}

Claude Desktop Configuration (claude_desktop_config.json)

{
  "mcpServers": {
    "hyperswap": {
      "command": "/home/drjones/comfy-mcp-venv/bin/python",
      "args": ["/home/drjones/unified-model-manager/mcp_server.py", "--stdio"]
    }
  }
}

---

4. Linux Kernel & Host Tuning

To ensure that 45–50 GB of model weights remain permanently in RAM without kernel eviction:

# Prioritize retaining model file cache in RAM (lower pressure = stronger cache retention)
sudo sysctl -w vm.vfs_cache_pressure=10

# Reduce swap aggression for active pages
sudo sysctl -w vm.swappiness=10

# Write changes permanently to /etc/sysctl.d/99-hyperswap.conf
echo "vm.vfs_cache_pressure = 10" | sudo tee /etc/sysctl.d/99-hyperswap.conf
echo "vm.swappiness = 10" | sudo tee -a /etc/sysctl.d/99-hyperswap.conf

---

5. Systemd Service Management

The manager runs as a persistent systemd user service:

# Check status
systemctl --user status hyperswap-manager.service

# Restart service
systemctl --user restart hyperswap-manager.service

# View live logs
journalctl --user -u hyperswap-manager.service -f

---

6. License

MIT License. Developed for Google Antigravity & High-Throughput Linux AI Deployments.