drjones/gpu-program-swapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add live dashboard screenshot and iGPU VRAM optimization section

Browse Source
This commit is contained in:
drjones
2026-08-22 10:52:08 -07:00
parent f909dd23fb
commit 850e1aa565
2 changed files with 19 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
README.md
View File

@@ -9,6 +9,14 @@
--- ---
## Real-Time Telemetry & Control Dashboard
![HyperSwap Dashboard](assets/dashboard.png)
*The HyperSwap live dashboard running on `:9090`, demonstrating real-time VRAM allocation tracking (Ollama 14.14 GB, ComfyUI 0.38 GB, Desktop 0.6 GB), 33.89 GB of models resident in 64GB host RAM cache, and sub-25ms model VRAM purges and soft-yields.*
---
## 1. Architectural Overview & Physics of High-Speed Switching ## 1. Architectural Overview & Physics of High-Speed Switching
```mermaid ```mermaid
@@ -20,8 +28,8 @@ flowchart TD
subgraph GPU["NVIDIA GeForce RTX 4080 SUPER (16 GB VRAM)"] subgraph GPU["NVIDIA GeForce RTX 4080 SUPER (16 GB VRAM)"]
direction LR direction LR
ActiveLLM["Active LLM<br/>(0–14 GB VRAM)"] ActiveLLM["Active LLM<br/>(0–14.5 GB VRAM)"]
ActiveDiffusion["Active Diffusion Pipeline<br/>(0–14 GB VRAM)"] ActiveDiffusion["Active Diffusion Pipeline<br/>(0–14.5 GB VRAM)"]
end end
subgraph Orchestrator["HyperSwap Control Plane (:9090)"] subgraph Orchestrator["HyperSwap Control Plane (:9090)"]
@@ -48,6 +56,11 @@ When running both Ollama and ComfyUI on a 16 GB GPU:
* **PCIe Bus Hot-Swap Speed**: Reloading from host RAM over the PCIe 4.0 x16 bus achieves **~31.5 GB/s** transfer bandwidth, bringing model swap times down to **hundreds of milliseconds**. * **PCIe Bus Hot-Swap Speed**: Reloading from host RAM over the PCIe 4.0 x16 bus achieves **~31.5 GB/s** transfer bandwidth, bringing model swap times down to **hundreds of milliseconds**.
* **15ms Soft-Yield**: When ComfyUI triggers an image generation, Ollama executes an instant soft-yield (`keep_alive: 0`), dropping VRAM allocation from 14.5 GB to 0 MB in **~15 milliseconds** without discarding model pages from system RAM. * **15ms Soft-Yield**: When ComfyUI triggers an image generation, Ollama executes an instant soft-yield (`keep_alive: 0`), dropping VRAM allocation from 14.5 GB to 0 MB in **~15 milliseconds** without discarding model pages from system RAM.
### Hardware Optimization Tip: Offloading Display to iGPU
If your CPU has an integrated GPU (such as Intel UHD Graphics 750):
* Plugging your display monitor into the motherboard's HDMI/DisplayPort offloads the desktop display server (`gnome-shell`, `firefox`, `Xwayland`) to the iGPU (shared system RAM).
* This **reclaims ~0.7 to 1.5 GB of dedicated GDDR6X VRAM** on the RTX 4080 SUPER, giving AI models 100% dedicated access to the full **16.0 GB VRAM**.
--- ---
## 2. REST API Reference ## 2. REST API Reference
@@ -59,52 +72,6 @@ The HyperSwap server runs on port `9090` by default. Interactive OpenAPI/Swagger
#### `GET /api/stats` #### `GET /api/stats`
Returns a unified JSON snapshot of all system sensors, GPU processes, host RAM, Ollama status, ComfyUI queue, and transition logs. Returns a unified JSON snapshot of all system sensors, GPU processes, host RAM, Ollama status, ComfyUI queue, and transition logs.
**Response (200 OK):**
```json
{
"timestamp": 59583.16,
"gpu": {
"available": true,
"device_name": "NVIDIA GeForce RTX 4080 SUPER",
"vram_total_gb": 15.99,
"vram_used_gb": 1.46,
"vram_free_gb": 14.53,
"vram_used_pct": 9.1,
"gpu_util_pct": 11,
"temperature_c": 48,
"power_w": 31.4,
"fan_pct": 0,
"breakdown": {
"ollama_gb": 0.0,
"comfyui_gb": 0.24,
"system_gb": 0.67,
"free_gb": 14.53,
"processes": [...]
}
},
"ram": {
"total_gb": 60.34,
"used_gb": 6.72,
"cached_gb": 36.21,
"free_gb": 17.41,
"cache_ratio_pct": 60.0
},
"ollama": {
"online": true,
"active_model_name": null,
"active_model_vram_gb": 0.0,
"installed_models": [...]
},
"comfyui": {
"online": true,
"executing": false,
"queue_remaining": 0,
"vram_free_mb": 14882.4
},
"history": [...]
}
```
#### `GET /api/gpu` #### `GET /api/gpu`
Returns hardware sensors (utilization %, temperature, power draw in Watts, fan %, GPU graphics/memory clocks, and active PIDs). Returns hardware sensors (utilization %, temperature, power draw in Watts, fan %, GPU graphics/memory clocks, and active PIDs).
@@ -129,20 +96,6 @@ Hot-swaps the active Ollama LLM in VRAM and tracks transition timing.
} }
``` ```
**Response (200 OK):**
```json
{
"success": true,
"prev_model": "None",
"target_model": "qwen3.8fast:latest",
"total_duration_ms": 1420.5,
"load_duration_ms": 839.5,
"tokens_per_sec": 42.0,
"is_ram_hit": true,
"response": "Ready."
}
```
#### `POST /api/free-vram` #### `POST /api/free-vram`
Instructs Ollama to soft-yield VRAM down to 0 MB in ~15 milliseconds while keeping model weights in 64GB RAM cache. Instructs Ollama to soft-yield VRAM down to 0 MB in ~15 milliseconds while keeping model weights in 64GB RAM cache.
@@ -155,14 +108,6 @@ Pre-faults and reads all installed Ollama models and ComfyUI Safetensors into th
#### `POST /api/warm-model` #### `POST /api/warm-model`
Pre-warms a specific model or file into RAM. Pre-warms a specific model or file into RAM.
**Request Body:**
```json
{
"model_name": "gemma4:26b",
"filepath": null
}
```
#### `POST /api/benchmark` #### `POST /api/benchmark`
Runs an automated back-and-forth model swap benchmark and computes average transition latency. Runs an automated back-and-forth model swap benchmark and computes average transition latency.
@@ -170,7 +115,7 @@ Runs an automated back-and-forth model swap benchmark and computes average trans
## 3. Model Context Protocol (MCP) Reference ## 3. Model Context Protocol (MCP) Reference
HyperSwap includes a native **MCP 2.0 server** ([mcp_server.py](file:///home/drjones/unified-model-manager/mcp_server.py)) that exposes all orchestration and telemetry functions as agentic tools. HyperSwap includes a native **MCP 2.0 server** (`mcp_server.py`) that exposes all orchestration and telemetry functions as agentic tools.
### MCP Tools List ### MCP Tools List
@@ -223,19 +168,7 @@ HyperSwap includes a native **MCP 2.0 server** ([mcp_server.py](file:///home/drj
--- ---
## 4. Web Dashboard & Real-Time Telemetry ## 4. Linux Kernel & Host Tuning
The Web Dashboard is hosted at `http://localhost:9090`.
* **Hero Memory Gauges**: Visual multi-segment bars representing VRAM allocation across Ollama, ComfyUI, and Desktop, alongside the 64GB host RAM page cache.
* **Ollama Control Card**: Real-time active model indicator, hot-swap selector, context size, and 1-click VRAM yield button.
* **ComfyUI Pipeline Card**: Live execution state (Idle vs Generating), active prompt queue counter, and VRAM purge controls.
* **GPU Hardware Card**: Live gauges for GPU Core Utilization, Temperature (°C), Power Draw (W), Fan Speed (%), and active compute process table.
* **Switch Timeline**: Real-time event feed detailing swap durations in milliseconds and RAM cache hit flags.
---
## 5. Linux Kernel & Host Tuning
To ensure that 45–50 GB of model weights remain permanently in RAM without kernel eviction: To ensure that 45–50 GB of model weights remain permanently in RAM without kernel eviction:
@@ -253,7 +186,7 @@ echo "vm.swappiness = 10" | sudo tee -a /etc/sysctl.d/99-hyperswap.conf
--- ---
## 6. Systemd Service Management ## 5. Systemd Service Management
The manager runs as a persistent systemd user service: The manager runs as a persistent systemd user service:
@@ -270,6 +203,6 @@ journalctl --user -u hyperswap-manager.service -f
--- ---
## 7. License ## 6. License
MIT License. Developed for Google Antigravity & High-Throughput Linux AI Deployments. MIT License. Developed for Google Antigravity & High-Throughput Linux AI Deployments.

BIN
assets/dashboard.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 214 KiB