Initial commit: HyperSwap GPU Program Swapper with REST API, MCP 2.0, Real-Time Dashboard and Memory Orchestrator
This commit is contained in:
drjones
7
.gitignore
vendored
Normal file
7
.gitignore
vendored
Normal file
@@ -0,0 +1,7 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.log
|
||||
.venv/
|
||||
venv/
|
||||
.DS_Store
|
||||
275
README.md
Normal file
275
README.md
Normal file
@@ -0,0 +1,275 @@
|
||||
# HyperSwap // GPU Program Swapper & Memory Orchestrator
|
||||
|
||||
[](https://fastapi.tiangolo.com)
|
||||
[](https://modelcontextprotocol.io)
|
||||
[](https://developer.nvidia.com/cuda-zone)
|
||||
[](https://ubuntu.com)
|
||||
|
||||
**HyperSwap** is an ultra-low-latency VRAM arbitrator, RAM cache pre-warmer, and real-time telemetry dashboard designed specifically for Linux deployment machines that simultaneously host **Ollama LLM workloads** and **ComfyUI Diffusion pipelines** on a single GPU.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architectural Overview & Physics of High-Speed Switching
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph HostRAM["64 GB DDR5 Host System RAM (Page Cache & Pinned Staging)"]
|
||||
OllamaGGUFs["Ollama GGUF Weights<br/>(Qwen, Gemma, Nemotron)"]
|
||||
ComfySafetensors["ComfyUI Safetensors & VAEs<br/>(53.6 GB Pinned Staging Buffer)"]
|
||||
end
|
||||
|
||||
subgraph GPU["NVIDIA GeForce RTX 4080 SUPER (16 GB VRAM)"]
|
||||
direction LR
|
||||
ActiveLLM["Active LLM<br/>(0–14 GB VRAM)"]
|
||||
ActiveDiffusion["Active Diffusion Pipeline<br/>(0–14 GB VRAM)"]
|
||||
end
|
||||
|
||||
subgraph Orchestrator["HyperSwap Control Plane (:9090)"]
|
||||
REST["REST API & OpenAPI Docs"]
|
||||
MCP["Model Context Protocol (MCP 2.0)"]
|
||||
SSE["1Hz Real-Time SSE Stream"]
|
||||
Arbitrator["VRAM Arbitrator (15ms Soft-Yield)"]
|
||||
Warmer["Page Cache Pre-Warmer"]
|
||||
end
|
||||
|
||||
HostRAM <== "PCIe 4.0 x16 Bus (~31.5 GB/s Hot-Swap)" ==> GPU
|
||||
Orchestrator --> GPU
|
||||
Orchestrator --> HostRAM
|
||||
```
|
||||
|
||||
### The Problem: Disk Bottleneck & VRAM Contention
|
||||
When running both Ollama and ComfyUI on a 16 GB GPU:
|
||||
* An active LLM (e.g. 27B–30B parameter quantized model) uses **11–15 GB VRAM**.
|
||||
* A diffusion model (SDXL, Flux, SD 1.5) requires **4–14 GB VRAM** during generation.
|
||||
* If models are evicted to NVMe storage, reloading weights takes **10–40 seconds** over disk I/O.
|
||||
|
||||
### The Solution: 64 GB RAM Cache + PCIe x16 Hot-Swapping
|
||||
* **Host RAM as Staging**: All active LLMs and diffusion checkpoints remain 100% resident in the 64 GB Linux OS Page Cache and pinned memory buffer.
|
||||
* **PCIe Bus Hot-Swap Speed**: Reloading from host RAM over the PCIe 4.0 x16 bus achieves **~31.5 GB/s** transfer bandwidth, bringing model swap times down to **hundreds of milliseconds**.
|
||||
* **15ms Soft-Yield**: When ComfyUI triggers an image generation, Ollama executes an instant soft-yield (`keep_alive: 0`), dropping VRAM allocation from 14.5 GB to 0 MB in **~15 milliseconds** without discarding model pages from system RAM.
|
||||
|
||||
---
|
||||
|
||||
## 2. REST API Reference
|
||||
|
||||
The HyperSwap server runs on port `9090` by default. Interactive OpenAPI/Swagger docs are accessible at `http://localhost:9090/docs`.
|
||||
|
||||
### Telemetry Endpoints
|
||||
|
||||
#### `GET /api/stats`
|
||||
Returns a unified JSON snapshot of all system sensors, GPU processes, host RAM, Ollama status, ComfyUI queue, and transition logs.
|
||||
|
||||
**Response (200 OK):**
|
||||
```json
|
||||
{
|
||||
"timestamp": 59583.16,
|
||||
"gpu": {
|
||||
"available": true,
|
||||
"device_name": "NVIDIA GeForce RTX 4080 SUPER",
|
||||
"vram_total_gb": 15.99,
|
||||
"vram_used_gb": 1.46,
|
||||
"vram_free_gb": 14.53,
|
||||
"vram_used_pct": 9.1,
|
||||
"gpu_util_pct": 11,
|
||||
"temperature_c": 48,
|
||||
"power_w": 31.4,
|
||||
"fan_pct": 0,
|
||||
"breakdown": {
|
||||
"ollama_gb": 0.0,
|
||||
"comfyui_gb": 0.24,
|
||||
"system_gb": 0.67,
|
||||
"free_gb": 14.53,
|
||||
"processes": [...]
|
||||
}
|
||||
},
|
||||
"ram": {
|
||||
"total_gb": 60.34,
|
||||
"used_gb": 6.72,
|
||||
"cached_gb": 36.21,
|
||||
"free_gb": 17.41,
|
||||
"cache_ratio_pct": 60.0
|
||||
},
|
||||
"ollama": {
|
||||
"online": true,
|
||||
"active_model_name": null,
|
||||
"active_model_vram_gb": 0.0,
|
||||
"installed_models": [...]
|
||||
},
|
||||
"comfyui": {
|
||||
"online": true,
|
||||
"executing": false,
|
||||
"queue_remaining": 0,
|
||||
"vram_free_mb": 14882.4
|
||||
},
|
||||
"history": [...]
|
||||
}
|
||||
```
|
||||
|
||||
#### `GET /api/gpu`
|
||||
Returns hardware sensors (utilization %, temperature, power draw in Watts, fan %, GPU graphics/memory clocks, and active PIDs).
|
||||
|
||||
#### `GET /api/memory`
|
||||
Returns precise `/proc/meminfo` metrics including Total, Used, OS Page Cache, and free memory.
|
||||
|
||||
#### `GET /api/stream`
|
||||
Server-Sent Events (SSE) stream pushing full telemetry updates at 1Hz (`Content-Type: text/event-stream`).
|
||||
|
||||
---
|
||||
|
||||
### Orchestration & Hot-Swap Endpoints
|
||||
|
||||
#### `POST /api/switch-model`
|
||||
Hot-swaps the active Ollama LLM in VRAM and tracks transition timing.
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"model": "qwen3.8fast:latest",
|
||||
"keep_alive": "30m"
|
||||
}
|
||||
```
|
||||
|
||||
**Response (200 OK):**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"prev_model": "None",
|
||||
"target_model": "qwen3.8fast:latest",
|
||||
"total_duration_ms": 1420.5,
|
||||
"load_duration_ms": 839.5,
|
||||
"tokens_per_sec": 42.0,
|
||||
"is_ram_hit": true,
|
||||
"response": "Ready."
|
||||
}
|
||||
```
|
||||
|
||||
#### `POST /api/free-vram`
|
||||
Instructs Ollama to soft-yield VRAM down to 0 MB in ~15 milliseconds while keeping model weights in 64GB RAM cache.
|
||||
|
||||
#### `POST /api/comfy-free`
|
||||
Instructs ComfyUI to purge loaded diffusion weights and VRAM cache.
|
||||
|
||||
#### `POST /api/warm-all`
|
||||
Pre-faults and reads all installed Ollama models and ComfyUI Safetensors into the Linux page cache.
|
||||
|
||||
#### `POST /api/warm-model`
|
||||
Pre-warms a specific model or file into RAM.
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"model_name": "gemma4:26b",
|
||||
"filepath": null
|
||||
}
|
||||
```
|
||||
|
||||
#### `POST /api/benchmark`
|
||||
Runs an automated back-and-forth model swap benchmark and computes average transition latency.
|
||||
|
||||
---
|
||||
|
||||
## 3. Model Context Protocol (MCP) Reference
|
||||
|
||||
HyperSwap includes a native **MCP 2.0 server** ([mcp_server.py](file:///home/drjones/unified-model-manager/mcp_server.py)) that exposes all orchestration and telemetry functions as agentic tools.
|
||||
|
||||
### MCP Tools List
|
||||
|
||||
| Tool Name | Parameters | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| **`get_gpu_status`** | *None* | Live NVIDIA GPU hardware telemetry, VRAM breakdown, temps, power, and PIDs. |
|
||||
| **`get_host_memory_status`** | *None* | 64GB host RAM breakdown, active page cache size, and cache ratio. |
|
||||
| **`switch_ollama_model`** | `model_name` (str), `keep_alive` (str, default "30m") | Hot-swaps active LLM in VRAM, measures latency (ms) and tokens/sec. |
|
||||
| **`soft_yield_ollama_vram`** | `model_name` (optional str) | Yields Ollama VRAM to 0 MB in ~15ms while keeping model weights in RAM cache. |
|
||||
| **`purge_comfyui_vram`** | *None* | Purges loaded diffusion models from ComfyUI pipeline VRAM. |
|
||||
| **`prewarm_all_models_to_ram`** | *None* | Faults all local LLM and diffusion checkpoints into Linux OS page cache. |
|
||||
| **`prewarm_single_model`** | `model_name` (optional str), `filepath` (optional str) | Pre-warms a single GGUF or Safetensors file into RAM. |
|
||||
| **`list_available_models`** | *None* | Lists all installed Ollama models and ComfyUI Safetensors on disk. |
|
||||
| **`get_switch_history`** | `limit` (int, default 20) | Retrieves recent switch events, millisecond latencies, and RAM hit status. |
|
||||
| **`run_model_switch_benchmark`**| `iterations` (int, default 2) | Automated round-trip latency benchmark between installed models. |
|
||||
|
||||
### MCP Resources List
|
||||
|
||||
* `gpu://metrics/live` - Real-time snapshot of GPU sensors and RAM page cache.
|
||||
* `gpu://models/catalog` - Catalog of all discovered GGUF and Safetensors models.
|
||||
* `gpu://history/switches` - Event log of recent model transitions and swap speeds.
|
||||
|
||||
---
|
||||
|
||||
### MCP Client Configurations
|
||||
|
||||
#### Antigravity Configuration (`~/.gemini/config/mcp_config.json`)
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"hyperswap": {
|
||||
"command": "/home/drjones/comfy-mcp-venv/bin/python",
|
||||
"args": ["/home/drjones/unified-model-manager/mcp_server.py", "--stdio"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Claude Desktop Configuration (`claude_desktop_config.json`)
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"hyperswap": {
|
||||
"command": "/home/drjones/comfy-mcp-venv/bin/python",
|
||||
"args": ["/home/drjones/unified-model-manager/mcp_server.py", "--stdio"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Web Dashboard & Real-Time Telemetry
|
||||
|
||||
The Web Dashboard is hosted at `http://localhost:9090`.
|
||||
|
||||
* **Hero Memory Gauges**: Visual multi-segment bars representing VRAM allocation across Ollama, ComfyUI, and Desktop, alongside the 64GB host RAM page cache.
|
||||
* **Ollama Control Card**: Real-time active model indicator, hot-swap selector, context size, and 1-click VRAM yield button.
|
||||
* **ComfyUI Pipeline Card**: Live execution state (Idle vs Generating), active prompt queue counter, and VRAM purge controls.
|
||||
* **GPU Hardware Card**: Live gauges for GPU Core Utilization, Temperature (°C), Power Draw (W), Fan Speed (%), and active compute process table.
|
||||
* **Switch Timeline**: Real-time event feed detailing swap durations in milliseconds and RAM cache hit flags.
|
||||
|
||||
---
|
||||
|
||||
## 5. Linux Kernel & Host Tuning
|
||||
|
||||
To ensure that 45–50 GB of model weights remain permanently in RAM without kernel eviction:
|
||||
|
||||
```bash
|
||||
# Prioritize retaining model file cache in RAM (lower pressure = stronger cache retention)
|
||||
sudo sysctl -w vm.vfs_cache_pressure=10
|
||||
|
||||
# Reduce swap aggression for active pages
|
||||
sudo sysctl -w vm.swappiness=10
|
||||
|
||||
# Write changes permanently to /etc/sysctl.d/99-hyperswap.conf
|
||||
echo "vm.vfs_cache_pressure = 10" | sudo tee /etc/sysctl.d/99-hyperswap.conf
|
||||
echo "vm.swappiness = 10" | sudo tee -a /etc/sysctl.d/99-hyperswap.conf
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Systemd Service Management
|
||||
|
||||
The manager runs as a persistent systemd user service:
|
||||
|
||||
```bash
|
||||
# Check status
|
||||
systemctl --user status hyperswap-manager.service
|
||||
|
||||
# Restart service
|
||||
systemctl --user restart hyperswap-manager.service
|
||||
|
||||
# View live logs
|
||||
journalctl --user -u hyperswap-manager.service -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. License
|
||||
|
||||
MIT License. Developed for Google Antigravity & High-Throughput Linux AI Deployments.
|
||||
153
mcp_server.py
Normal file
153
mcp_server.py
Normal file
@@ -0,0 +1,153 @@
|
||||
"""Model Context Protocol (MCP) Server for GPU Program Swapper and Memory Orchestrator."""
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Dict, List, Any, Optional
|
||||
|
||||
from mcp.server import MCPServer
|
||||
import ram_optimizer
|
||||
import vram_arbitrator
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s")
|
||||
logger = logging.getLogger("gpu_swapper_mcp")
|
||||
|
||||
mcp = MCPServer(
|
||||
name="gpu-program-swapper",
|
||||
version="1.0.0",
|
||||
description="Orchestrates high-speed GPU VRAM hot-swaps between Ollama LLMs and ComfyUI with 64GB RAM cache telemetry."
|
||||
)
|
||||
|
||||
# ==========================================
|
||||
# MCP TOOLS
|
||||
# ==========================================
|
||||
|
||||
@mcp.tool()
|
||||
def get_gpu_status() -> str:
|
||||
"""Get live NVIDIA GPU hardware telemetry, VRAM allocation (Ollama vs ComfyUI vs System), temperature, power, and active compute PIDs."""
|
||||
stats = vram_arbitrator.get_gpu_hardware_stats()
|
||||
return json.dumps(stats, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
def get_host_memory_status() -> str:
|
||||
"""Get host system RAM breakdown (Total, Apps Used, OS Page Cache containing models in RAM, Free memory) and cache hit percentage."""
|
||||
mem = ram_optimizer.get_detailed_meminfo()
|
||||
return json.dumps(mem, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
async def switch_ollama_model(model_name: str, keep_alive: str = "30m") -> str:
|
||||
"""Hot-swap active Ollama LLM in VRAM. Measures swap latency (ms), eval speed (tokens/sec), and whether it was an instant RAM cache hit."""
|
||||
res = await vram_arbitrator.switch_ollama_model(model_name, keep_alive=keep_alive)
|
||||
return json.dumps(res, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
async def soft_yield_ollama_vram(model_name: Optional[str] = None) -> str:
|
||||
"""Soft-yield Ollama VRAM down to 0 MB in ~15 milliseconds while keeping all model weights resident in 64GB host RAM cache."""
|
||||
res = await vram_arbitrator.instant_free_ollama_vram(model_name)
|
||||
return json.dumps(res, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
async def purge_comfyui_vram() -> str:
|
||||
"""Purge loaded diffusion models and VRAM cache from the ComfyUI pipeline."""
|
||||
res = await vram_arbitrator.instant_free_comfyui_vram()
|
||||
return json.dumps(res, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
async def prewarm_all_models_to_ram() -> str:
|
||||
"""Pre-read and fault all installed Ollama GGUF models and ComfyUI Safetensors checkpoints into the Linux OS Page Cache for PCIe-speed hot swapping."""
|
||||
res = await ram_optimizer.warm_all_models()
|
||||
return json.dumps(res, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
async def prewarm_single_model(model_name: Optional[str] = None, filepath: Optional[str] = None) -> str:
|
||||
"""Pre-warm a specific Ollama model or individual model file path into Linux RAM cache."""
|
||||
if model_name:
|
||||
res = await ram_optimizer.warm_ollama_model(model_name, keep_alive="1m")
|
||||
elif filepath:
|
||||
res = ram_optimizer.warm_file_to_ram(filepath)
|
||||
else:
|
||||
res = {"error": "Either model_name or filepath must be provided"}
|
||||
return json.dumps(res, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
async def list_available_models() -> str:
|
||||
"""List all installed Ollama models and discovered ComfyUI model checkpoints/safetensors on disk with sizes and quantization levels."""
|
||||
ollama_state = await vram_arbitrator.get_ollama_live_state()
|
||||
comfy_models = ram_optimizer.find_comfy_model_files()
|
||||
result = {
|
||||
"ollama_models": ollama_state.get("installed_models", []),
|
||||
"comfy_models": comfy_models,
|
||||
}
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
def get_switch_history(limit: int = 20) -> str:
|
||||
"""Get the recent history log of model switch events, swap durations (in ms), and RAM cache hit status."""
|
||||
history = vram_arbitrator.get_switch_history()
|
||||
return json.dumps(history[:limit], indent=2)
|
||||
|
||||
@mcp.tool()
|
||||
async def run_model_switch_benchmark(iterations: int = 2) -> str:
|
||||
"""Run an automated benchmark swapping between available models to measure round-trip latency and RAM cache effectiveness."""
|
||||
ollama_state = await vram_arbitrator.get_ollama_live_state()
|
||||
installed = [m.get("name") for m in ollama_state.get("installed_models", []) if m.get("name")]
|
||||
if len(installed) < 2:
|
||||
return json.dumps({"error": "Need at least 2 installed models to benchmark", "installed": installed})
|
||||
|
||||
m1, m2 = installed[0], installed[1]
|
||||
results = []
|
||||
|
||||
for i in range(iterations):
|
||||
# Swap to m1
|
||||
r1 = await vram_arbitrator.switch_ollama_model(m1, keep_alive="5m")
|
||||
results.append({"iteration": i+1, "direction": f"-> {m1}", "latency_ms": r1.get("total_duration_ms", 0), "load_ms": r1.get("load_duration_ms", 0), "is_ram_hit": r1.get("is_ram_hit", False)})
|
||||
await asyncio.sleep(0.5)
|
||||
# Swap to m2
|
||||
r2 = await vram_arbitrator.switch_ollama_model(m2, keep_alive="5m")
|
||||
results.append({"iteration": i+1, "direction": f"-> {m2}", "latency_ms": r2.get("total_duration_ms", 0), "load_ms": r2.get("load_duration_ms", 0), "is_ram_hit": r2.get("is_ram_hit", False)})
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
avg_latency = round(sum(r["latency_ms"] for r in results) / len(results), 2) if results else 0
|
||||
return json.dumps({
|
||||
"status": "completed",
|
||||
"tested_models": [m1, m2],
|
||||
"iterations": iterations,
|
||||
"average_swap_latency_ms": avg_latency,
|
||||
"rounds": results,
|
||||
}, indent=2)
|
||||
|
||||
# ==========================================
|
||||
# MCP RESOURCES
|
||||
# ==========================================
|
||||
|
||||
@mcp.resource("gpu://metrics/live")
|
||||
def get_live_metrics_resource() -> str:
|
||||
"""Live snapshot of GPU hardware sensors, VRAM, and RAM cache."""
|
||||
gpu = vram_arbitrator.get_gpu_hardware_stats()
|
||||
ram = ram_optimizer.get_detailed_meminfo()
|
||||
return json.dumps({"gpu": gpu, "ram": ram}, indent=2)
|
||||
|
||||
@mcp.resource("gpu://models/catalog")
|
||||
async def get_models_catalog_resource() -> str:
|
||||
"""Catalog of all local Ollama and ComfyUI models."""
|
||||
return await list_available_models()
|
||||
|
||||
@mcp.resource("gpu://history/switches")
|
||||
def get_switch_history_resource() -> str:
|
||||
"""Recent model switch events and latencies."""
|
||||
return json.dumps(vram_arbitrator.get_switch_history(), indent=2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description="HyperSwap GPU Program Swapper MCP Server")
|
||||
parser.add_argument("--stdio", action="store_true", help="Run in stdio mode (default)")
|
||||
parser.add_argument("--sse", action="store_true", help="Run with SSE transport")
|
||||
parser.add_argument("--port", type=int, default=8001, help="Port for SSE transport")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.sse:
|
||||
mcp.run(transport="sse", host="0.0.0.0", port=args.port)
|
||||
else:
|
||||
mcp.run(transport="stdio")
|
||||
176
ram_optimizer.py
Normal file
176
ram_optimizer.py
Normal file
@@ -0,0 +1,176 @@
|
||||
"""RAM Optimizer and Model Pre-warmer for High-Speed Switching."""
|
||||
import os
|
||||
import glob
|
||||
import time
|
||||
import httpx
|
||||
import logging
|
||||
from typing import Dict, List, Any
|
||||
|
||||
logger = logging.getLogger("ram_optimizer")
|
||||
|
||||
OLLAMA_API_BASE = "http://localhost:11434"
|
||||
COMFY_API_BASE = "http://127.0.0.1:8188"
|
||||
COMFY_MODELS_DIR = "/home/drjones/ComfyUI/models"
|
||||
|
||||
def get_detailed_meminfo() -> Dict[str, Any]:
|
||||
"""Parse /proc/meminfo for precise page cache and RAM stats."""
|
||||
info = {}
|
||||
try:
|
||||
with open("/proc/meminfo", "r") as f:
|
||||
for line in f:
|
||||
parts = line.split(":")
|
||||
if len(parts) == 2:
|
||||
key = parts[0].strip()
|
||||
val = parts[1].strip().split()[0]
|
||||
info[key] = int(val) * 1024 # Convert kB to bytes
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read /proc/meminfo: {e}")
|
||||
|
||||
total = info.get("MemTotal", 0)
|
||||
free = info.get("MemFree", 0)
|
||||
available = info.get("MemAvailable", 0)
|
||||
cached = info.get("Cached", 0) + info.get("Buffers", 0)
|
||||
dirty = info.get("Dirty", 0)
|
||||
used = total - free - cached
|
||||
if used < 0:
|
||||
used = total - available
|
||||
|
||||
return {
|
||||
"total_bytes": total,
|
||||
"total_gb": round(total / (1024**3), 2),
|
||||
"used_bytes": used,
|
||||
"used_gb": round(used / (1024**3), 2),
|
||||
"cached_bytes": cached,
|
||||
"cached_gb": round(cached / (1024**3), 2),
|
||||
"free_bytes": free,
|
||||
"free_gb": round(free / (1024**3), 2),
|
||||
"available_bytes": available,
|
||||
"available_gb": round(available / (1024**3), 2),
|
||||
"dirty_bytes": dirty,
|
||||
"dirty_mb": round(dirty / (1024**2), 2),
|
||||
"cache_ratio_pct": round((cached / total * 100) if total > 0 else 0, 1),
|
||||
}
|
||||
|
||||
def warm_file_to_ram(filepath: str, chunk_size: int = 16 * 1024 * 1024) -> Dict[str, Any]:
|
||||
"""Pre-fault/read file into Linux OS Page Cache at maximum disk read speed."""
|
||||
if not os.path.exists(filepath):
|
||||
return {"success": False, "error": f"File not found: {filepath}", "duration_ms": 0}
|
||||
|
||||
t0 = time.perf_counter()
|
||||
file_size = os.path.getsize(filepath)
|
||||
bytes_read = 0
|
||||
try:
|
||||
with open(filepath, "rb") as f:
|
||||
# Hint kernel that we will read this sequentially
|
||||
try:
|
||||
os.posix_fadvise(f.fileno(), 0, file_size, os.POSIX_FADV_WILLNEED)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
buf = bytearray(chunk_size)
|
||||
while True:
|
||||
n = f.readinto(buf)
|
||||
if not n:
|
||||
break
|
||||
bytes_read += n
|
||||
|
||||
duration = time.perf_counter() - t0
|
||||
duration_ms = round(duration * 1000, 2)
|
||||
speed_mb_s = round((bytes_read / (1024**2)) / duration if duration > 0 else 0, 2)
|
||||
return {
|
||||
"success": True,
|
||||
"filepath": filepath,
|
||||
"size_bytes": file_size,
|
||||
"size_mb": round(file_size / (1024**2), 2),
|
||||
"bytes_read": bytes_read,
|
||||
"duration_ms": duration_ms,
|
||||
"speed_mb_s": speed_mb_s,
|
||||
}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
async def warm_ollama_model(model_name: str, keep_alive: str = "5m") -> Dict[str, Any]:
|
||||
"""Warm an Ollama model into memory and measure time."""
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
resp = await client.post(
|
||||
f"{OLLAMA_API_BASE}/api/generate",
|
||||
json={"model": model_name, "prompt": "", "keep_alive": keep_alive},
|
||||
)
|
||||
duration = time.perf_counter() - t0
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
return {
|
||||
"success": True,
|
||||
"model": model_name,
|
||||
"duration_ms": round(duration * 1000, 2),
|
||||
"load_duration_ms": round(data.get("load_duration", 0) / 1e6, 2),
|
||||
"total_duration_ms": round(data.get("total_duration", 0) / 1e6, 2),
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"success": False,
|
||||
"model": model_name,
|
||||
"error": f"HTTP {resp.status_code}: {resp.text}",
|
||||
"duration_ms": round(duration * 1000, 2),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"success": False, "model": model_name, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
def find_comfy_model_files() -> List[Dict[str, Any]]:
|
||||
"""Discover all model files under ComfyUI models."""
|
||||
results = []
|
||||
extensions = ("*.safetensors", "*.ckpt", "*.pt", "*.bin")
|
||||
if os.path.exists(COMFY_MODELS_DIR):
|
||||
for root, _, files in os.walk(COMFY_MODELS_DIR):
|
||||
for file in files:
|
||||
if any(file.endswith(ext.replace("*", "")) for ext in extensions):
|
||||
full_path = os.path.join(root, file)
|
||||
rel_path = os.path.relpath(full_path, COMFY_MODELS_DIR)
|
||||
size = os.path.getsize(full_path)
|
||||
results.append({
|
||||
"filename": file,
|
||||
"rel_path": rel_path,
|
||||
"full_path": full_path,
|
||||
"size_bytes": size,
|
||||
"size_mb": round(size / (1024**2), 2),
|
||||
"size_gb": round(size / (1024**3), 3),
|
||||
})
|
||||
return results
|
||||
|
||||
async def warm_all_models() -> Dict[str, Any]:
|
||||
"""Warm all available Ollama and ComfyUI models into Linux RAM Cache."""
|
||||
t0 = time.perf_counter()
|
||||
warmed_ollama = []
|
||||
warmed_comfy = []
|
||||
|
||||
# 1. Ollama models
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
tags_resp = await client.get(f"{OLLAMA_API_BASE}/api/tags")
|
||||
if tags_resp.status_code == 200:
|
||||
models = tags_resp.json().get("models", [])
|
||||
for m in models:
|
||||
name = m.get("name")
|
||||
res = await warm_ollama_model(name, keep_alive="1m")
|
||||
warmed_ollama.append(res)
|
||||
except Exception as e:
|
||||
logger.error(f"Error discovering Ollama models: {e}")
|
||||
|
||||
# 2. ComfyUI models
|
||||
comfy_files = find_comfy_model_files()
|
||||
for f in comfy_files:
|
||||
res = warm_file_to_ram(f["full_path"])
|
||||
warmed_comfy.append(res)
|
||||
|
||||
total_duration_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
meminfo = get_detailed_meminfo()
|
||||
|
||||
return {
|
||||
"status": "completed",
|
||||
"total_duration_ms": total_duration_ms,
|
||||
"ollama_models_warmed": warmed_ollama,
|
||||
"comfy_files_warmed": warmed_comfy,
|
||||
"meminfo_after": meminfo,
|
||||
}
|
||||
174
server.py
Normal file
174
server.py
Normal file
@@ -0,0 +1,174 @@
|
||||
"""FastAPI Backend Server with SSE Real-Time Telemetry and Model Orchestration API."""
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from typing import Dict, Any, Optional, List
|
||||
from fastapi import FastAPI, Request, HTTPException, Query
|
||||
from fastapi.responses import HTMLResponse, StreamingResponse, JSONResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
import ram_optimizer
|
||||
import vram_arbitrator
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s")
|
||||
logger = logging.getLogger("model_manager_server")
|
||||
|
||||
app = FastAPI(
|
||||
title="HyperSwap // GPU Program Swapper & Telemetry API",
|
||||
version="1.0.0",
|
||||
description="High-performance VRAM arbitration and 64GB RAM cache orchestrator for simultaneous Ollama and ComfyUI workloads on Linux.",
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc",
|
||||
)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Pydantic Request Models
|
||||
class SwitchRequest(BaseModel):
|
||||
model: str = Field(..., description="Name of the Ollama model to hot-swap to in VRAM", example="qwen3.8fast:latest")
|
||||
keep_alive: Optional[str] = Field("30m", description="Keep-alive duration in VRAM (e.g. 5m, 30m, 0)", example="30m")
|
||||
|
||||
class WarmRequest(BaseModel):
|
||||
model_name: Optional[str] = Field(None, description="Ollama model name to warm into OS page cache", example="gemma4:26b")
|
||||
filepath: Optional[str] = Field(None, description="Absolute file path of Safetensors/GGUF to warm into RAM", example="/home/drjones/ComfyUI/models/checkpoints/v1-5-pruned-emaonly-fp16.safetensors")
|
||||
|
||||
class BenchmarkRequest(BaseModel):
|
||||
iterations: Optional[int] = Field(2, description="Number of back-and-forth switch iterations to measure", example=2)
|
||||
models: Optional[List[str]] = Field(None, description="Optional pair of models to benchmark between", example=["qwen3.8fast:latest", "smtek/Qwen3.8-27B:Q2_K_XL"])
|
||||
|
||||
|
||||
# ==========================================
|
||||
# REST API ENDPOINTS
|
||||
# ==========================================
|
||||
|
||||
@app.get("/api/stats", summary="Full System Snapshot", tags=["Telemetry"])
|
||||
async def get_all_stats() -> Dict[str, Any]:
|
||||
"""Gather complete live snapshot of GPU hardware, host RAM, Ollama, ComfyUI, and switch history."""
|
||||
gpu_stats = vram_arbitrator.get_gpu_hardware_stats()
|
||||
mem_stats = ram_optimizer.get_detailed_meminfo()
|
||||
ollama_state = await vram_arbitrator.get_ollama_live_state()
|
||||
comfy_state = await vram_arbitrator.get_comfyui_live_state()
|
||||
history = vram_arbitrator.get_switch_history()
|
||||
comfy_models = ram_optimizer.find_comfy_model_files()
|
||||
|
||||
return {
|
||||
"timestamp": asyncio.get_event_loop().time(),
|
||||
"gpu": gpu_stats,
|
||||
"ram": mem_stats,
|
||||
"ollama": ollama_state,
|
||||
"comfyui": comfy_state,
|
||||
"history": history,
|
||||
"comfy_models_count": len(comfy_models),
|
||||
}
|
||||
|
||||
@app.get("/api/gpu", summary="GPU Sensors and VRAM Breakdown", tags=["Telemetry"])
|
||||
async def get_gpu_metrics() -> Dict[str, Any]:
|
||||
"""Retrieve detailed NVML sensors (utilization %, temp, power, fan, clocks, and per-process VRAM allocation)."""
|
||||
return vram_arbitrator.get_gpu_hardware_stats()
|
||||
|
||||
@app.get("/api/memory", summary="Host RAM and Page Cache Breakdown", tags=["Telemetry"])
|
||||
async def get_ram_metrics() -> Dict[str, Any]:
|
||||
"""Retrieve precise host 64GB DDR5 RAM breakdown, active cache size, and cache hit ratios."""
|
||||
return ram_optimizer.get_detailed_meminfo()
|
||||
|
||||
@app.get("/api/stream", summary="Real-Time SSE Telemetry Stream", tags=["Telemetry"])
|
||||
async def sse_telemetry_stream(request: Request):
|
||||
"""Server-Sent Events (SSE) streaming real-time statistics at 1Hz for dynamic dashboards."""
|
||||
async def event_generator():
|
||||
while True:
|
||||
if await request.is_disconnected():
|
||||
break
|
||||
try:
|
||||
stats = await get_all_stats()
|
||||
yield f"data: {json.dumps(stats)}\n\n"
|
||||
except Exception as e:
|
||||
logger.error(f"SSE stream error: {e}")
|
||||
yield f"data: {json.dumps({'error': str(e)})}\n\n"
|
||||
await asyncio.sleep(1.0)
|
||||
|
||||
return StreamingResponse(
|
||||
event_generator(),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
}
|
||||
)
|
||||
|
||||
@app.post("/api/switch-model", summary="Hot-Swap Ollama LLM in VRAM", tags=["Orchestration"])
|
||||
async def api_switch_model(req: SwitchRequest):
|
||||
"""Hot-swap the active Ollama model in VRAM and measure exact load duration and token evaluation speed."""
|
||||
res = await vram_arbitrator.switch_ollama_model(req.model, keep_alive=req.keep_alive or "30m")
|
||||
if not res.get("success"):
|
||||
raise HTTPException(status_code=500, detail=res.get("error"))
|
||||
return res
|
||||
|
||||
@app.post("/api/free-vram", summary="Soft-Yield Ollama VRAM", tags=["Orchestration"])
|
||||
async def api_free_vram():
|
||||
"""Instruct Ollama to instantly yield VRAM to 0MB in ~15ms while preserving model weights in the 64GB host RAM page cache."""
|
||||
return await vram_arbitrator.instant_free_ollama_vram()
|
||||
|
||||
@app.post("/api/comfy-free", summary="Purge ComfyUI VRAM Cache", tags=["Orchestration"])
|
||||
async def api_comfy_free():
|
||||
"""Purge loaded diffusion models and VRAM cache from the ComfyUI pipeline."""
|
||||
return await vram_arbitrator.instant_free_comfyui_vram()
|
||||
|
||||
@app.post("/api/warm-all", summary="Pre-warm All Models into RAM Cache", tags=["Memory Optimization"])
|
||||
async def api_warm_all():
|
||||
"""Pre-fault and read all installed Ollama GGUF models and ComfyUI Safetensors checkpoints into the Linux OS Page Cache."""
|
||||
return await ram_optimizer.warm_all_models()
|
||||
|
||||
@app.post("/api/warm-model", summary="Pre-warm Single Model or File", tags=["Memory Optimization"])
|
||||
async def api_warm_model(req: WarmRequest):
|
||||
"""Pre-warm a specific Ollama model or individual file path into Linux RAM cache."""
|
||||
if req.model_name:
|
||||
return await ram_optimizer.warm_ollama_model(req.model_name, keep_alive="1m")
|
||||
elif req.filepath:
|
||||
return ram_optimizer.warm_file_to_ram(req.filepath)
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="model_name or filepath required")
|
||||
|
||||
@app.get("/api/models", summary="List All Installed Models", tags=["Catalog"])
|
||||
async def api_get_models():
|
||||
"""List all installed Ollama models and discovered ComfyUI model checkpoints/safetensors on disk with sizes and quantization levels."""
|
||||
ollama_state = await vram_arbitrator.get_ollama_live_state()
|
||||
comfy_models = ram_optimizer.find_comfy_model_files()
|
||||
return {
|
||||
"ollama_models": ollama_state.get("installed_models", []),
|
||||
"comfy_models": comfy_models,
|
||||
}
|
||||
|
||||
@app.get("/api/history", summary="Model Switch History Log", tags=["Analytics"])
|
||||
async def api_get_history(limit: int = Query(20, description="Max history items to return")):
|
||||
"""Get the recent history log of model switch events, swap durations (in ms), and RAM cache hit status."""
|
||||
history = vram_arbitrator.get_switch_history()
|
||||
return history[:limit]
|
||||
|
||||
@app.post("/api/benchmark", summary="Run Latency Benchmark", tags=["Analytics"])
|
||||
async def api_run_benchmark(req: BenchmarkRequest):
|
||||
"""Run an automated benchmark swapping between available models to measure round-trip latency and RAM cache effectiveness."""
|
||||
from mcp_server import run_model_switch_benchmark
|
||||
res_str = await run_model_switch_benchmark(iterations=req.iterations or 2)
|
||||
return json.loads(res_str)
|
||||
|
||||
# Mount static web UI files
|
||||
app.mount("/static", StaticFiles(directory="/home/drjones/unified-model-manager/static"), name="static")
|
||||
|
||||
@app.get("/", summary="Dashboard Web UI", tags=["UI"])
|
||||
async def root_index():
|
||||
with open("/home/drjones/unified-model-manager/static/index.html", "r") as f:
|
||||
content = f.read()
|
||||
return HTMLResponse(content=content)
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run("server:app", host="0.0.0.0", port=9090, reload=False, log_level="info")
|
||||
4
start_manager.sh
Executable file
4
start_manager.sh
Executable file
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
exec /home/drjones/comfy-mcp-venv/bin/python server.py
|
||||
288
static/app.js
Normal file
288
static/app.js
Normal file
@@ -0,0 +1,288 @@
|
||||
let evtSource = null;
|
||||
let currentInstalledModels = [];
|
||||
|
||||
// Initialize SSE Stream
|
||||
function initSSE() {
|
||||
if (evtSource) {
|
||||
evtSource.close();
|
||||
}
|
||||
|
||||
evtSource = new EventSource('/api/stream');
|
||||
|
||||
evtSource.onopen = () => {
|
||||
document.getElementById('sse-badge').textContent = 'SSE LIVE';
|
||||
document.getElementById('sse-badge').className = 'text-emerald-400 font-bold';
|
||||
};
|
||||
|
||||
evtSource.onmessage = (event) => {
|
||||
try {
|
||||
const data = JSON.parse(event.data);
|
||||
updateDashboard(data);
|
||||
} catch (err) {
|
||||
console.error('Error parsing SSE event:', err);
|
||||
}
|
||||
};
|
||||
|
||||
evtSource.onerror = (err) => {
|
||||
console.warn('SSE disconnected, retrying...', err);
|
||||
document.getElementById('sse-badge').textContent = 'RECONNECTING...';
|
||||
document.getElementById('sse-badge').className = 'text-rose-400 font-bold';
|
||||
};
|
||||
}
|
||||
|
||||
function updateDashboard(data) {
|
||||
if (!data) return;
|
||||
|
||||
// 1. GPU VRAM Stats
|
||||
const gpu = data.gpu || {};
|
||||
if (gpu.available) {
|
||||
document.getElementById('gpu-chip-name').textContent = gpu.device_name || 'NVIDIA GPU';
|
||||
document.getElementById('vram-total-used').textContent = gpu.vram_used_gb || '0.0';
|
||||
document.getElementById('vram-used-pct').textContent = `${gpu.vram_used_pct || 0}% USED`;
|
||||
|
||||
const bd = gpu.breakdown || {};
|
||||
const totalBytes = gpu.vram_total_bytes || (16 * 1024**3);
|
||||
|
||||
const ollamaPct = ((bd.ollama_gb || 0) / (gpu.vram_total_gb || 16)) * 100;
|
||||
const comfyPct = ((bd.comfyui_gb || 0) / (gpu.vram_total_gb || 16)) * 100;
|
||||
const systemPct = ((bd.system_gb || 0) / (gpu.vram_total_gb || 16)) * 100;
|
||||
const freePct = Math.max(0, 100 - (ollamaPct + comfyPct + systemPct));
|
||||
|
||||
document.getElementById('bar-ollama').style.width = `${ollamaPct}%`;
|
||||
document.getElementById('bar-comfy').style.width = `${comfyPct}%`;
|
||||
document.getElementById('bar-system').style.width = `${systemPct}%`;
|
||||
document.getElementById('bar-free').style.width = `${freePct}%`;
|
||||
|
||||
document.getElementById('tooltip-ollama').textContent = `${bd.ollama_gb || 0} GB`;
|
||||
document.getElementById('tooltip-comfy').textContent = `${bd.comfyui_gb || 0} GB`;
|
||||
document.getElementById('tooltip-system').textContent = `${bd.system_gb || 0} GB`;
|
||||
|
||||
document.getElementById('legend-ollama').textContent = `${bd.ollama_gb || 0} GB`;
|
||||
document.getElementById('legend-comfy').textContent = `${bd.comfyui_gb || 0} GB`;
|
||||
document.getElementById('legend-system').textContent = `${bd.system_gb || 0} GB`;
|
||||
document.getElementById('legend-free').textContent = `${bd.free_gb || 0} GB`;
|
||||
|
||||
// Hardware sensors
|
||||
document.getElementById('gpu-util-val').textContent = `${gpu.gpu_util_pct || 0}%`;
|
||||
document.getElementById('gpu-temp-val').textContent = `${gpu.temperature_c || 0}°C`;
|
||||
document.getElementById('gpu-power-val').textContent = `${gpu.power_w || 0} W`;
|
||||
document.getElementById('gpu-fan-val').textContent = `${gpu.fan_pct || 0}%`;
|
||||
|
||||
// Processes table
|
||||
const tbody = document.getElementById('gpu-proc-table');
|
||||
if (bd.processes && bd.processes.length > 0) {
|
||||
tbody.innerHTML = bd.processes.map(p => {
|
||||
let tagClass = 'text-slate-400';
|
||||
let badge = '';
|
||||
if (p.is_ollama) {
|
||||
tagClass = 'text-purple-400 font-bold';
|
||||
badge = '<span class="px-1 py-0.2 bg-purple-950 text-purple-300 rounded border border-purple-800 text-[9px] mr-1">OLLAMA</span>';
|
||||
} else if (p.is_comfy) {
|
||||
tagClass = 'text-cyan-400 font-bold';
|
||||
badge = '<span class="px-1 py-0.2 bg-cyan-950 text-cyan-300 rounded border border-cyan-800 text-[9px] mr-1">COMFY</span>';
|
||||
}
|
||||
return `
|
||||
<tr class="hover:bg-slate-900/50">
|
||||
<td class="py-1 text-slate-500 font-mono text-[10px]">${p.pid}</td>
|
||||
<td class="py-1 ${tagClass} text-[11px] truncate max-w-[140px]">${badge}${p.name}</td>
|
||||
<td class="py-1 text-right font-mono font-bold text-slate-200 text-[11px]">${p.vram_mb} MB</td>
|
||||
</tr>
|
||||
`;
|
||||
}).join('');
|
||||
} else {
|
||||
tbody.innerHTML = '<tr><td colspan="3" class="py-2 text-center text-slate-500">No compute processes running</td></tr>';
|
||||
}
|
||||
}
|
||||
|
||||
// 2. System RAM & Page Cache
|
||||
const ram = data.ram || {};
|
||||
if (ram.total_bytes) {
|
||||
document.getElementById('ram-cached-gb').textContent = ram.cached_gb || '0.0';
|
||||
document.getElementById('ram-total-text').textContent = `${ram.total_gb || 0} GB Total (${ram.cache_ratio_pct || 0}% in Cache)`;
|
||||
|
||||
const usedPct = ((ram.used_bytes || 0) / ram.total_bytes) * 100;
|
||||
const cachePct = ((ram.cached_bytes || 0) / ram.total_bytes) * 100;
|
||||
const freePct = Math.max(0, 100 - (usedPct + cachePct));
|
||||
|
||||
document.getElementById('bar-ram-used').style.width = `${usedPct}%`;
|
||||
document.getElementById('bar-ram-cache').style.width = `${cachePct}%`;
|
||||
document.getElementById('bar-ram-free').style.width = `${freePct}%`;
|
||||
|
||||
document.getElementById('legend-ram-used').textContent = `${ram.used_gb || 0} GB`;
|
||||
document.getElementById('legend-ram-cached').textContent = `${ram.cached_gb || 0} GB`;
|
||||
document.getElementById('legend-ram-free').textContent = `${ram.free_gb || 0} GB`;
|
||||
}
|
||||
|
||||
// 3. Ollama State
|
||||
const ollama = data.ollama || {};
|
||||
if (ollama.online) {
|
||||
document.getElementById('ollama-status-text').textContent = 'ONLINE';
|
||||
document.getElementById('ollama-status-text').className = 'text-emerald-400 font-mono font-bold';
|
||||
document.getElementById('ollama-dot').className = 'relative inline-flex rounded-full h-2.5 w-2.5 bg-emerald-500';
|
||||
document.getElementById('ollama-pulse').className = 'animate-ping absolute inline-flex h-full w-full rounded-full bg-emerald-400 opacity-75';
|
||||
|
||||
if (ollama.active_model_name) {
|
||||
document.getElementById('ollama-active-model').textContent = ollama.active_model_name;
|
||||
document.getElementById('ollama-vram-badge').textContent = `${ollama.active_model_vram_gb || 0} GB VRAM`;
|
||||
document.getElementById('ollama-vram-badge').className = 'px-2 py-0.5 text-[10px] font-mono rounded bg-purple-900/80 text-purple-200 border border-purple-600 font-bold';
|
||||
document.getElementById('ollama-context').textContent = `${ollama.active_context || 0} ctx`;
|
||||
} else {
|
||||
document.getElementById('ollama-active-model').textContent = 'None Loaded (VRAM Free)';
|
||||
document.getElementById('ollama-vram-badge').textContent = '0.0 GB VRAM';
|
||||
document.getElementById('ollama-vram-badge').className = 'px-2 py-0.5 text-[10px] font-mono rounded bg-slate-800 text-slate-400 border border-slate-700';
|
||||
document.getElementById('ollama-context').textContent = 'Idle';
|
||||
}
|
||||
|
||||
if (ollama.installed_models && ollama.installed_models.length > 0) {
|
||||
document.getElementById('ollama-total-models').textContent = ollama.installed_models.length;
|
||||
updateModelSelect(ollama.installed_models, ollama.active_model_name);
|
||||
}
|
||||
} else {
|
||||
document.getElementById('ollama-status-text').textContent = 'OFFLINE';
|
||||
document.getElementById('ollama-status-text').className = 'text-rose-400 font-mono font-bold';
|
||||
document.getElementById('ollama-dot').className = 'relative inline-flex rounded-full h-2.5 w-2.5 bg-rose-500';
|
||||
document.getElementById('ollama-pulse').className = 'hidden';
|
||||
}
|
||||
|
||||
// 4. ComfyUI State
|
||||
const comfy = data.comfyui || {};
|
||||
if (comfy.online) {
|
||||
document.getElementById('comfy-status-text').textContent = 'ONLINE';
|
||||
document.getElementById('comfy-status-text').className = 'text-emerald-400 font-mono font-bold';
|
||||
document.getElementById('comfy-dot').className = 'relative inline-flex rounded-full h-2.5 w-2.5 bg-emerald-500';
|
||||
document.getElementById('comfy-pulse').className = 'animate-ping absolute inline-flex h-full w-full rounded-full bg-emerald-400 opacity-75';
|
||||
|
||||
document.getElementById('comfy-queue-count').textContent = comfy.queue_running + comfy.queue_remaining;
|
||||
if (comfy.executing) {
|
||||
document.getElementById('comfy-exec-badge').textContent = 'GENERATING...';
|
||||
document.getElementById('comfy-exec-badge').className = 'px-2 py-0.5 text-[10px] font-mono rounded bg-amber-950 text-amber-300 border border-amber-600 animate-pulse font-bold';
|
||||
document.getElementById('comfy-model-info').textContent = `Prompt #${comfy.current_prompt_id || 'Active'}`;
|
||||
} else {
|
||||
document.getElementById('comfy-exec-badge').textContent = 'IDLE / READY';
|
||||
document.getElementById('comfy-exec-badge').className = 'px-2 py-0.5 text-[10px] font-mono rounded bg-emerald-950 text-emerald-300 border border-emerald-800';
|
||||
document.getElementById('comfy-model-info').textContent = 'Dynamic Model Offloader';
|
||||
}
|
||||
|
||||
document.getElementById('comfy-vram-avail').textContent = `${(comfy.vram_free_mb / 1024).toFixed(1)} GB`;
|
||||
} else {
|
||||
document.getElementById('comfy-status-text').textContent = 'OFFLINE';
|
||||
document.getElementById('comfy-status-text').className = 'text-rose-400 font-mono font-bold';
|
||||
document.getElementById('comfy-dot').className = 'relative inline-flex rounded-full h-2.5 w-2.5 bg-rose-500';
|
||||
document.getElementById('comfy-pulse').className = 'hidden';
|
||||
}
|
||||
|
||||
document.getElementById('comfy-discovered-count').textContent = `${data.comfy_models_count || 0} Files`;
|
||||
|
||||
// 5. Switch History Timeline
|
||||
const history = data.history || [];
|
||||
const logContainer = document.getElementById('switch-log-container');
|
||||
if (history.length > 0) {
|
||||
const latest = history[0];
|
||||
document.getElementById('ollama-last-swap').textContent = `${latest.duration_ms} ms`;
|
||||
document.getElementById('ollama-cache-hit').textContent = latest.cache_status || 'OK';
|
||||
|
||||
logContainer.innerHTML = history.slice(0, 10).map(item => {
|
||||
const isHit = (item.cache_status || '').includes('RAM Cache Hit') || (item.cache_status || '').includes('RAM-Cached');
|
||||
const badgeClass = isHit
|
||||
? 'bg-emerald-950/80 text-emerald-300 border-emerald-800'
|
||||
: 'bg-amber-950/80 text-amber-300 border-amber-800';
|
||||
return `
|
||||
<div class="p-2.5 rounded-xl bg-slate-950 border border-slate-800 text-xs font-mono flex items-center justify-between">
|
||||
<div class="space-y-0.5">
|
||||
<div class="flex items-center space-x-1.5">
|
||||
<span class="text-slate-500 text-[10px]">${item.timestamp}</span>
|
||||
<span class="text-indigo-400 font-bold">${item.event_type}</span>
|
||||
</div>
|
||||
<div class="text-[11px] text-slate-300 truncate max-w-[260px]">
|
||||
<span class="text-slate-500">${item.source}</span> → <span class="text-cyan-300 font-bold">${item.target}</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="text-right">
|
||||
<div class="text-slate-100 font-bold">${item.duration_ms} ms</div>
|
||||
<span class="px-1.5 py-0.2 text-[9px] rounded border ${badgeClass}">${item.cache_status}</span>
|
||||
</div>
|
||||
</div>
|
||||
`;
|
||||
}).join('');
|
||||
}
|
||||
}
|
||||
|
||||
function updateModelSelect(models, activeModel) {
|
||||
const select = document.getElementById('ollama-model-select');
|
||||
const currentVal = select.value;
|
||||
|
||||
if (JSON.stringify(models.map(m => m.name)) === JSON.stringify(currentInstalledModels)) {
|
||||
return;
|
||||
}
|
||||
|
||||
currentInstalledModels = models.map(m => m.name);
|
||||
select.innerHTML = models.map(m => {
|
||||
const isSelected = m.name === activeModel || m.name === currentVal;
|
||||
const sizeGb = (m.size / (1024**3)).toFixed(1);
|
||||
const quant = m.details?.quantization_level || '';
|
||||
return `<option value="${m.name}" ${isSelected ? 'selected' : ''}>${m.name} (${sizeGb} GB ${quant})</option>`;
|
||||
}).join('');
|
||||
}
|
||||
|
||||
// User Actions
|
||||
async function triggerModelSwitch() {
|
||||
const select = document.getElementById('ollama-model-select');
|
||||
const targetModel = select.value;
|
||||
if (!targetModel) return;
|
||||
|
||||
const btn = document.getElementById('btn-switch-model');
|
||||
btn.disabled = true;
|
||||
btn.innerHTML = '<i class="fa-solid fa-spinner fa-spin"></i> <span>Swapping...</span>';
|
||||
|
||||
try {
|
||||
const resp = await fetch('/api/switch-model', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ model: targetModel, keep_alive: '30m' })
|
||||
});
|
||||
const result = await resp.json();
|
||||
if (!resp.ok) {
|
||||
alert(`Switch failed: ${result.detail || 'Unknown error'}`);
|
||||
}
|
||||
} catch (err) {
|
||||
alert(`Error: ${err}`);
|
||||
} finally {
|
||||
btn.disabled = false;
|
||||
btn.innerHTML = '<i class="fa-solid fa-shuffle"></i> <span>Hot Swap</span>';
|
||||
}
|
||||
}
|
||||
|
||||
async function freeOllamaVRAM() {
|
||||
try {
|
||||
const resp = await fetch('/api/free-vram', { method: 'POST' });
|
||||
const data = await resp.json();
|
||||
console.log('Ollama VRAM yielded:', data);
|
||||
} catch (err) {
|
||||
alert(`Error: ${err}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function freeComfyVRAM() {
|
||||
try {
|
||||
const resp = await fetch('/api/comfy-free', { method: 'POST' });
|
||||
const data = await resp.json();
|
||||
console.log('ComfyUI VRAM freed:', data);
|
||||
} catch (err) {
|
||||
alert(`Error: ${err}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function warmAllModels() {
|
||||
try {
|
||||
const resp = await fetch('/api/warm-all', { method: 'POST' });
|
||||
const data = await resp.json();
|
||||
alert(`Warming completed in ${data.total_duration_ms} ms! All models are now cached in 64GB RAM.`);
|
||||
} catch (err) {
|
||||
alert(`Error warming models: ${err}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Startup
|
||||
document.addEventListener('DOMContentLoaded', () => {
|
||||
initSSE();
|
||||
});
|
||||
426
static/index.html
Normal file
426
static/index.html
Normal file
@@ -0,0 +1,426 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en" class="dark">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>HYPERSWAP // Dual-Engine Model Orchestrator & Live Telemetry</title>
|
||||
<script src="https://cdn.tailwindcss.com"></script>
|
||||
<link rel="stylesheet" href="/static/styles.css">
|
||||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css">
|
||||
</head>
|
||||
<body class="bg-slate-950 text-slate-100 min-h-screen font-sans antialiased selection:bg-cyan-500 selection:text-white">
|
||||
|
||||
<!-- TOP HEADER -->
|
||||
<header class="border-b border-slate-800 bg-slate-900/80 backdrop-blur sticky top-0 z-50">
|
||||
<div class="max-w-7xl mx-auto px-4 sm:px-6 lg:px-8 py-3 flex flex-wrap items-center justify-between gap-4">
|
||||
<div class="flex items-center space-x-3">
|
||||
<div class="w-10 h-10 rounded-xl bg-gradient-to-tr from-cyan-500 via-indigo-500 to-purple-500 flex items-center justify-center shadow-lg shadow-cyan-500/20">
|
||||
<i class="fa-solid fa-bolt-lightning text-white text-lg"></i>
|
||||
</div>
|
||||
<div>
|
||||
<div class="flex items-center space-x-2">
|
||||
<h1 class="text-xl font-bold tracking-tight bg-clip-text text-transparent bg-gradient-to-r from-cyan-400 via-sky-300 to-indigo-400">
|
||||
HYPERSWAP
|
||||
</h1>
|
||||
<span class="text-xs uppercase tracking-widest px-2 py-0.5 rounded bg-cyan-950/80 text-cyan-400 border border-cyan-800 font-mono">
|
||||
v1.0-DEPLOY
|
||||
</span>
|
||||
</div>
|
||||
<p class="text-xs text-slate-400 font-mono">NVIDIA RTX 4080 SUPER 16GB // 64GB DDR5 RAM // Ubuntu Linux</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="flex items-center space-x-3">
|
||||
<!-- Service Status Indicators -->
|
||||
<div class="flex items-center space-x-2 px-3 py-1.5 rounded-lg bg-slate-800/80 border border-slate-700/60 text-xs">
|
||||
<span class="relative flex h-2.5 w-2.5">
|
||||
<span id="ollama-pulse" class="animate-ping absolute inline-flex h-full w-full rounded-full bg-emerald-400 opacity-75"></span>
|
||||
<span id="ollama-dot" class="relative inline-flex rounded-full h-2.5 w-2.5 bg-emerald-500"></span>
|
||||
</span>
|
||||
<span class="font-medium text-slate-300">Ollama LLM:</span>
|
||||
<span id="ollama-status-text" class="text-emerald-400 font-mono font-bold">ONLINE</span>
|
||||
</div>
|
||||
|
||||
<div class="flex items-center space-x-2 px-3 py-1.5 rounded-lg bg-slate-800/80 border border-slate-700/60 text-xs">
|
||||
<span class="relative flex h-2.5 w-2.5">
|
||||
<span id="comfy-pulse" class="animate-ping absolute inline-flex h-full w-full rounded-full bg-emerald-400 opacity-75"></span>
|
||||
<span id="comfy-dot" class="relative inline-flex rounded-full h-2.5 w-2.5 bg-emerald-500"></span>
|
||||
</span>
|
||||
<span class="font-medium text-slate-300">ComfyUI:</span>
|
||||
<span id="comfy-status-text" class="text-emerald-400 font-mono font-bold">ONLINE</span>
|
||||
</div>
|
||||
|
||||
<div class="flex items-center space-x-1.5 px-3 py-1.5 rounded-lg bg-slate-800/80 border border-slate-700/60 text-xs text-slate-400 font-mono">
|
||||
<i class="fa-solid fa-satellite-dish text-cyan-400 text-xs animate-pulse"></i>
|
||||
<span id="sse-badge">SSE 1Hz</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<!-- MAIN CONTAINER -->
|
||||
<main class="max-w-7xl mx-auto px-4 sm:px-6 lg:px-8 py-6 space-y-6">
|
||||
|
||||
<!-- HERO MEMORY GAUGES -->
|
||||
<div class="grid grid-cols-1 md:grid-cols-2 gap-6">
|
||||
<!-- GPU VRAM ALLOCATION GAUGE -->
|
||||
<div class="bg-slate-900/70 border border-slate-800 rounded-2xl p-5 relative overflow-hidden backdrop-blur">
|
||||
<div class="flex justify-between items-start mb-3">
|
||||
<div>
|
||||
<div class="flex items-center space-x-2">
|
||||
<i class="fa-solid fa-microchip text-cyan-400"></i>
|
||||
<h2 class="text-sm font-semibold text-slate-200 tracking-wide uppercase">GPU VRAM (16 GB Dedicated)</h2>
|
||||
</div>
|
||||
<p class="text-xs text-slate-400 mt-0.5">Real-time allocation breakdown on RTX 4080 SUPER</p>
|
||||
</div>
|
||||
<div class="text-right font-mono">
|
||||
<span id="vram-total-used" class="text-2xl font-bold text-slate-100">0.0</span>
|
||||
<span class="text-xs text-slate-400">/ 16.0 GB</span>
|
||||
<div id="vram-used-pct" class="text-xs text-cyan-400 font-bold">0% USED</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Segmented Bar -->
|
||||
<div class="h-6 w-full bg-slate-950 rounded-lg overflow-hidden flex border border-slate-800 p-0.5">
|
||||
<div id="bar-ollama" class="bg-gradient-to-r from-purple-600 to-indigo-500 h-full rounded-l transition-all duration-500 relative group" style="width: 0%">
|
||||
<div class="opacity-0 group-hover:opacity-100 absolute -top-8 left-1/2 -translate-x-1/2 bg-slate-800 text-[10px] text-white px-2 py-0.5 rounded border border-slate-700 whitespace-nowrap z-20">
|
||||
Ollama: <span id="tooltip-ollama">0 GB</span>
|
||||
</div>
|
||||
</div>
|
||||
<div id="bar-comfy" class="bg-gradient-to-r from-cyan-500 to-sky-400 h-full transition-all duration-500 relative group" style="width: 0%">
|
||||
<div class="opacity-0 group-hover:opacity-100 absolute -top-8 left-1/2 -translate-x-1/2 bg-slate-800 text-[10px] text-white px-2 py-0.5 rounded border border-slate-700 whitespace-nowrap z-20">
|
||||
ComfyUI: <span id="tooltip-comfy">0 GB</span>
|
||||
</div>
|
||||
</div>
|
||||
<div id="bar-system" class="bg-slate-600 h-full transition-all duration-500 relative group" style="width: 0%">
|
||||
<div class="opacity-0 group-hover:opacity-100 absolute -top-8 left-1/2 -translate-x-1/2 bg-slate-800 text-[10px] text-white px-2 py-0.5 rounded border border-slate-700 whitespace-nowrap z-20">
|
||||
System: <span id="tooltip-system">0 GB</span>
|
||||
</div>
|
||||
</div>
|
||||
<div id="bar-free" class="bg-slate-900 h-full rounded-r transition-all duration-500" style="width: 100%"></div>
|
||||
</div>
|
||||
|
||||
<!-- Legend -->
|
||||
<div class="grid grid-cols-4 gap-2 mt-3 text-xs font-mono">
|
||||
<div class="flex items-center space-x-1.5">
|
||||
<span class="w-2.5 h-2.5 rounded-full bg-purple-500"></span>
|
||||
<span class="text-slate-400">Ollama:</span>
|
||||
<span id="legend-ollama" class="text-slate-200 font-bold">0 GB</span>
|
||||
</div>
|
||||
<div class="flex items-center space-x-1.5">
|
||||
<span class="w-2.5 h-2.5 rounded-full bg-cyan-400"></span>
|
||||
<span class="text-slate-400">Comfy:</span>
|
||||
<span id="legend-comfy" class="text-slate-200 font-bold">0 GB</span>
|
||||
</div>
|
||||
<div class="flex items-center space-x-1.5">
|
||||
<span class="w-2.5 h-2.5 rounded-full bg-slate-500"></span>
|
||||
<span class="text-slate-400">System:</span>
|
||||
<span id="legend-system" class="text-slate-200 font-bold">0 GB</span>
|
||||
</div>
|
||||
<div class="flex items-center space-x-1.5">
|
||||
<span class="w-2.5 h-2.5 rounded-full bg-slate-800 border border-slate-700"></span>
|
||||
<span class="text-slate-400">Free:</span>
|
||||
<span id="legend-free" class="text-emerald-400 font-bold">0 GB</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- SYSTEM RAM & PAGE CACHE GAUGE -->
|
||||
<div class="bg-slate-900/70 border border-slate-800 rounded-2xl p-5 relative overflow-hidden backdrop-blur">
|
||||
<div class="flex justify-between items-start mb-3">
|
||||
<div>
|
||||
<div class="flex items-center space-x-2">
|
||||
<i class="fa-solid fa-memory text-amber-400"></i>
|
||||
<h2 class="text-sm font-semibold text-slate-200 tracking-wide uppercase">Host RAM & Model Page Cache (64 GB)</h2>
|
||||
</div>
|
||||
<p class="text-xs text-slate-400 mt-0.5">Models stay resident in RAM for instant PCIe hot-swaps</p>
|
||||
</div>
|
||||
<div class="text-right font-mono">
|
||||
<span id="ram-cached-gb" class="text-2xl font-bold text-amber-400">0.0</span>
|
||||
<span class="text-xs text-slate-400">GB CACHED</span>
|
||||
<div id="ram-total-text" class="text-xs text-slate-400">60.3 GB Total</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Segmented Bar -->
|
||||
<div class="h-6 w-full bg-slate-950 rounded-lg overflow-hidden flex border border-slate-800 p-0.5">
|
||||
<div id="bar-ram-used" class="bg-gradient-to-r from-rose-600 to-orange-500 h-full rounded-l transition-all duration-500" style="width: 10%"></div>
|
||||
<div id="bar-ram-cache" class="bg-gradient-to-r from-amber-500 to-yellow-400 h-full transition-all duration-500" style="width: 50%"></div>
|
||||
<div id="bar-ram-free" class="bg-slate-900 h-full rounded-r transition-all duration-500" style="width: 40%"></div>
|
||||
</div>
|
||||
|
||||
<!-- Legend -->
|
||||
<div class="grid grid-cols-3 gap-2 mt-3 text-xs font-mono">
|
||||
<div class="flex items-center space-x-1.5">
|
||||
<span class="w-2.5 h-2.5 rounded-full bg-rose-500"></span>
|
||||
<span class="text-slate-400">Apps Used:</span>
|
||||
<span id="legend-ram-used" class="text-slate-200 font-bold">0 GB</span>
|
||||
</div>
|
||||
<div class="flex items-center space-x-1.5">
|
||||
<span class="w-2.5 h-2.5 rounded-full bg-amber-400"></span>
|
||||
<span class="text-slate-400">Models in RAM:</span>
|
||||
<span id="legend-ram-cached" class="text-amber-400 font-bold">0 GB</span>
|
||||
</div>
|
||||
<div class="flex items-center space-x-1.5">
|
||||
<span class="w-2.5 h-2.5 rounded-full bg-slate-800 border border-slate-700"></span>
|
||||
<span class="text-slate-400">Free RAM:</span>
|
||||
<span id="legend-ram-free" class="text-emerald-400 font-bold">0 GB</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 4-CARD CONTROL & TELEMETRY GRID -->
|
||||
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
||||
|
||||
<!-- CARD 1: OLLAMA LLM ENGINE -->
|
||||
<div class="bg-slate-900/80 border border-slate-800 rounded-2xl p-5 flex flex-col justify-between space-y-4">
|
||||
<div>
|
||||
<div class="flex items-center justify-between pb-3 border-b border-slate-800">
|
||||
<div class="flex items-center space-x-2">
|
||||
<div class="p-2 rounded-lg bg-purple-950/80 border border-purple-800 text-purple-400">
|
||||
<i class="fa-solid fa-brain text-sm"></i>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="font-bold text-slate-100 text-sm">Ollama LLM Engine</h3>
|
||||
<p class="text-xs text-slate-400">Port :11434 // FlashAttention + Q4 KV Cache</p>
|
||||
</div>
|
||||
</div>
|
||||
<button onclick="freeOllamaVRAM()" class="px-2.5 py-1 text-xs font-semibold rounded-lg bg-rose-950/70 border border-rose-800 text-rose-300 hover:bg-rose-900 transition flex items-center space-x-1">
|
||||
<i class="fa-solid fa-arrow-down-to-bracket"></i>
|
||||
<span>Soft-Yield VRAM</span>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<!-- Active Model Box -->
|
||||
<div class="mt-4 p-4 rounded-xl bg-slate-950 border border-slate-800/80 space-y-3">
|
||||
<div class="flex justify-between items-center">
|
||||
<span class="text-xs text-slate-400 uppercase font-mono">Active Model in VRAM</span>
|
||||
<span id="ollama-vram-badge" class="px-2 py-0.5 text-[10px] font-mono rounded bg-purple-900/60 text-purple-300 border border-purple-700">
|
||||
0.0 GB VRAM
|
||||
</span>
|
||||
</div>
|
||||
<div class="flex items-center justify-between">
|
||||
<div class="flex items-center space-x-2 overflow-hidden">
|
||||
<i class="fa-solid fa-cube text-purple-400"></i>
|
||||
<span id="ollama-active-model" class="text-base font-bold font-mono text-slate-100 truncate">None Loaded</span>
|
||||
</div>
|
||||
<span id="ollama-context" class="text-xs font-mono text-slate-400">0 ctx</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Quick Hot-Swap Control -->
|
||||
<div class="mt-4 space-y-2">
|
||||
<label class="text-xs font-semibold text-slate-300 flex items-center justify-between">
|
||||
<span>Instant Model Hot-Swap:</span>
|
||||
<span class="text-[10px] text-cyan-400 font-mono">RAM Cache Optimized</span>
|
||||
</label>
|
||||
<div class="flex space-x-2">
|
||||
<select id="ollama-model-select" class="flex-1 bg-slate-950 border border-slate-700 rounded-lg px-3 py-2 text-xs font-mono text-slate-200 focus:outline-none focus:border-purple-500">
|
||||
<option value="">Loading installed models...</option>
|
||||
</select>
|
||||
<button id="btn-switch-model" onclick="triggerModelSwitch()" class="px-4 py-2 bg-gradient-to-r from-purple-600 to-indigo-600 hover:from-purple-500 hover:to-indigo-500 text-white text-xs font-bold rounded-lg shadow-md transition flex items-center space-x-1.5">
|
||||
<i class="fa-solid fa-shuffle"></i>
|
||||
<span>Hot Swap</span>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Ollama Stats Footer -->
|
||||
<div class="pt-3 border-t border-slate-800 grid grid-cols-3 gap-2 text-center font-mono text-xs">
|
||||
<div class="p-2 rounded bg-slate-950 border border-slate-800">
|
||||
<span class="text-[10px] text-slate-500 block">LAST SWAP TIME</span>
|
||||
<span id="ollama-last-swap" class="text-cyan-400 font-bold">-- ms</span>
|
||||
</div>
|
||||
<div class="p-2 rounded bg-slate-950 border border-slate-800">
|
||||
<span class="text-[10px] text-slate-500 block">RAM HIT STATUS</span>
|
||||
<span id="ollama-cache-hit" class="text-emerald-400 font-bold">--</span>
|
||||
</div>
|
||||
<div class="p-2 rounded bg-slate-950 border border-slate-800">
|
||||
<span class="text-[10px] text-slate-500 block">TOTAL MODELS</span>
|
||||
<span id="ollama-total-models" class="text-slate-300 font-bold">0</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- CARD 2: COMFYUI DIFFUSION PIPELINE -->
|
||||
<div class="bg-slate-900/80 border border-slate-800 rounded-2xl p-5 flex flex-col justify-between space-y-4">
|
||||
<div>
|
||||
<div class="flex items-center justify-between pb-3 border-b border-slate-800">
|
||||
<div class="flex items-center space-x-2">
|
||||
<div class="p-2 rounded-lg bg-cyan-950/80 border border-cyan-800 text-cyan-400">
|
||||
<i class="fa-solid fa-palette text-sm"></i>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="font-bold text-slate-100 text-sm">ComfyUI Diffusion Engine</h3>
|
||||
<p class="text-xs text-slate-400">Port :8188 // DynamicVRAM + Pinned Async Offload</p>
|
||||
</div>
|
||||
</div>
|
||||
<button onclick="freeComfyVRAM()" class="px-2.5 py-1 text-xs font-semibold rounded-lg bg-rose-950/70 border border-rose-800 text-rose-300 hover:bg-rose-900 transition flex items-center space-x-1">
|
||||
<i class="fa-solid fa-broom"></i>
|
||||
<span>Purge VRAM</span>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<!-- Comfy Execution Box -->
|
||||
<div class="mt-4 p-4 rounded-xl bg-slate-950 border border-slate-800/80 space-y-3">
|
||||
<div class="flex justify-between items-center">
|
||||
<span class="text-xs text-slate-400 uppercase font-mono">Pipeline Status</span>
|
||||
<span id="comfy-exec-badge" class="px-2 py-0.5 text-[10px] font-mono rounded bg-emerald-950 text-emerald-300 border border-emerald-800">
|
||||
IDLE / READY
|
||||
</span>
|
||||
</div>
|
||||
<div class="flex items-center justify-between">
|
||||
<div class="flex items-center space-x-2">
|
||||
<i class="fa-solid fa-layer-group text-cyan-400"></i>
|
||||
<span id="comfy-model-info" class="text-sm font-mono text-slate-200">Dynamic Model Offloader</span>
|
||||
</div>
|
||||
<div class="text-xs font-mono text-slate-400">
|
||||
Queue: <span id="comfy-queue-count" class="text-cyan-400 font-bold">0</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Comfy Feature Checklist -->
|
||||
<div class="mt-4 space-y-2 text-xs">
|
||||
<div class="p-2.5 rounded-lg bg-slate-950 border border-slate-800 space-y-1.5">
|
||||
<div class="flex items-center justify-between text-slate-300">
|
||||
<span class="flex items-center space-x-1.5">
|
||||
<i class="fa-solid fa-check text-emerald-400 text-xs"></i>
|
||||
<span>Host Pinned Memory:</span>
|
||||
</span>
|
||||
<span class="font-mono font-bold text-emerald-400">53.6 GB Staging Buffer</span>
|
||||
</div>
|
||||
<div class="flex items-center justify-between text-slate-300">
|
||||
<span class="flex items-center space-x-1.5">
|
||||
<i class="fa-solid fa-check text-emerald-400 text-xs"></i>
|
||||
<span>Async PCIe Offloading:</span>
|
||||
</span>
|
||||
<span class="font-mono font-bold text-cyan-400">Enabled (2 Streams)</span>
|
||||
</div>
|
||||
<div class="flex items-center justify-between text-slate-300">
|
||||
<span class="flex items-center space-x-1.5">
|
||||
<i class="fa-solid fa-check text-emerald-400 text-xs"></i>
|
||||
<span>Fast Disk RAM Mmap:</span>
|
||||
</span>
|
||||
<span class="font-mono font-bold text-amber-400">Active</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Comfy Stats Footer -->
|
||||
<div class="pt-3 border-t border-slate-800 grid grid-cols-2 gap-2 text-center font-mono text-xs">
|
||||
<div class="p-2 rounded bg-slate-950 border border-slate-800">
|
||||
<span class="text-[10px] text-slate-500 block">DISCOVERED MODELS</span>
|
||||
<span id="comfy-discovered-count" class="text-cyan-400 font-bold">0 Files</span>
|
||||
</div>
|
||||
<div class="p-2 rounded bg-slate-950 border border-slate-800">
|
||||
<span class="text-[10px] text-slate-500 block">VRAM AVAILABLE</span>
|
||||
<span id="comfy-vram-avail" class="text-emerald-400 font-bold">15.9 GB</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- CARD 3: GPU HARDWARE TELEMETRY -->
|
||||
<div class="bg-slate-900/80 border border-slate-800 rounded-2xl p-5 space-y-4">
|
||||
<div class="flex items-center justify-between pb-3 border-b border-slate-800">
|
||||
<div class="flex items-center space-x-2">
|
||||
<div class="p-2 rounded-lg bg-emerald-950/80 border border-emerald-800 text-emerald-400">
|
||||
<i class="fa-solid fa-gauge-high text-sm"></i>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="font-bold text-slate-100 text-sm">GPU Live Telemetry (NVML)</h3>
|
||||
<p class="text-xs text-slate-400" id="gpu-chip-name">NVIDIA GeForce RTX 4080 SUPER</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Quick Gauges Grid -->
|
||||
<div class="grid grid-cols-2 sm:grid-cols-4 gap-3 font-mono">
|
||||
<div class="p-3 rounded-xl bg-slate-950 border border-slate-800 text-center">
|
||||
<span class="text-[10px] text-slate-400 uppercase block">GPU Util</span>
|
||||
<span id="gpu-util-val" class="text-xl font-bold text-cyan-400">0%</span>
|
||||
</div>
|
||||
<div class="p-3 rounded-xl bg-slate-950 border border-slate-800 text-center">
|
||||
<span class="text-[10px] text-slate-400 uppercase block">Temperature</span>
|
||||
<span id="gpu-temp-val" class="text-xl font-bold text-emerald-400">0°C</span>
|
||||
</div>
|
||||
<div class="p-3 rounded-xl bg-slate-950 border border-slate-800 text-center">
|
||||
<span class="text-[10px] text-slate-400 uppercase block">Power Draw</span>
|
||||
<span id="gpu-power-val" class="text-xl font-bold text-amber-400">0 W</span>
|
||||
</div>
|
||||
<div class="p-3 rounded-xl bg-slate-950 border border-slate-800 text-center">
|
||||
<span class="text-[10px] text-slate-400 uppercase block">Fan Speed</span>
|
||||
<span id="gpu-fan-val" class="text-xl font-bold text-slate-300">0%</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Process Table -->
|
||||
<div class="space-y-2">
|
||||
<span class="text-xs font-semibold text-slate-400 uppercase font-mono">Active Compute Processes</span>
|
||||
<div class="rounded-xl bg-slate-950 border border-slate-800 p-2 max-h-36 overflow-y-auto font-mono text-xs">
|
||||
<table class="w-full text-left">
|
||||
<thead>
|
||||
<tr class="text-slate-500 border-b border-slate-800/80 text-[10px]">
|
||||
<th class="pb-1">PID</th>
|
||||
<th class="pb-1">Process</th>
|
||||
<th class="pb-1 text-right">VRAM</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="gpu-proc-table" class="divide-y divide-slate-900">
|
||||
<tr><td colspan="3" class="py-2 text-center text-slate-500">Scanning GPU processes...</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- CARD 4: REAL-TIME SWITCH TIMELINE & OPTIMIZER -->
|
||||
<div class="bg-slate-900/80 border border-slate-800 rounded-2xl p-5 flex flex-col justify-between space-y-4">
|
||||
<div>
|
||||
<div class="flex items-center justify-between pb-3 border-b border-slate-800">
|
||||
<div class="flex items-center space-x-2">
|
||||
<div class="p-2 rounded-lg bg-indigo-950/80 border border-indigo-800 text-indigo-400">
|
||||
<i class="fa-solid fa-clock-rotate-left text-sm"></i>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="font-bold text-slate-100 text-sm">Model Switch Timeline & Optimizer</h3>
|
||||
<p class="text-xs text-slate-400">Real-time latency logger and memory warmer</p>
|
||||
</div>
|
||||
</div>
|
||||
<button onclick="warmAllModels()" class="px-2.5 py-1 text-xs font-semibold rounded-lg bg-amber-950/70 border border-amber-800 text-amber-300 hover:bg-amber-900 transition flex items-center space-x-1">
|
||||
<i class="fa-solid fa-fire text-amber-400"></i>
|
||||
<span>Warm All to RAM</span>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<!-- Switch Events Log -->
|
||||
<div class="mt-4 space-y-2">
|
||||
<span class="text-xs font-semibold text-slate-400 uppercase font-mono">Recent Model Swaps</span>
|
||||
<div id="switch-log-container" class="space-y-2 max-h-48 overflow-y-auto pr-1">
|
||||
<div class="p-3 rounded-xl bg-slate-950 border border-slate-800 text-xs text-slate-400 text-center font-mono">
|
||||
No recent model swaps recorded yet.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Optimizer Banner -->
|
||||
<div class="p-3 rounded-xl bg-gradient-to-r from-cyan-950/50 via-slate-950 to-purple-950/50 border border-slate-800 text-xs flex items-center justify-between">
|
||||
<div class="flex items-center space-x-2">
|
||||
<i class="fa-solid fa-wand-magic-sparkles text-cyan-400"></i>
|
||||
<span class="text-slate-300">64GB RAM Cache holds all models in memory</span>
|
||||
</div>
|
||||
<span class="font-mono text-cyan-400 font-bold">PCIe x16 (~31.5 GB/s)</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
</main>
|
||||
|
||||
<script src="/static/app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
28
static/styles.css
Normal file
28
static/styles.css
Normal file
@@ -0,0 +1,28 @@
|
||||
/* Custom Scrollbars */
|
||||
::-webkit-scrollbar {
|
||||
width: 6px;
|
||||
height: 6px;
|
||||
}
|
||||
::-webkit-scrollbar-track {
|
||||
background: #020617;
|
||||
}
|
||||
::-webkit-scrollbar-thumb {
|
||||
background: #1e293b;
|
||||
border-radius: 4px;
|
||||
}
|
||||
::-webkit-scrollbar-thumb:hover {
|
||||
background: #334155;
|
||||
}
|
||||
|
||||
@keyframes pulseGlow {
|
||||
0%, 100% {
|
||||
box-shadow: 0 0 15px rgba(6, 182, 212, 0.2);
|
||||
}
|
||||
50% {
|
||||
box-shadow: 0 0 25px rgba(6, 182, 212, 0.4);
|
||||
}
|
||||
}
|
||||
|
||||
.glow-card {
|
||||
animation: pulseGlow 4s infinite ease-in-out;
|
||||
}
|
||||
322
vram_arbitrator.py
Normal file
322
vram_arbitrator.py
Normal file
@@ -0,0 +1,322 @@
|
||||
"""VRAM Arbitrator and High-Speed Switch Manager for Ollama and ComfyUI."""
|
||||
import time
|
||||
import httpx
|
||||
import psutil
|
||||
import logging
|
||||
from typing import Dict, List, Any, Optional
|
||||
from collections import deque
|
||||
|
||||
try:
|
||||
import pynvml
|
||||
pynvml.nvmlInit()
|
||||
NVML_AVAILABLE = True
|
||||
except Exception as e:
|
||||
NVML_AVAILABLE = False
|
||||
|
||||
logger = logging.getLogger("vram_arbitrator")
|
||||
|
||||
OLLAMA_API_BASE = "http://localhost:11434"
|
||||
COMFY_API_BASE = "http://127.0.0.1:8188"
|
||||
|
||||
# Circular buffer for transition events
|
||||
SWITCH_HISTORY = deque(maxlen=50)
|
||||
|
||||
def get_gpu_hardware_stats() -> Dict[str, Any]:
|
||||
"""Retrieve comprehensive GPU hardware and process metrics via NVML."""
|
||||
if not NVML_AVAILABLE:
|
||||
return {"available": False, "error": "NVML not initialized"}
|
||||
|
||||
try:
|
||||
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
|
||||
name = pynvml.nvmlDeviceGetName(handle)
|
||||
if isinstance(name, bytes):
|
||||
name = name.decode("utf-8")
|
||||
|
||||
mem_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
|
||||
util_rates = pynvml.nvmlDeviceGetUtilizationRates(handle)
|
||||
temp_c = pynvml.nvmlDeviceGetTemperature(handle, pynvml.NVML_TEMPERATURE_GPU)
|
||||
try:
|
||||
power_mw = pynvml.nvmlDeviceGetPowerUsage(handle)
|
||||
power_w = round(power_mw / 1000.0, 1)
|
||||
except Exception:
|
||||
power_w = 0.0
|
||||
|
||||
try:
|
||||
fan_pct = pynvml.nvmlDeviceGetFanSpeed(handle)
|
||||
except Exception:
|
||||
fan_pct = 0
|
||||
|
||||
try:
|
||||
clock_graphics = pynvml.nvmlDeviceGetClockInfo(handle, pynvml.NVML_CLOCK_GRAPHICS)
|
||||
clock_mem = pynvml.nvmlDeviceGetClockInfo(handle, pynvml.NVML_CLOCK_MEM)
|
||||
except Exception:
|
||||
clock_graphics = 0
|
||||
clock_mem = 0
|
||||
|
||||
# Discover processes on GPU
|
||||
proc_breakdown = {
|
||||
"ollama_bytes": 0,
|
||||
"comfyui_bytes": 0,
|
||||
"system_bytes": 0,
|
||||
"processes": []
|
||||
}
|
||||
|
||||
try:
|
||||
procs = pynvml.nvmlDeviceGetComputeRunningProcesses(handle)
|
||||
graphics_procs = pynvml.nvmlDeviceGetGraphicsRunningProcesses(handle)
|
||||
all_procs = {p.pid: p.usedGpuMemory for p in procs}
|
||||
for p in graphics_procs:
|
||||
all_procs[p.pid] = max(all_procs.get(p.pid, 0), p.usedGpuMemory or 0)
|
||||
|
||||
for pid, used_mem in all_procs.items():
|
||||
pname = "Unknown"
|
||||
cmdline = ""
|
||||
try:
|
||||
proc = psutil.Process(pid)
|
||||
pname = proc.name()
|
||||
cmdline = " ".join(proc.cmdline())
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
is_ollama = "ollama" in pname.lower() or "llama-server" in cmdline.lower()
|
||||
is_comfy = "comfy" in cmdline.lower() or "main.py" in cmdline.lower()
|
||||
|
||||
if is_ollama:
|
||||
proc_breakdown["ollama_bytes"] += used_mem
|
||||
elif is_comfy:
|
||||
proc_breakdown["comfyui_bytes"] += used_mem
|
||||
else:
|
||||
proc_breakdown["system_bytes"] += used_mem
|
||||
|
||||
proc_breakdown["processes"].append({
|
||||
"pid": pid,
|
||||
"name": pname,
|
||||
"cmdline": cmdline[:60],
|
||||
"vram_bytes": used_mem,
|
||||
"vram_mb": round(used_mem / (1024**2), 1),
|
||||
"is_ollama": is_ollama,
|
||||
"is_comfy": is_comfy,
|
||||
})
|
||||
except Exception as e:
|
||||
logger.error(f"Error enumerating GPU processes: {e}")
|
||||
|
||||
total_vram = mem_info.total
|
||||
used_vram = mem_info.used
|
||||
free_vram = mem_info.free
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"device_name": name,
|
||||
"vram_total_bytes": total_vram,
|
||||
"vram_total_gb": round(total_vram / (1024**3), 2),
|
||||
"vram_used_bytes": used_vram,
|
||||
"vram_used_gb": round(used_vram / (1024**3), 2),
|
||||
"vram_free_bytes": free_vram,
|
||||
"vram_free_gb": round(free_vram / (1024**3), 2),
|
||||
"vram_used_pct": round((used_vram / total_vram * 100) if total_vram > 0 else 0, 1),
|
||||
"gpu_util_pct": util_rates.gpu,
|
||||
"mem_util_pct": util_rates.memory,
|
||||
"temperature_c": temp_c,
|
||||
"power_w": power_w,
|
||||
"fan_pct": fan_pct,
|
||||
"clock_graphics_mhz": clock_graphics,
|
||||
"clock_mem_mhz": clock_mem,
|
||||
"breakdown": {
|
||||
"ollama_mb": round(proc_breakdown["ollama_bytes"] / (1024**2), 1),
|
||||
"ollama_gb": round(proc_breakdown["ollama_bytes"] / (1024**3), 2),
|
||||
"comfyui_mb": round(proc_breakdown["comfyui_bytes"] / (1024**2), 1),
|
||||
"comfyui_gb": round(proc_breakdown["comfyui_bytes"] / (1024**3), 2),
|
||||
"system_mb": round(proc_breakdown["system_bytes"] / (1024**2), 1),
|
||||
"system_gb": round(proc_breakdown["system_bytes"] / (1024**3), 2),
|
||||
"free_mb": round(free_vram / (1024**2), 1),
|
||||
"free_gb": round(free_vram / (1024**3), 2),
|
||||
"processes": proc_breakdown["processes"],
|
||||
}
|
||||
}
|
||||
except Exception as e:
|
||||
return {"available": False, "error": str(e)}
|
||||
|
||||
async def get_ollama_live_state() -> Dict[str, Any]:
|
||||
"""Get active models, running status, and VRAM expiration from Ollama."""
|
||||
state = {
|
||||
"online": False,
|
||||
"loaded_models": [],
|
||||
"active_model_name": None,
|
||||
"active_model_vram_gb": 0.0,
|
||||
"active_context": 0,
|
||||
"expires_at": None,
|
||||
"installed_models": []
|
||||
}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=3.0) as client:
|
||||
# Check running models (ps)
|
||||
ps_resp = await client.get(f"{OLLAMA_API_BASE}/api/ps")
|
||||
if ps_resp.status_code == 200:
|
||||
state["online"] = True
|
||||
models = ps_resp.json().get("models", [])
|
||||
state["loaded_models"] = models
|
||||
if models:
|
||||
first = models[0]
|
||||
state["active_model_name"] = first.get("name")
|
||||
vram_bytes = first.get("size_vram", first.get("size", 0))
|
||||
state["active_model_vram_gb"] = round(vram_bytes / (1024**3), 2)
|
||||
state["active_context"] = first.get("context_length", 0)
|
||||
state["expires_at"] = first.get("expires_at")
|
||||
|
||||
# Check all tags
|
||||
tags_resp = await client.get(f"{OLLAMA_API_BASE}/api/tags")
|
||||
if tags_resp.status_code == 200:
|
||||
state["installed_models"] = tags_resp.json().get("models", [])
|
||||
except Exception as e:
|
||||
logger.debug(f"Ollama check error: {e}")
|
||||
|
||||
return state
|
||||
|
||||
async def get_comfyui_live_state() -> Dict[str, Any]:
|
||||
"""Get prompt queue, device status, and active execution from ComfyUI."""
|
||||
state = {
|
||||
"online": False,
|
||||
"executing": False,
|
||||
"queue_remaining": 0,
|
||||
"queue_running": 0,
|
||||
"current_node": None,
|
||||
"current_prompt_id": None,
|
||||
"vram_free_mb": 0,
|
||||
"vram_total_mb": 0,
|
||||
}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=3.0) as client:
|
||||
# Check system stats
|
||||
stats_resp = await client.get(f"{COMFY_API_BASE}/system_stats")
|
||||
if stats_resp.status_code == 200:
|
||||
state["online"] = True
|
||||
data = stats_resp.json()
|
||||
devices = data.get("devices", [])
|
||||
if devices:
|
||||
dev = devices[0]
|
||||
state["vram_free_mb"] = round(dev.get("vram_free", 0) / (1024**2), 1)
|
||||
state["vram_total_mb"] = round(dev.get("vram_total", 0) / (1024**2), 1)
|
||||
|
||||
# Check queue
|
||||
queue_resp = await client.get(f"{COMFY_API_BASE}/queue")
|
||||
if queue_resp.status_code == 200:
|
||||
qdata = queue_resp.json()
|
||||
running = qdata.get("queue_running", [])
|
||||
pending = qdata.get("queue_pending", [])
|
||||
state["queue_running"] = len(running)
|
||||
state["queue_remaining"] = len(pending)
|
||||
state["executing"] = len(running) > 0
|
||||
if running:
|
||||
state["current_prompt_id"] = running[0][1] if len(running[0]) > 1 else str(running[0])
|
||||
except Exception as e:
|
||||
logger.debug(f"ComfyUI check error: {e}")
|
||||
|
||||
return state
|
||||
|
||||
async def instant_free_ollama_vram(model_name: Optional[str] = None) -> Dict[str, Any]:
|
||||
"""Tell Ollama to instantly yield VRAM without evicting from OS page cache."""
|
||||
t0 = time.perf_counter()
|
||||
if not model_name:
|
||||
ollama_state = await get_ollama_live_state()
|
||||
model_name = ollama_state.get("active_model_name")
|
||||
|
||||
if not model_name:
|
||||
return {"success": True, "message": "No active Ollama model in VRAM", "duration_ms": 0}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
resp = await client.post(
|
||||
f"{OLLAMA_API_BASE}/api/generate",
|
||||
json={"model": model_name, "keep_alive": 0},
|
||||
)
|
||||
duration_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
event = {
|
||||
"timestamp": time.strftime("%H:%M:%S"),
|
||||
"event_type": "Ollama VRAM Yield",
|
||||
"source": model_name,
|
||||
"target": "VRAM 0MB (Kept in RAM)",
|
||||
"duration_ms": duration_ms,
|
||||
"cache_status": "RAM-Cached",
|
||||
}
|
||||
SWITCH_HISTORY.appendleft(event)
|
||||
return {"success": True, "model": model_name, "duration_ms": duration_ms}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
async def instant_free_comfyui_vram() -> Dict[str, Any]:
|
||||
"""Tell ComfyUI to purge loaded diffusion models from VRAM."""
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
resp = await client.post(
|
||||
f"{COMFY_API_BASE}/free",
|
||||
json={"unload_models": True, "free_memory": True},
|
||||
)
|
||||
duration_ms = round((time.perf_counter() - t0) * 1000, 2)
|
||||
event = {
|
||||
"timestamp": time.strftime("%H:%M:%S"),
|
||||
"event_type": "ComfyUI VRAM Purge",
|
||||
"source": "ComfyUI Pipeline",
|
||||
"target": "VRAM Free",
|
||||
"duration_ms": duration_ms,
|
||||
"cache_status": "Cleaned",
|
||||
}
|
||||
SWITCH_HISTORY.appendleft(event)
|
||||
return {"success": True, "duration_ms": duration_ms}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
async def switch_ollama_model(target_model: str, keep_alive: str = "30m") -> Dict[str, Any]:
|
||||
"""High-speed hot-swap to target Ollama model, tracking swap metrics."""
|
||||
t0 = time.perf_counter()
|
||||
cur_state = await get_ollama_live_state()
|
||||
prev_model = cur_state.get("active_model_name") or "None"
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=180.0) as client:
|
||||
resp = await client.post(
|
||||
f"{OLLAMA_API_BASE}/api/generate",
|
||||
json={"model": target_model, "prompt": "Ready check", "stream": False, "keep_alive": keep_alive},
|
||||
)
|
||||
total_duration = time.perf_counter() - t0
|
||||
total_duration_ms = round(total_duration * 1000, 2)
|
||||
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
load_dur_ms = round(data.get("load_duration", 0) / 1e6, 2)
|
||||
eval_dur_ms = round(data.get("eval_duration", 0) / 1e6, 2)
|
||||
eval_count = data.get("eval_count", 0)
|
||||
tokens_per_sec = round((eval_count / (eval_dur_ms / 1000)) if eval_dur_ms > 0 else 0, 1)
|
||||
|
||||
# Check if it was a RAM cache hit (load duration < 1500ms for large model indicates RAM hit)
|
||||
is_ram_hit = load_dur_ms < 2500
|
||||
|
||||
event = {
|
||||
"timestamp": time.strftime("%H:%M:%S"),
|
||||
"event_type": "LLM Model Switch",
|
||||
"source": prev_model,
|
||||
"target": target_model,
|
||||
"duration_ms": total_duration_ms,
|
||||
"load_duration_ms": load_dur_ms,
|
||||
"tokens_per_sec": tokens_per_sec,
|
||||
"cache_status": "RAM Cache Hit ⚡" if is_ram_hit else "Cold Disk Load 💾",
|
||||
}
|
||||
SWITCH_HISTORY.appendleft(event)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"prev_model": prev_model,
|
||||
"target_model": target_model,
|
||||
"total_duration_ms": total_duration_ms,
|
||||
"load_duration_ms": load_dur_ms,
|
||||
"tokens_per_sec": tokens_per_sec,
|
||||
"is_ram_hit": is_ram_hit,
|
||||
"response": data.get("response", ""),
|
||||
}
|
||||
else:
|
||||
return {"success": False, "error": f"HTTP {resp.status_code}: {resp.text}", "duration_ms": total_duration_ms}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "duration_ms": round((time.perf_counter() - t0) * 1000, 2)}
|
||||
|
||||
def get_switch_history() -> List[Dict[str, Any]]:
|
||||
return list(SWITCH_HISTORY)
|
||||
Reference in New Issue
Block a user