drjones/gpu-program-swapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stop a stale ComfyUI queue entry from disabling half the arbitration

Browse Source 
Chasing why the reverse-direction reclaim never fired turned up something worse than
the reclaim itself.

The starvation check was never running. Instrumenting the watchdog showed busy=6,
idle_check=0: every poll took the "ComfyUI is busy" branch. ComfyUI's /queue was
reporting a WAN 2.1 i2v job in queue_running while the GPU sat at 0% and ComfyUI held
0.56 GB. The job was dead; ComfyUI had simply never cleared the row.

Believing that flag meant this service thought ComfyUI was permanently busy, so it
yielded the LLM's VRAM on every poll, never ran the idle purge, and never checked
whether the LLM had been squeezed onto the CPU. One stale row disabled half of the
arbitration, and it very likely explains the earlier burst of yields against a
cron-driven model.

A running entry is now corroborated before it is believed. The first attempt used GPU
utilisation, which does not work: utilisation is shared with Ollama and with the
third-party process on this box, so peak utilisation stayed above any sensible
threshold and a stuck entry never looked stale. ComfyUI's own VRAM is the right
signal -- a real diffusion job loads gigabytes of checkpoint, a dead one holds only
its CUDA context. After the fix the same watchdog reports busy=3, idle_check=32.

Every early return in the starvation check now records why it bailed, because with
four of them there was no way to tell which had fired. /api/health reports a stale
queue entry with its impact and how to clear it.

Also confirmed, contradicting an earlier conclusion in this branch: Ollama on this box
*does* spill to the CPU. smtek/Qwen3.8-27B:Q2_K_XL held steady at 29.2% on GPU
(size=15.59 GB, size_vram=4.56 GB) across twelve seconds of polling -- a stable
placement, not a progressive load. Both failure modes are real; which one occurs
depends on the model.

Tests: 206 (was 199).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
drjones
2026-09-07 08:48:37 -07:00
parent aeba1b47fd
commit c3d9b36035
6 changed files with 218 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
README.md
View File

@@ -21,6 +21,15 @@
### ⚡ Bidirectional VRAM Hot-Swapping & Arbitration
* **A stale ComfyUI queue entry no longer disables arbitration.** ComfyUI can leave a
dead job in `queue_running` indefinitely; one was found sitting there with the GPU idle
and ComfyUI holding 0.56 GB. Trusting that flag made this service believe ComfyUI was
permanently busy — so it evicted the LLM on every poll, never ran the idle purge, and
never checked for CPU spill. Instrumenting the watchdog showed `busy=6, idle_check=0`.
A running entry is now corroborated against ComfyUI's own VRAM (a real job loads
gigabytes; a dead one holds only its CUDA context) before it is believed, and a stale
entry is reported by `/api/health`. Utilisation is deliberately *not* the signal — it is
shared with Ollama and any third-party process.
* **Both directions are now automatic.** Yielding Ollama for ComfyUI always was; the
reverse was not, despite "bidirectional" in this heading. Which way an LLM fails when
it cannot fit depends on configuration: with `n_gpu_layers` left to Ollama it spills
@@ -168,7 +177,7 @@ only if the card actually needs it.
## 1a. Tests
```bash
/home/drjones/comfy-mcp-venv/bin/python -m pytest tests/ -q # 192 passed in ~3.7s
/home/drjones/comfy-mcp-venv/bin/python -m pytest tests/ -q # 206 passed in ~3.7s
```
Hermetic: no GPU, no network, no sleeps. An autouse fixture stubs `overclock_manager._sh`