Stop a stale ComfyUI queue entry from disabling half the arbitration

Chasing why the reverse-direction reclaim never fired turned up something worse than
the reclaim itself.

The starvation check was never running. Instrumenting the watchdog showed busy=6,
idle_check=0: every poll took the "ComfyUI is busy" branch. ComfyUI's /queue was
reporting a WAN 2.1 i2v job in queue_running while the GPU sat at 0% and ComfyUI held
0.56 GB. The job was dead; ComfyUI had simply never cleared the row.

Believing that flag meant this service thought ComfyUI was permanently busy, so it
yielded the LLM's VRAM on every poll, never ran the idle purge, and never checked
whether the LLM had been squeezed onto the CPU. One stale row disabled half of the
arbitration, and it very likely explains the earlier burst of yields against a
cron-driven model.

A running entry is now corroborated before it is believed. The first attempt used GPU
utilisation, which does not work: utilisation is shared with Ollama and with the
third-party process on this box, so peak utilisation stayed above any sensible
threshold and a stuck entry never looked stale. ComfyUI's own VRAM is the right
signal -- a real diffusion job loads gigabytes of checkpoint, a dead one holds only
its CUDA context. After the fix the same watchdog reports busy=3, idle_check=32.

Every early return in the starvation check now records why it bailed, because with
four of them there was no way to tell which had fired. /api/health reports a stale
queue entry with its impact and how to clear it.

Also confirmed, contradicting an earlier conclusion in this branch: Ollama on this box
*does* spill to the CPU. smtek/Qwen3.8-27B:Q2_K_XL held steady at 29.2% on GPU
(size=15.59 GB, size_vram=4.56 GB) across twelve seconds of polling -- a stable
placement, not a progressive load. Both failure modes are real; which one occurs
depends on the model.

Tests: 206 (was 199).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

health.py

@@ -117,6 +117,25 @@ def _check_comfy_ws() -> Dict[str, Any]:
     return _check("comfyui websocket", OK, "subscribed")
 
 
+def _check_comfy_queue() -> Dict[str, Any]:
+    """A stale ComfyUI queue entry disables half of this service's logic."""
+    arb = vram_arbitrator.arbitrator
+    if arb.comfy_stale_job:
+        return _check("comfyui queue", DEGRADED,
+                      f"prompt {arb.comfy_stale_job} claims to be running but the GPU is idle",
+                      "ComfyUI looks permanently busy, so the LLM is evicted repeatedly, "
+                      "the idle purge never runs and CPU-spill is never checked",
+                      "Clear it from the ComfyUI queue, or POST /queue with "
+                      "{\"clear\": true} to ComfyUI")
+    branches = arb.watchdog_branches
+    if branches.get("idle_check", 0) == 0 and branches.get("busy", 0) > 20:
+        return _check("comfyui queue", DEGRADED,
+                      "the watchdog has only ever seen ComfyUI as busy",
+                      "The idle purge and starvation check are not running",
+                      "Check the ComfyUI queue for a stuck entry")
+    return _check("comfyui queue", OK, "queue state corroborated against GPU activity")
+
+
 def _check_store() -> Dict[str, Any]:
     info = telemetry_store.db_info()
     if not info.get("exists"):