Make the verifier's results mean what they say

The first full run passed every stage, and three of those passes were worth less
than they looked.

Diffusion was reported as 0.67 it/s. That single run included loading the SDXL
checkpoint from disk, so it understated throughput roughly tenfold against a
steady-state 5.49 it/s. Cold and warm are now timed and labelled separately.

The idle-purge stage checked the flag immediately, racing the ComfyUI websocket
event that sets it, and reported "no purge pending" as a warning about ComfyUI
rather than about its own timing. It now waits for the event.

The reclaim stage passed while proving nothing: the model chosen was small enough to
fit alongside ComfyUI's checkpoint, so the reclaim path never ran. It now picks a
model that genuinely will not fit, and reports a warning rather than a pass when the
path is not exercised.

Sizing that model correctly took two corrections, both real. On-disk weight size is
not the VRAM footprint -- a 12.87 GB blob occupies 14.9 GB once context and KV cache
are allocated -- and unmanaged VRAM is not reclaimable, so it cannot count toward
what a reclaim will free. Ignoring the second picked a model that failed even after a
correct reclaim: the service returned 507 and logged "could not fit with ComfyUI
holding 7.03 GB -- reclaiming and retrying", which was right. On this box an 842 MB
third-party process is the difference between a 14.9 GB model fitting and not.

The verifier also died on the 507 instead of reporting it, since a helper called
raise_for_status() on responses a stage deliberately provokes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
drjones
2026-09-06 17:31:48 -07:00
parent aed1c360f0
commit 81e5d88426
2 changed files with 121 additions and 17 deletions

View File

@@ -185,6 +185,33 @@ which contradicts the hardware fails loudly rather than silently:
| Busy yield | VRAM held at ≥50% GPU utilisation | A mid-generation model is finishing, not failing |
| Residency confidence | probe trusted only at 100% | A 12-window probe once cleared 90% on a mostly-cold file |
## 1b. End-to-End Verification
```bash
python verify_arbitration.py # full cycle, a few minutes
python verify_arbitration.py --quick # skip the diffusion stages
```
The unit suite covers logic in isolation. This exercises the promise the service exists
to make — an LLM and a diffusion pipeline sharing one 16 GB card — against real hardware,
and reports what actually happened at each stage. It restores what it changes and refuses
to start if ComfyUI is busy.
A representative run on this machine:
| Stage | Result |
| :--- | :--- |
| LLM load, classified by achieved bandwidth | 1.96 GB in 1327 ms → 1.47 GB/s → Partial Cache |
| VRAM yield confirmed against NVML | released in 43 ms, 2.39 GB freed |
| Diffusion, cold (includes checkpoint load) | 17863 ms → 1.12 it/s |
| Diffusion, warm | 3645 ms → **5.49 it/s** |
| ComfyUI retains its checkpoint | 7.03 GB held through the idle window |
| VRAM attribution adds up | 15.58 GB attributed vs 15.80 GB NVML — Ollama 7.71 + ComfyUI 7.03 coexisting |
| Reported GPU state matches hardware | profile asks 320 W, card reports 320 W |
The stages report warnings rather than passes when they did not actually prove anything —
a reclaim that was never needed is not evidence that reclaiming works.
## 2. Architectural Overview
```mermaid