Make the verifier's results mean what they say
The first full run passed every stage, and three of those passes were worth less than they looked. Diffusion was reported as 0.67 it/s. That single run included loading the SDXL checkpoint from disk, so it understated throughput roughly tenfold against a steady-state 5.49 it/s. Cold and warm are now timed and labelled separately. The idle-purge stage checked the flag immediately, racing the ComfyUI websocket event that sets it, and reported "no purge pending" as a warning about ComfyUI rather than about its own timing. It now waits for the event. The reclaim stage passed while proving nothing: the model chosen was small enough to fit alongside ComfyUI's checkpoint, so the reclaim path never ran. It now picks a model that genuinely will not fit, and reports a warning rather than a pass when the path is not exercised. Sizing that model correctly took two corrections, both real. On-disk weight size is not the VRAM footprint -- a 12.87 GB blob occupies 14.9 GB once context and KV cache are allocated -- and unmanaged VRAM is not reclaimable, so it cannot count toward what a reclaim will free. Ignoring the second picked a model that failed even after a correct reclaim: the service returned 507 and logged "could not fit with ComfyUI holding 7.03 GB -- reclaiming and retrying", which was right. On this box an 842 MB third-party process is the difference between a 14.9 GB model fitting and not. The verifier also died on the 507 instead of reporting it, since a helper called raise_for_status() on responses a stage deliberately provokes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
27
README.md
27
README.md
@@ -185,6 +185,33 @@ which contradicts the hardware fails loudly rather than silently:
|
||||
| Busy yield | VRAM held at ≥50% GPU utilisation | A mid-generation model is finishing, not failing |
|
||||
| Residency confidence | probe trusted only at 100% | A 12-window probe once cleared 90% on a mostly-cold file |
|
||||
|
||||
## 1b. End-to-End Verification
|
||||
|
||||
```bash
|
||||
python verify_arbitration.py # full cycle, a few minutes
|
||||
python verify_arbitration.py --quick # skip the diffusion stages
|
||||
```
|
||||
|
||||
The unit suite covers logic in isolation. This exercises the promise the service exists
|
||||
to make — an LLM and a diffusion pipeline sharing one 16 GB card — against real hardware,
|
||||
and reports what actually happened at each stage. It restores what it changes and refuses
|
||||
to start if ComfyUI is busy.
|
||||
|
||||
A representative run on this machine:
|
||||
|
||||
| Stage | Result |
|
||||
| :--- | :--- |
|
||||
| LLM load, classified by achieved bandwidth | 1.96 GB in 1327 ms → 1.47 GB/s → Partial Cache |
|
||||
| VRAM yield confirmed against NVML | released in 43 ms, 2.39 GB freed |
|
||||
| Diffusion, cold (includes checkpoint load) | 17863 ms → 1.12 it/s |
|
||||
| Diffusion, warm | 3645 ms → **5.49 it/s** |
|
||||
| ComfyUI retains its checkpoint | 7.03 GB held through the idle window |
|
||||
| VRAM attribution adds up | 15.58 GB attributed vs 15.80 GB NVML — Ollama 7.71 + ComfyUI 7.03 coexisting |
|
||||
| Reported GPU state matches hardware | profile asks 320 W, card reports 320 W |
|
||||
|
||||
The stages report warnings rather than passes when they did not actually prove anything —
|
||||
a reclaim that was never needed is not evidence that reclaiming works.
|
||||
|
||||
## 2. Architectural Overview
|
||||
|
||||
```mermaid
|
||||
|
||||
Reference in New Issue
Block a user