The health check now reports what unmanaged VRAM actually costs rather than just
how much of it there is: "0.82 GB held by python (842 MB)" becomes "5 model(s) fit
within 15.42 GB but not the 14.60 GB actually available", naming them.
Getting that arithmetic right took a correction. The first version subtracted only
the desktop and the unmanaged process, and so reported a 14.93 GB model as fitting
against a real ceiling of 14.60 GB -- the same model the service had just refused
with 507. ComfyUI keeps a few hundred MB of CUDA context for as long as the process
lives, which a purge does not free, so it is not available either. The floor is taken
from the minimum ComfyUI VRAM in recent telemetry rather than its current value,
which could be a 7 GB checkpoint mid-generation.
The verifier's reclaim stage now re-runs a graph immediately beforehand to reset the
30 s idle window, since a large model takes longer than that to load and the purge
was freeing ComfyUI mid-load, so the reclaim path was never reached.
Tests: 199 (was 192). The new ones pin the ceiling arithmetic, including that a model
too large to fit on the card at all is not blamed on the third-party process.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first full run passed every stage, and three of those passes were worth less
than they looked.
Diffusion was reported as 0.67 it/s. That single run included loading the SDXL
checkpoint from disk, so it understated throughput roughly tenfold against a
steady-state 5.49 it/s. Cold and warm are now timed and labelled separately.
The idle-purge stage checked the flag immediately, racing the ComfyUI websocket
event that sets it, and reported "no purge pending" as a warning about ComfyUI
rather than about its own timing. It now waits for the event.
The reclaim stage passed while proving nothing: the model chosen was small enough to
fit alongside ComfyUI's checkpoint, so the reclaim path never ran. It now picks a
model that genuinely will not fit, and reports a warning rather than a pass when the
path is not exercised.
Sizing that model correctly took two corrections, both real. On-disk weight size is
not the VRAM footprint -- a 12.87 GB blob occupies 14.9 GB once context and KV cache
are allocated -- and unmanaged VRAM is not reclaimable, so it cannot count toward
what a reclaim will free. Ignoring the second picked a model that failed even after a
correct reclaim: the service returned 507 and logged "could not fit with ComfyUI
holding 7.03 GB -- reclaiming and retrying", which was right. On this box an 842 MB
third-party process is the difference between a 14.9 GB model fitting and not.
The verifier also died on the 507 instead of reporting it, since a helper called
raise_for_status() on responses a stage deliberately provokes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The unit suite covers logic in isolation, but the promise this service exists to make
-- an LLM and a diffusion pipeline sharing one 16 GB card without either failing --
had only ever been checked by hand, piecemeal. verify_arbitration.py walks the whole
cycle against real hardware and reports what happened at each stage: load and its
bandwidth classification, the confirmed yield, a real SDXL graph, the deferred idle
purge, reclaim-and-retry, and finally whether VRAM attribution adds up and the
reported GPU state still matches the card. It restores what it changes and refuses
to start if ComfyUI is busy. Kept out of pytest deliberately: it moves real VRAM and
takes minutes.
Running it immediately found two bugs.
/api/switch-model reported every upstream failure as 500. Asking an embedding model
to generate makes Ollama return 400 -- the request is unusable, the service is fine
-- and calling that an Internal Server Error blames this service for the caller's
mistake. Failures now map to 400 for an upstream client error, 507 for a model that
will not fit (valid request, healthy service, no room), and 502 when Ollama itself
errors.
The verifier also picked the smallest installed model, which here is
nomic-embed-text -- an embedding model with no generate endpoint. It now filters
those out by family and name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
