Let a higher-priority job preempt lower-priority work in progress

Measured failure: a diffusion job took 46 s instead of 3 s, squeezed into 1.6 GB.
The LLM yielded correctly, an inference reloaded it two seconds later, and
plan_release then refused to touch it because it was "busy" -- so ComfyUI crawled
while Ollama held 13 GB for the whole run.

"Never interrupt busy work" looks like the safe rule and is not. Preempting a
lower-priority tenant is safe precisely because releasing is asynchronous: an Ollama
unload queues behind its running request and applies when that finishes, so nothing
is killed mid-flight. That is what makes fast handoff possible at all, and refusing
to do it defeats the purpose of the service.

A tenant may now be asked for memory if it is idle, whatever its rank, or if it is
busy and ranks strictly below the demander. Equal or higher priority is never
interrupted, so peers cannot fight. Idle tenants are still preferred over preempting
busy ones.

Verified against the real contention: with Ollama at 12.38 GB and 98% utilisation, a
diffusion job released it within three seconds and completed in 18 s rather than 46 s.

Tests updated to the corrected rule, and the fixture's priorities aligned with what
actually ships -- it still had the LLM outranking diffusion from before that was
swapped.

Tests: 246.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tenants.py

@@ -394,22 +394,32 @@ def plan_release(demanding: str, tenants_state: List[Dict[str, Any]],
         return {"possible": True, "reason": "enough VRAM is already free",
                 "release": [], "shortfall_gb": 0.0}
 
-    # Any idle reclaimable tenant is a candidate, whatever its priority. An idle tenant
-    # is not using its VRAM, so ranking above the demander should not protect it -- an
-    # earlier version filtered on priority and thereby broke both directions in turn:
-    # ComfyUI could not preempt Ollama, and once that was corrected a starved Ollama
-    # could no longer reclaim from an idle ComfyUI.
+    # Who may be asked for memory:
     #
-    # Priority decides who is asked *first* (lowest gives up memory soonest) and, being
-    # applied only to idle tenants, never interrupts work.
+    #   * any idle reclaimable tenant, whatever its rank -- idle memory is not in use;
+    #   * a *busy* tenant that ranks strictly below the demander.
+    #
+    # That second clause is the point of the whole service and was nearly lost. Refusing
+    # to touch anything busy looks safe and is not: a diffusion job measured here ran for
+    # 46 s instead of 3 s, squeezed into 1.6 GB, because the LLM reloaded straight after
+    # yielding and was then protected as "busy" while ComfyUI starved. Preempting a
+    # lower-priority tenant is safe precisely because releasing is asynchronous -- an
+    # Ollama unload queues behind its running request and applies when that finishes, so
+    # nothing is killed mid-flight.
+    #
+    # Equal or higher priority is never interrupted, so peers cannot fight.
+    demander_priority = demander.get("priority", 0)
     candidates = [
         s for s in tenants_state
         if s["name"] != demanding
         and s.get("reclaimable")
-        and not s.get("busy")
         and s.get("vram_gb", 0) > 0
+        and (not s.get("busy") or s.get("priority", 0) < demander_priority)
     ]
-    candidates.sort(key=lambda s: (s.get("priority", 0), -s.get("vram_gb", 0)))
+    # Idle tenants first, then lowest priority: never disturb working software while
+    # something idle still has memory to give.
+    candidates.sort(key=lambda s: (bool(s.get("busy")), s.get("priority", 0),
+                                   -s.get("vram_gb", 0)))
 
     plan, freed = [], 0.0
     for c in candidates:
@@ -420,7 +430,7 @@ def plan_release(demanding: str, tenants_state: List[Dict[str, Any]],
 
     blockers = [
         {"name": s["name"], "vram_gb": s.get("vram_gb", 0.0),
-         "why": ("busy" if s.get("busy") else
+         "why": ("busy and ranks at or above the demander" if s.get("busy") else
                  "declares no release mechanism" if not s.get("reclaimable") else
                  "enough was freed without it")}
         for s in tenants_state