Let any tenant declare its GPU profile and event source; fix priority semantics
Two remaining pieces of the two-application coupling are gone. Overclock profiles were switched by naming 'comfy' and 'ollama' directly, so a third application could never get tuned clocks. A tenant declares overclock_profile and the arbitrator applies whichever the highest-priority *working* tenant asks for, falling back to the idle profile when nothing is running. The websocket listener parsed ComfyUI's message schema -- status, execution_start, executing, execution_success -- which tied the fast path to one application. An event source is now declarative and the messages are not parsed at all: any message means "look now", and the tenant's own busy probe decides what is true. That gives the same sub-second reaction to any application that emits anything on state change, with no knowledge of what it emits. Generalising this exposed a design error in the priority rule I had introduced. plan_release excluded candidates ranking above the demander, which broke both directions in turn. With the LLM at priority 60 and diffusion at 50, ComfyUI could never reclaim from Ollama -- the premise the whole service is built on, and preserved until now only by the ComfyUI-specific trigger that was about to be removed. Swapping the ranks then broke the reverse: a starved Ollama could no longer reclaim from an idle ComfyUI. Priority now orders rather than vetoes. Any idle reclaimable tenant is a candidate, because an idle tenant is not using its VRAM; priority decides who is asked first, and busy tenants are never interrupted whatever their rank. Diffusion outranks the LLM, whose weights reload from page cache in seconds. All three cases are pinned by tests, including that busy work is never interrupted even by a far higher-priority demander. Tests: 244 (was 242). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -224,10 +224,42 @@ class TestReleasePlanning:
|
||||
assert blockers["stt-relay"] == "declares no release mechanism"
|
||||
assert plan["possible"] is False
|
||||
|
||||
def test_higher_priority_tenants_are_not_victims(self):
|
||||
def test_an_idle_tenant_yields_even_if_it_outranks_the_demander(self):
|
||||
"""Priority orders who is asked first; it does not protect idle memory.
|
||||
|
||||
Filtering candidates by priority broke both directions in turn: with the LLM
|
||||
ranked above diffusion, ComfyUI could never preempt Ollama -- the service's
|
||||
central behaviour -- and once the ranks were swapped, a starved Ollama could no
|
||||
longer reclaim from an idle ComfyUI. An idle tenant is not using its VRAM, so
|
||||
outranking the demander is not a reason to keep it.
|
||||
"""
|
||||
state = self._state(comfyui={"priority": 99, "busy": False})
|
||||
plan = T.plan_release("ollama", state, free_gb=1.0, needed_gb=14.9)
|
||||
assert "comfyui" not in plan["release"]
|
||||
assert plan["release"] == ["comfyui"]
|
||||
|
||||
def test_priority_decides_who_is_asked_first(self):
|
||||
state = [
|
||||
{"name": "demander", "priority": 50, "vram_gb": 0.0, "busy": True,
|
||||
"reclaimable": True},
|
||||
{"name": "high", "priority": 90, "vram_gb": 4.0, "busy": False,
|
||||
"reclaimable": True},
|
||||
{"name": "low", "priority": 10, "vram_gb": 4.0, "busy": False,
|
||||
"reclaimable": True},
|
||||
]
|
||||
plan = T.plan_release("demander", state, free_gb=0.0, needed_gb=5.0)
|
||||
# The lowest-priority idle tenant gives up memory first.
|
||||
assert plan["release"][0] == "low"
|
||||
|
||||
def test_busy_work_is_never_interrupted_whatever_the_priority(self):
|
||||
state = [
|
||||
{"name": "demander", "priority": 99, "vram_gb": 0.0, "busy": True,
|
||||
"reclaimable": True},
|
||||
{"name": "worker", "priority": 1, "vram_gb": 8.0, "busy": True,
|
||||
"reclaimable": True},
|
||||
]
|
||||
plan = T.plan_release("demander", state, free_gb=0.0, needed_gb=8.0)
|
||||
assert plan["release"] == []
|
||||
assert plan["blockers"][0]["why"] == "busy"
|
||||
|
||||
def test_lowest_priority_is_released_first(self):
|
||||
state = self._state() + [
|
||||
|
||||
Reference in New Issue
Block a user