Tune profiles from measurement; add a diffusion benchmark to close the loop

The profiles were hand-written and had never been checked against the hardware. Adding
a ComfyUI benchmark alongside the existing decode one made the compute side measurable
for the first time, and most of what the profiles configured turned out to do nothing.

Measured on this card (RTX 4080 SUPER, driver 595.84):

- LLM decode is not power-bound: 73.0-73.5 tok/s flat from 222W to 370W, with the card
  never drawing more than 224W at any limit. The ollama profile's 370W did nothing.
- Diffusion is power-bound: 5.48 it/s @222W rising to 6.71 @370W, so comfy's 370W is
  worth a real +2.8% over the 320W stock default.
- Clock locks did nothing for either workload: 72.6 tok/s locked at 11251MHz vs 72.7
  unlocked; 6.77 it/s locked at 3105MHz vs 6.73 unlocked, and 6.78 at 2400MHz.
- Memory bandwidth is still the decode bottleneck (5001MHz halves throughput to 35.9
  tok/s), confirming the profile's premise -- the card just gets there unaided.
- Fans: 48,435 samples show 81C all-time max and zero thermal throttle events, while
  the ollama profile held 49.6C average by running fans at 87%. All profiles now use
  automatic fans and let the thermal governor escalate on demand.

Code changes supporting that:
- _diffusion_benchmark() queues a fixed SDXL graph via ComfyUI's API. The seed must
  vary per run: ComfyUI caches by node inputs, so a fixed seed returned in ~1ms without
  executing. Implausibly fast results are now rejected as cache hits rather than
  recorded as record scores.
- The arbitrator's automatic profile switching is suspended during a sweep. A diffusion
  benchmark trips trigger_comfy_priority, which reapplies the whole profile and would
  silently overwrite the clock being measured.
- _supported_clocks() queries the mem,gr pair; asking for a single field returned one
  column and reading index 1 yielded an empty list rather than an error. Graphics clocks
  are subsampled (the card enumerates 194 of them) and lock sweeps include an explicit
  unlocked control step.
- offsets_supported() probes once and apply_profile skips inert offset levers with an
  explanation instead of pretending they applied.
- Profiles carry a 'measured' field recording the evidence behind each setting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

server.py

@@ -3,6 +3,7 @@ import asyncio
 import contextlib
 import json
 import logging
+import signal
 import time
 from contextlib import asynccontextmanager
 from typing import Dict, Any, Optional, List, Set
@@ -64,6 +65,13 @@ class TelemetryBroker:
         self.running = True
         self.task = asyncio.create_task(self._loop())
 
+    def begin_shutdown(self) -> None:
+        """Release every SSE subscriber. Safe to call from a signal handler."""
+        self.closing = True
+        for q in list(self.subscribers):
+            with contextlib.suppress(asyncio.QueueFull):
+                q.put_nowait(None)
+
     async def stop(self) -> None:
         self.running = False
         self.closing = True
@@ -150,11 +158,36 @@ class TelemetryBroker:
 broker = TelemetryBroker()
 
 
+def _install_shutdown_hook() -> None:
+    """Close SSE streams the moment a shutdown signal arrives.
+
+    uvicorn runs the lifespan shutdown only after it has finished waiting on open
+    connections, so releasing subscribers from there is too late: the streams keep the
+    server busy until the graceful timeout expires and every one of them is force
+    cancelled, which logs a CancelledError traceback apiece. Chaining onto the existing
+    signal handler lets us drain them first and leaves uvicorn's own shutdown intact.
+    """
+    loop = asyncio.get_running_loop()
+    for sig in (signal.SIGTERM, signal.SIGINT):
+        previous = signal.getsignal(sig)
+
+        def handler(signum, frame, _prev=previous):
+            broker.begin_shutdown()
+            if callable(_prev):
+                _prev(signum, frame)
+
+        try:
+            signal.signal(sig, handler)
+        except (ValueError, OSError):
+            pass  # not on the main thread; the lifespan path still cleans up
+
+
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     telemetry_store.start()
     await broker.start()
     await vram_arbitrator.arbitrator.start()
+    _install_shutdown_hook()
     yield
     await vram_arbitrator.arbitrator.stop()
     await broker.stop()
@@ -218,13 +251,16 @@ class GovernorRequest(BaseModel):
     reset: bool = Field(False, description="Clear any active derate and reapply the full profile")
 
 class SweepRequest(BaseModel):
-    knob: str = Field("mem_offset_mhz", description="mem_offset_mhz | core_offset_mhz")
+    knob: str = Field("mem_offset_mhz", description="mem_offset_mhz | core_offset_mhz | lock_mem_mhz | lock_core_max")
     profile: str = Field("ollama", description="Profile to tune")
+    workload: str = Field("auto", description="ollama (decode tok/s) | comfy (diffusion it/s) | auto")
     model: Optional[str] = Field(None, description="Model to benchmark with; defaults to the loaded one")
     start: Optional[int] = Field(None, description="First offset value")
     stop: Optional[int] = Field(None, description="Last offset value")
     step: Optional[int] = Field(None, description="Offset increment")
     repeats: int = Field(1, description="Benchmark runs per step")
+    max_steps: int = Field(6, description="Cap on swept values for discrete clock knobs")
+    include_unlocked: bool = Field(True, description="Include an unlocked (0) control step")
     apply_best: bool = Field(False, description="Write the winning value into the profile")
 
 class RequestVramRequest(BaseModel):
@@ -263,7 +299,7 @@ async def sse_telemetry_stream(request: Request):
                 if await request.is_disconnected():
                     break
                 try:
-                    snap = await asyncio.wait_for(q.get(), timeout=15.0)
+                    snap = await asyncio.wait_for(q.get(), timeout=5.0)
                     if snap is None:      # shutdown sentinel
                         break
                     yield f"data: {json.dumps(snap)}\n\n"
@@ -483,9 +519,10 @@ async def api_autotune_status():
 async def api_autotune_sweep(req: SweepRequest):
     """Walk a clock offset upward, measuring tok/s and watching for instability at each step."""
     res = await autotune.sweep(
-        knob=req.knob, profile=req.profile, model=req.model,
+        knob=req.knob, profile=req.profile, workload=req.workload, model=req.model,
         start=req.start, stop=req.stop, step=req.step,
-        repeats=req.repeats, apply_best=req.apply_best,
+        repeats=req.repeats, max_steps=req.max_steps,
+        include_unlocked=req.include_unlocked, apply_best=req.apply_best,
     )
     if not res.get("success"):
         raise HTTPException(status_code=400, detail=res.get("error"))