Tune profiles from measurement; add a diffusion benchmark to close the loop

The profiles were hand-written and had never been checked against the hardware. Adding
a ComfyUI benchmark alongside the existing decode one made the compute side measurable
for the first time, and most of what the profiles configured turned out to do nothing.

Measured on this card (RTX 4080 SUPER, driver 595.84):

- LLM decode is not power-bound: 73.0-73.5 tok/s flat from 222W to 370W, with the card
  never drawing more than 224W at any limit. The ollama profile's 370W did nothing.
- Diffusion is power-bound: 5.48 it/s @222W rising to 6.71 @370W, so comfy's 370W is
  worth a real +2.8% over the 320W stock default.
- Clock locks did nothing for either workload: 72.6 tok/s locked at 11251MHz vs 72.7
  unlocked; 6.77 it/s locked at 3105MHz vs 6.73 unlocked, and 6.78 at 2400MHz.
- Memory bandwidth is still the decode bottleneck (5001MHz halves throughput to 35.9
  tok/s), confirming the profile's premise -- the card just gets there unaided.
- Fans: 48,435 samples show 81C all-time max and zero thermal throttle events, while
  the ollama profile held 49.6C average by running fans at 87%. All profiles now use
  automatic fans and let the thermal governor escalate on demand.

Code changes supporting that:
- _diffusion_benchmark() queues a fixed SDXL graph via ComfyUI's API. The seed must
  vary per run: ComfyUI caches by node inputs, so a fixed seed returned in ~1ms without
  executing. Implausibly fast results are now rejected as cache hits rather than
  recorded as record scores.
- The arbitrator's automatic profile switching is suspended during a sweep. A diffusion
  benchmark trips trigger_comfy_priority, which reapplies the whole profile and would
  silently overwrite the clock being measured.
- _supported_clocks() queries the mem,gr pair; asking for a single field returned one
  column and reading index 1 yielded an empty list rather than an error. Graphics clocks
  are subsampled (the card enumerates 194 of them) and lock sweeps include an explicit
  unlocked control step.
- offsets_supported() probes once and apply_profile skips inert offset levers with an
  explanation instead of pretending they applied.
- Profiles carry a 'measured' field recording the evidence behind each setting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
drjones
2026-08-28 11:18:35 -07:00
parent a30444ef8e
commit c689ec8711
6 changed files with 373 additions and 56 deletions

View File

@@ -67,6 +67,46 @@ DEFAULT_PROFILES: Dict[str, Dict[str, Any]] = {
},
}
_OFFSETS_SUPPORTED: Optional[bool] = None
def offsets_supported(recheck: bool = False) -> bool:
"""Whether nvidia-settings clock offsets actually take effect on this driver.
Driver 595.84 accepts GPUGraphicsClockOffset/GPUMemoryTransferRateOffset and silently
discards them: assigning 0 returns success and the attribute still reads back its old
value. Profiles carrying core_offset_mhz/mem_offset_mhz were therefore configuring
nothing. Probed once and cached.
"""
global _OFFSETS_SUPPORTED
if _OFFSETS_SUPPORTED is not None and not recheck:
return _OFFSETS_SUPPORTED
def _read() -> Optional[int]:
q = _nvidia_settings("-q", "[gpu:0]/GPUGraphicsClockOffset[3]")
for line in (q.get("out") or "").splitlines():
if "Attribute" in line and "):" in line:
try:
return int(line.split("):")[-1].split(".")[0].strip())
except Exception:
pass
return None
before = _read()
if before is None:
_OFFSETS_SUPPORTED = False
return False
probe = before + 25
_nvidia_settings("-a", f"[gpu:0]/GPUGraphicsClockOffset[3]={probe}")
after = _read()
_nvidia_settings("-a", f"[gpu:0]/GPUGraphicsClockOffset[3]={before}")
_OFFSETS_SUPPORTED = (after is not None and after != before)
if not _OFFSETS_SUPPORTED:
logger.warning("Clock offsets are not honoured by this driver "
f"(set {probe}, read back {after}); profile offset fields are inert.")
return _OFFSETS_SUPPORTED
ACTIVE_PROFILE = "balanced"
_LAST_RESULT: Dict[str, Any] = {}
FAN_MANUAL = False
@@ -231,7 +271,11 @@ def apply_profile(name: str, overrides: Optional[Dict[str, Any]] = None) -> Dict
"power_limit": _apply_power_limit(int(cfg.get("power_limit_w", 370))),
"clock_lock": _apply_clock_lock(int(cfg.get("lock_core_min", 0)), int(cfg.get("lock_core_max", 0))),
"mem_lock": _apply_mem_lock(int(cfg.get("lock_mem_mhz", 0))),
"offsets": _apply_offsets(int(cfg.get("core_offset_mhz", 0)), int(cfg.get("mem_offset_mhz", 0))),
"offsets": (_apply_offsets(int(cfg.get("core_offset_mhz", 0)),
int(cfg.get("mem_offset_mhz", 0)))
if offsets_supported() else
{"applied": False, "supported": False,
"detail": "skipped: this driver accepts clock offsets and ignores them"}),
"fan": apply_fan_control(fan_mode, fan_speed),
}
result["gpu"] = get_gpu_state()
@@ -382,6 +426,9 @@ def get_status() -> Dict[str, Any]:
"""Full overclock status for the dashboard."""
return {
"active_profile": ACTIVE_PROFILE,
"offsets_supported": offsets_supported(),
"effective_levers": (["power_limit", "clock_lock", "mem_lock", "fan"]
+ (["offsets"] if offsets_supported() else [])),
"profiles": load_profiles(),
"gpu": get_gpu_state(),
"fan": get_fan_status(),