Fix two more stale-cache bugs; account for VRAM this service cannot reclaim
The stale-readback bug fixed in f5917a0 was a class, not an instance. Two more:
- ram_optimizer's residency report cached for 15s and was never invalidated when
anything warmed a file, so warming a model and then looking at residency showed
the state from before the warm. warm_file_to_ram now invalidates it.
- _PID_KIND_CACHE was keyed on pid alone and never expired. Linux recycles PIDs, so
a stale entry could attribute a new process's VRAM to Ollama or ComfyUI -- inside
the very snapshot the yield barrier trusts to decide whether VRAM was released.
Now keyed by (pid, process start time) and bounded.
Unmanaged VRAM. Investigating a persistence-mode warning turned up a third GPU
consumer this service does not model: stt_relay.py, holding 842 MB for nearly three
days. It was bucketed as "system" alongside gnome-shell's 3.9 MB. That conflation
matters, because ComfyUI's memory can be reclaimed and a third party's cannot, and
the reclaim path assumed ComfyUI was always to blame for missing headroom.
Processes are now bucketed ollama | comfy | desktop | unmanaged. The breakdown
reports desktop_gb and unmanaged_gb separately and names the unmanaged processes;
when a reclaim-and-retry still fails, the error identifies them rather than
implying ComfyUI was at fault; and the dashboard shows the unreclaimable total, so
headroom the arbitrator can never give back is visible rather than inferred.
Checked and deliberately not changed: persistence mode reads Disabled, but
nvidia-persistenced is active and two clients hold the GPU open continuously, so
the driver never unloads. The nvidia-smi warning is legacy noise here and is not a
source of the profile drift.
Tests: 169 (was 164). The new ones cover the bucketing, and one existing test used
Xorg as its "unknown process" fixture -- correct before a display server had its own
bucket, wrong after.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -35,7 +35,7 @@ function updateDashboard(data) {
|
||||
|
||||
// Governor and arbitration state ride along in the shared snapshot.
|
||||
if (data.governor) renderGovernor(data.governor);
|
||||
if (data.arbitrator) renderArbitrator(data.arbitrator);
|
||||
if (data.arbitrator) renderArbitrator(data.arbitrator, data.gpu);
|
||||
|
||||
// 1. GPU VRAM Stats
|
||||
const gpu = data.gpu || {};
|
||||
@@ -886,11 +886,23 @@ document.addEventListener('DOMContentLoaded', () => {
|
||||
|
||||
// ---------------------------------------------------------------- arbitration
|
||||
|
||||
function renderArbitrator(arb) {
|
||||
function renderArbitrator(arb, gpu) {
|
||||
const el = (id) => document.getElementById(id);
|
||||
if (!el('arb-action')) return;
|
||||
const c = arb.counters || {};
|
||||
|
||||
// VRAM held by processes HyperSwap cannot reclaim. Worth showing: it is headroom the
|
||||
// arbitrator can never give back, no matter how much it purges.
|
||||
const bd = (gpu && gpu.breakdown) || {};
|
||||
const un = el('arb-unmanaged');
|
||||
if (un) {
|
||||
const procs = bd.unmanaged || [];
|
||||
un.innerHTML = procs.length
|
||||
? `<span class="text-amber-400">${bd.unmanaged_gb} GB unreclaimable</span> — ` +
|
||||
procs.map(p => `${p.name} (${p.vram_mb} MB)`).join(', ')
|
||||
: '';
|
||||
}
|
||||
|
||||
el('arb-action').textContent = arb.last_action || 'Idle';
|
||||
el('arb-yields').textContent = c.yields ?? 0;
|
||||
el('arb-busy').textContent = c.yield_deferred_busy ?? 0;
|
||||
|
||||
@@ -686,6 +686,7 @@
|
||||
<div class="mt-4">
|
||||
<div id="arb-action" class="text-sm text-slate-200 bg-slate-950/60 border border-slate-800 rounded-lg px-3 py-2 mb-3 font-mono">Idle</div>
|
||||
<div id="arb-backoff" class="text-[11px] font-mono text-amber-400 mb-3"></div>
|
||||
<div id="arb-unmanaged" class="text-[11px] font-mono text-slate-400 mb-3"></div>
|
||||
<div class="grid grid-cols-3 sm:grid-cols-6 gap-2 text-center">
|
||||
<div class="bg-slate-950/60 rounded-lg p-2 border border-slate-800">
|
||||
<div id="arb-yields" class="text-lg font-bold text-emerald-400">0</div>
|
||||
|
||||
Reference in New Issue
Block a user