drjones a30444ef8e Stop open SSE streams from blocking graceful shutdown ...

Every connected dashboard holds a StreamingResponse open indefinitely, so uvicorn's
graceful shutdown waited on them, systemd hit its 90s stop timeout and SIGKILLed the
unit. That skipped the in-process restore hook entirely, leaving the GPU restore to
ExecStopPost alone.

- broker.stop() sets a closing flag and pushes a sentinel to every subscriber queue so
  the SSE generators return instead of parking on q.get().
- uvicorn gets timeout_graceful_shutdown=10 and the unit TimeoutStopSec=20, bounding
  the worst case rather than relying on the 90s default.

Restart now completes in ~11s with 'Restoring GPU to safe stock state (server
shutdown)' running in-process, and no SIGKILL.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-28 09:16:36 -07:00

23 KiB

Raw Blame History

View Raw