The Collector

A smooth-operator conversational voice AI running on drjones' homelab. Lives on Proxmox LXC CT 771 @ 10.30.20.172:8766.

What it is

A latency-optimized voice assistant. It listens, responds naturally through a local LLM, speaks with a natural voice, and can make outbound phone calls on request.

Pipeline (latency budget)

Stage	Engine	Latency
STT	faster-whisper small.en (GPU, nightmare .29:8770)	~0.35s / utterance
LLM	ornith-1.5:9b-64k @ nightmare .29:11434 (fallback .222)	~600ms first token, streamed
TTS	Piper lessac-high (CPU, 22.05k → 16k)	~400ms / sentence, overlapped w/ LLM

Sentence-level TTS/LLM overlap: audio starts flowing after the first sentence completes while the model keeps generating. Server-side energy VAD with barge-in.

Interfaces

• Browser voice — GET / (WebSocket /ws, PCM16 16 kHz full-duplex)
• Phone — SignalWire (outbound via POST /api/call {to, task}; inbound stays on JONES's number for now). Media stream /signalwire/stream (mulaw 8 kHz).
• Text — POST /api/text {text}
• Health — GET /api/health

Public URL

Cloudflare quick tunnel (ephemeral): https://placed-ivory-unto-effective.trycloudflare.com Self-registers via POST /api/config {public_base_url} (systemd collector-tunnel.service).

Key decisions

• LLM lane = nightmare .29, not .222: .222 holds ornith resident but is saturated by ~92 cron jobs + Kalshi bots and hangs on /api/chat; .29 answers in ~1s. keep_alive:30m keeps ornith warm there.
• STT on GPU (nightmare), not the CT CPU: whisper on the CT's 2 cores was 7-8s (the real latency killer); faster-whisper on nightmare's 4080S (collector-stt.service @ :8770, CUDA) is ~0.35s — 20×. TTS stays on CPU (Piper ~400ms, not the bottleneck).
• Phone reuses the SignalWire account + number (+1 254 513 9895) JONES uses; outbound TwiML points at this app's tunnel.

Files

/opt/collector/
  app.py               self-contained FastAPI app (STT/TTS/LLM/phone)
  config.yaml          model lane, TTS voice, SignalWire creds, persona
  selftest.py          Piper→whisper round-trip self-test
  static/index.html    browser voice UI (AudioWorklet)
  tunnel.sh            cloudflared quick tunnel + self-registration
  collector.service    systemd unit (app)
  collector-tunnel.service  systemd unit (tunnel)

Ops

# restart
ssh root@10.30.20.85 "pct exec 771 -- systemctl restart collector"
# logs
ssh root@10.30.20.85 "pct exec 771 -- journalctl -u collector -n 30"
# self-test the audio pipeline
ssh root@10.30.20.85 "pct exec 771 -- python3 /opt/collector/selftest.py"
# make a phone call
curl -s -X POST http://10.30.20.172:8766/api/call \
  -H 'Content-Type: application/json' -d '{"to":"+1XXXXXXXXXX","task":"..."}'