Snapshot: full project state
This commit is contained in:
17
.gitignore
vendored
Normal file
17
.gitignore
vendored
Normal file
@@ -0,0 +1,17 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
node_modules/
|
||||
.venv/
|
||||
venv/
|
||||
.env
|
||||
*.db
|
||||
*.sqlite*
|
||||
*.log
|
||||
.DS_Store
|
||||
out/
|
||||
work/
|
||||
.pio/
|
||||
briefs/
|
||||
dns-backup/
|
||||
archive/
|
||||
*.png
|
||||
66
README.md
Normal file
66
README.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# The Collector
|
||||
|
||||
A smooth-operator conversational voice AI running on drjones' homelab. Lives on
|
||||
Proxmox LXC **CT 771** @ `10.30.20.172:8766`.
|
||||
|
||||
## What it is
|
||||
A latency-optimized voice assistant. It listens, responds naturally through a
|
||||
local LLM, speaks with a natural voice, and can make outbound phone calls on
|
||||
request.
|
||||
|
||||
## Pipeline (latency budget)
|
||||
| Stage | Engine | Latency |
|
||||
|-------|--------|---------|
|
||||
| STT | faster-whisper `small.en` (GPU, nightmare `.29:8770`) | ~0.35s / utterance |
|
||||
| LLM | `ornith-1.5:9b-64k` @ nightmare `.29:11434` (fallback `.222`) | ~600ms first token, streamed |
|
||||
| TTS | Piper `lessac-high` (CPU, 22.05k → 16k) | ~400ms / sentence, overlapped w/ LLM |
|
||||
|
||||
Sentence-level TTS/LLM overlap: audio starts flowing after the first sentence
|
||||
completes while the model keeps generating. Server-side energy VAD with barge-in.
|
||||
|
||||
## Interfaces
|
||||
- **Browser voice** — `GET /` (WebSocket `/ws`, PCM16 16 kHz full-duplex)
|
||||
- **Phone** — SignalWire (outbound via `POST /api/call {to, task}`; inbound stays
|
||||
on JONES's number for now). Media stream `/signalwire/stream` (mulaw 8 kHz).
|
||||
- **Text** — `POST /api/text {text}`
|
||||
- **Health** — `GET /api/health`
|
||||
|
||||
## Public URL
|
||||
Cloudflare quick tunnel (ephemeral): `https://placed-ivory-unto-effective.trycloudflare.com`
|
||||
Self-registers via `POST /api/config {public_base_url}` (systemd `collector-tunnel.service`).
|
||||
|
||||
## Key decisions
|
||||
- **LLM lane = nightmare `.29`, not `.222`**: `.222` holds ornith resident but is
|
||||
saturated by ~92 cron jobs + Kalshi bots and hangs on `/api/chat`; `.29` answers
|
||||
in ~1s. `keep_alive:30m` keeps ornith warm there.
|
||||
- **STT on GPU (nightmare), not the CT CPU**: whisper on the CT's 2 cores was 7-8s
|
||||
(the real latency killer); `faster-whisper` on nightmare's 4080S
|
||||
(`collector-stt.service` @ `:8770`, CUDA) is ~0.35s — 20×. TTS stays on CPU (Piper
|
||||
~400ms, not the bottleneck).
|
||||
- **Phone** reuses the SignalWire account + number (`+1 254 513 9895`) JONES uses;
|
||||
outbound TwiML points at this app's tunnel.
|
||||
|
||||
## Files
|
||||
```
|
||||
/opt/collector/
|
||||
app.py self-contained FastAPI app (STT/TTS/LLM/phone)
|
||||
config.yaml model lane, TTS voice, SignalWire creds, persona
|
||||
selftest.py Piper→whisper round-trip self-test
|
||||
static/index.html browser voice UI (AudioWorklet)
|
||||
tunnel.sh cloudflared quick tunnel + self-registration
|
||||
collector.service systemd unit (app)
|
||||
collector-tunnel.service systemd unit (tunnel)
|
||||
```
|
||||
|
||||
## Ops
|
||||
```bash
|
||||
# restart
|
||||
ssh root@10.30.20.85 "pct exec 771 -- systemctl restart collector"
|
||||
# logs
|
||||
ssh root@10.30.20.85 "pct exec 771 -- journalctl -u collector -n 30"
|
||||
# self-test the audio pipeline
|
||||
ssh root@10.30.20.85 "pct exec 771 -- python3 /opt/collector/selftest.py"
|
||||
# make a phone call
|
||||
curl -s -X POST http://10.30.20.172:8766/api/call \
|
||||
-H 'Content-Type: application/json' -d '{"to":"+1XXXXXXXXXX","task":"..."}'
|
||||
```
|
||||
587
app.py
Normal file
587
app.py
Normal file
@@ -0,0 +1,587 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The Collector — a smooth-operator conversational voice AI.
|
||||
|
||||
Pipeline (latency-optimized):
|
||||
STT faster-whisper small.en (CPU, int8) ~1-2s / short utterance
|
||||
LLM ornith-1.5:9b-64k @ .222 (warm) ~600ms first token, streamed
|
||||
TTS Piper lessac-high (CPU, 22.05k->16k) ~400ms / sentence, overlapped w/ LLM
|
||||
|
||||
Interfaces:
|
||||
WS /ws browser full-duplex voice (PCM16 16 kHz)
|
||||
WS /signalwire/stream phone media stream (mulaw 8 kHz)
|
||||
POST /api/call {to, task} outbound phone call on request
|
||||
POST /api/text {text} text chat (returns text)
|
||||
GET /api/health
|
||||
"""
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
|
||||
import audioop
|
||||
import httpx
|
||||
import numpy as np
|
||||
import yaml
|
||||
from fastapi import FastAPI, Request, WebSocket, WebSocketDisconnect
|
||||
from fastapi.responses import FileResponse, JSONResponse, PlainTextResponse
|
||||
|
||||
logging.basicConfig(level=logging.INFO,
|
||||
format="%(asctime)s %(name)s %(levelname)s %(message)s")
|
||||
log = logging.getLogger("collector")
|
||||
|
||||
BASE = "/opt/collector"
|
||||
|
||||
|
||||
def load_config() -> dict:
|
||||
with open(os.path.join(BASE, "config.yaml")) as f:
|
||||
return yaml.safe_load(f)
|
||||
|
||||
|
||||
CFG = load_config()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# audio helpers (stdlib audioop)
|
||||
# ---------------------------------------------------------------------------
|
||||
def ulaw_to_pcm(data: bytes) -> bytes:
|
||||
return audioop.ulaw2lin(data, 2)
|
||||
|
||||
|
||||
def pcm_to_ulaw(pcm: bytes) -> bytes:
|
||||
return audioop.lin2ulaw(pcm, 2)
|
||||
|
||||
|
||||
def resample(pcm: bytes, src: int, dst: int) -> bytes:
|
||||
if src == dst or not pcm:
|
||||
return pcm
|
||||
out, _ = audioop.ratecv(pcm, 2, 1, src, dst, None)
|
||||
return out
|
||||
|
||||
|
||||
def rms(pcm: bytes) -> int:
|
||||
return audioop.rms(pcm, 2) if pcm else 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# STT
|
||||
# ---------------------------------------------------------------------------
|
||||
class STT:
|
||||
def __init__(self, cfg):
|
||||
self.cfg = cfg
|
||||
self.model = None
|
||||
self.remote_url = (cfg.get("remote_url") or "").rstrip("/")
|
||||
|
||||
def load(self):
|
||||
# Remote GPU STT (nightmare) is primary; local whisper is a lazy fallback.
|
||||
return self
|
||||
|
||||
def transcribe(self, pcm16k: bytes) -> str:
|
||||
if self.remote_url:
|
||||
try:
|
||||
return self._remote(pcm16k)
|
||||
except Exception as e:
|
||||
log.warning("remote STT failed (%s); falling back to local", e)
|
||||
return self._local(pcm16k)
|
||||
|
||||
def _remote(self, pcm16k: bytes) -> str:
|
||||
import requests
|
||||
r = requests.post(self.remote_url + "/transcribe", data=pcm16k, timeout=30)
|
||||
r.raise_for_status()
|
||||
return (r.json().get("text") or "").strip()
|
||||
|
||||
def _local(self, pcm16k: bytes) -> str:
|
||||
if self.model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
c = self.cfg
|
||||
log.info("Loading local faster-whisper %s (fallback)...", c["model"])
|
||||
self.model = WhisperModel(c["model"], device=c["device"],
|
||||
compute_type=c["compute_type"],
|
||||
cpu_threads=c.get("cpu_threads", 2))
|
||||
audio = np.frombuffer(pcm16k, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
segs, _ = self.model.transcribe(
|
||||
audio, beam_size=1, language="en", vad_filter=True,
|
||||
condition_on_previous_text=False, no_speech_threshold=0.6)
|
||||
return " ".join(s.text.strip() for s in segs).strip()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TTS
|
||||
# ---------------------------------------------------------------------------
|
||||
class TTS:
|
||||
def __init__(self, cfg):
|
||||
self.cfg = cfg
|
||||
self.voice_path = cfg.get("piper_voice") or \
|
||||
"/opt/collector/voices/en_US-lessac-high.onnx"
|
||||
self._piper = None
|
||||
|
||||
def load(self):
|
||||
if self._piper is None:
|
||||
# onnxruntime pins threads to CPUs 0..N-1; the unprivileged LXC
|
||||
# cpuset (e.g. 8,17) rejects that. Setting an explicit thread count
|
||||
# disables the affinity pin (which is otherwise just log noise).
|
||||
import onnxruntime
|
||||
_orig = onnxruntime.InferenceSession.__init__
|
||||
|
||||
def _patched(self, *a, **k):
|
||||
so = k.get("sess_options") or onnxruntime.SessionOptions()
|
||||
so.intra_op_num_threads = 2
|
||||
k["sess_options"] = so
|
||||
_orig(self, *a, **k)
|
||||
|
||||
onnxruntime.InferenceSession.__init__ = _patched
|
||||
|
||||
from piper import PiperVoice
|
||||
log.info("Loading Piper voice %s ...", self.voice_path)
|
||||
self._piper = PiperVoice.load(self.voice_path)
|
||||
log.info("TTS ready")
|
||||
return self._piper
|
||||
|
||||
def synthesize_16k(self, text: str) -> bytes:
|
||||
"""Return PCM16 mono 16 kHz for `text` (piper yields AudioChunk iterators)."""
|
||||
if not text.strip():
|
||||
return b""
|
||||
speed = float(self.cfg.get("speed", 1.05))
|
||||
from piper import SynthesisConfig
|
||||
syn = SynthesisConfig(length_scale=1.0 / speed)
|
||||
chunks = []
|
||||
rate = 22050
|
||||
for chunk in self._piper.synthesize(text, syn_config=syn):
|
||||
audio = (chunk.audio_float_array * 32767).astype(np.int16).tobytes()
|
||||
rate = chunk.sample_rate
|
||||
chunks.append(audio)
|
||||
pcm = b"".join(chunks)
|
||||
return resample(pcm, rate, 16000)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM (ornith streaming, /api/chat, think:false)
|
||||
# ---------------------------------------------------------------------------
|
||||
async def stream_llm(messages, cfg):
|
||||
payload = {
|
||||
"model": cfg["ollama"]["model"],
|
||||
"messages": messages,
|
||||
"stream": True,
|
||||
"think": False,
|
||||
"keep_alive": "30m",
|
||||
"options": {
|
||||
"temperature": cfg["ollama"].get("temperature", 0.5),
|
||||
"num_predict": cfg["ollama"].get("num_predict", 256),
|
||||
},
|
||||
}
|
||||
urls = [cfg["ollama"]["base_url"], cfg["ollama"].get("fallback_url", "")]
|
||||
for base in urls:
|
||||
if not base:
|
||||
continue
|
||||
url = base.rstrip("/") + "/api/chat"
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(10.0, read=60.0)) as client:
|
||||
async with client.stream("POST", url, json=payload) as r:
|
||||
r.raise_for_status()
|
||||
async for line in r.aiter_lines():
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if obj.get("done"):
|
||||
return
|
||||
msg = obj.get("message", {})
|
||||
content = msg.get("content") or msg.get("thinking") or ""
|
||||
if content:
|
||||
yield content
|
||||
return
|
||||
except Exception as e:
|
||||
log.warning("LLM host %s failed: %s", base, e)
|
||||
log.error("All LLM hosts failed")
|
||||
|
||||
|
||||
async def sentence_stream(chunks):
|
||||
"""Yield sentences as they complete, enabling TTS/LLM overlap."""
|
||||
buf = ""
|
||||
async for chunk in chunks:
|
||||
buf += chunk
|
||||
while True:
|
||||
m = re.search(r"[.!?\n]", buf)
|
||||
if not m:
|
||||
break
|
||||
idx = m.end()
|
||||
sentence = buf[:idx].strip()
|
||||
buf = buf[idx:].strip()
|
||||
if sentence:
|
||||
yield sentence
|
||||
if buf.strip():
|
||||
yield buf.strip()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Conversation session (shared by browser WS + phone stream)
|
||||
# ---------------------------------------------------------------------------
|
||||
SPEECH_RMS = 400 # 16-bit PCM: voice onset
|
||||
BARGE_RMS = 900 # sustained level that interrupts playback
|
||||
SILENCE_END_MS = 700 # ms of quiet after speech before we turn
|
||||
FRAME_MS = 20
|
||||
MAX_UTT_BYTES = 18 * 16000 * 2 # 18s cap
|
||||
|
||||
|
||||
class Session:
|
||||
def __init__(self, cfg, system_prompt):
|
||||
self.cfg = cfg
|
||||
self.stt = STT(cfg["stt"])
|
||||
self.tts = TTS(cfg["tts"])
|
||||
self.messages = [{"role": "system", "content": system_prompt}]
|
||||
self.buf = bytearray()
|
||||
self.talking = False
|
||||
self.silence_ms = 0
|
||||
self._barge = False
|
||||
self._processing = False
|
||||
self._queue = asyncio.Queue()
|
||||
self._worker = None
|
||||
|
||||
async def warm(self):
|
||||
await asyncio.to_thread(self.stt.load)
|
||||
await asyncio.to_thread(self.tts.load)
|
||||
try:
|
||||
async for _ in stream_llm(
|
||||
[{"role": "user", "content": "hi"}], self.cfg):
|
||||
break
|
||||
except Exception as e:
|
||||
log.warning("LLM warmup: %s", e)
|
||||
log.info("Collector warmed (STT + TTS + LLM)")
|
||||
|
||||
def start_worker(self, send_audio, send_text):
|
||||
"""Begin the continuous conversation loop for this connection."""
|
||||
self._worker = asyncio.create_task(
|
||||
self._process_loop(send_audio, send_text))
|
||||
|
||||
async def feed_audio(self, pcm16k: bytes):
|
||||
"""Continuous VAD. Call for every 20ms frame. Never drops speech —
|
||||
even during a reply, overlapping audio is buffered and processed next
|
||||
(barge-in)."""
|
||||
level = rms(pcm16k)
|
||||
if level > SPEECH_RMS:
|
||||
self.talking = True
|
||||
self.silence_ms = 0
|
||||
if self._processing:
|
||||
self._barge = True # user started talking over the reply
|
||||
elif self.talking:
|
||||
self.silence_ms += FRAME_MS
|
||||
self.buf.extend(pcm16k)
|
||||
if self.talking and (self.silence_ms >= SILENCE_END_MS or
|
||||
len(self.buf) > MAX_UTT_BYTES):
|
||||
self.talking = False
|
||||
self.silence_ms = 0
|
||||
pcm = bytes(self.buf)
|
||||
self.buf.clear()
|
||||
self._queue.put_nowait(pcm)
|
||||
|
||||
async def _process_loop(self, send_audio, send_text):
|
||||
while True:
|
||||
pcm = await self._queue.get()
|
||||
self._processing = True
|
||||
self._barge = False
|
||||
text = await asyncio.to_thread(self.stt.transcribe, pcm)
|
||||
if text:
|
||||
await send_text(text, "user")
|
||||
self.messages.append({"role": "user", "content": text})
|
||||
await self._reply(send_audio, send_text)
|
||||
self._processing = False
|
||||
|
||||
async def speak(self, text: str, send_audio, send_text):
|
||||
"""Speak an injected message (outbound preamble), then keep listening."""
|
||||
self.messages.append({"role": "user", "content": text})
|
||||
self._processing = True
|
||||
self._barge = False
|
||||
await self._reply(send_audio, send_text)
|
||||
self._processing = False
|
||||
|
||||
async def _reply(self, send_audio, send_text):
|
||||
full = []
|
||||
async for sentence in sentence_stream(stream_llm(self.messages, self.cfg)):
|
||||
if self._barge:
|
||||
log.info("barge-in: stopping reply")
|
||||
break
|
||||
full.append(sentence)
|
||||
pcm = await asyncio.to_thread(self.tts.synthesize_16k, sentence)
|
||||
if pcm:
|
||||
await send_audio(pcm)
|
||||
if full:
|
||||
reply = " ".join(full)
|
||||
await send_text(reply, "assistant")
|
||||
self.messages.append({"role": "assistant", "content": reply})
|
||||
if len(self.messages) > 24:
|
||||
self.messages = [self.messages[0]] + self.messages[-23:]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SignalWire client
|
||||
# ---------------------------------------------------------------------------
|
||||
class Phone:
|
||||
def __init__(self, cfg):
|
||||
t = cfg.get("signalwire", {})
|
||||
self.sid = t.get("account_sid", "")
|
||||
self.token = t.get("auth_token", "")
|
||||
self.number = t.get("phone_number", "")
|
||||
self.client = None
|
||||
self.tasks = {} # call_sid -> outbound task
|
||||
if self.sid and self.token:
|
||||
from signalwire.rest import Client as SWClient
|
||||
self.client = SWClient(self.sid, self.token,
|
||||
signalwire_space_url="templeofdoom.signalwire.com")
|
||||
|
||||
@property
|
||||
def ready(self):
|
||||
return self.client is not None and bool(self.number)
|
||||
|
||||
def create_call(self, to: str, url: str):
|
||||
return self.client.calls.create(to=to, from_=self.number, url=url)
|
||||
|
||||
|
||||
PHONE = Phone(CFG)
|
||||
app = FastAPI(title="The Collector")
|
||||
|
||||
|
||||
def system_prompt() -> str:
|
||||
return CFG["agent"].get("system_prompt", "You are The Collector.")
|
||||
|
||||
|
||||
def stream_twiml(direction: str, public_base: str) -> str:
|
||||
# SignalWire <Stream> requires wss://, not https://
|
||||
wss_base = public_base.replace("https://", "wss://", 1)
|
||||
stream_url = f"{wss_base}/signalwire/stream?direction={direction}"
|
||||
return (f'<?xml version="1.0" encoding="UTF-8"?><Response><Connect>'
|
||||
f'<Stream url="{stream_url}"/></Connect></Response>')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Health
|
||||
# ---------------------------------------------------------------------------
|
||||
@app.get("/")
|
||||
async def index():
|
||||
return FileResponse(os.path.join(BASE, "static", "index.html"))
|
||||
|
||||
|
||||
@app.get("/api/health")
|
||||
async def health():
|
||||
return {
|
||||
"ok": True,
|
||||
"name": CFG["agent"]["name"],
|
||||
"model": CFG["ollama"]["model"],
|
||||
"llm": CFG["ollama"]["base_url"],
|
||||
"phone": PHONE.ready,
|
||||
"number": PHONE.number if PHONE.ready else None,
|
||||
}
|
||||
|
||||
|
||||
@app.post("/api/config")
|
||||
async def api_config(request: Request):
|
||||
"""Accept runtime config (tunnel URL, temperature, tts speed) + persist."""
|
||||
body = await request.json()
|
||||
changed = {}
|
||||
if "public_base_url" in body:
|
||||
url = (body["public_base_url"] or "").strip().rstrip("/")
|
||||
CFG.setdefault("server", {})["public_base_url"] = url
|
||||
changed["public_base_url"] = url
|
||||
if "temperature" in body:
|
||||
t = float(body["temperature"])
|
||||
CFG.setdefault("ollama", {})["temperature"] = max(0.0, min(1.5, t))
|
||||
changed["temperature"] = CFG["ollama"]["temperature"]
|
||||
if "speed" in body:
|
||||
s = float(body["speed"])
|
||||
CFG.setdefault("tts", {})["speed"] = max(0.5, min(2.0, s))
|
||||
changed["speed"] = CFG["tts"]["speed"]
|
||||
if changed:
|
||||
try:
|
||||
with open(os.path.join(BASE, "config.yaml")) as f:
|
||||
cfg = yaml.safe_load(f)
|
||||
if "public_base_url" in changed:
|
||||
cfg.setdefault("server", {})["public_base_url"] = changed["public_base_url"]
|
||||
if "temperature" in changed:
|
||||
cfg.setdefault("ollama", {})["temperature"] = changed["temperature"]
|
||||
if "speed" in changed:
|
||||
cfg.setdefault("tts", {})["speed"] = changed["speed"]
|
||||
with open(os.path.join(BASE, "config.yaml"), "w") as f:
|
||||
yaml.safe_dump(cfg, f, sort_keys=False)
|
||||
except Exception as e:
|
||||
log.warning("config persist failed: %s", e)
|
||||
return {"ok": True, **changed}
|
||||
return JSONResponse({"ok": False, "error": "nothing to set"}, 400)
|
||||
|
||||
|
||||
@app.get("/api/settings")
|
||||
async def api_settings():
|
||||
return {
|
||||
"temperature": CFG["ollama"].get("temperature", 0.5),
|
||||
"speed": CFG["tts"].get("speed", 1.05),
|
||||
"model": CFG["ollama"]["model"],
|
||||
"voice": "lessac-high",
|
||||
"llm": CFG["ollama"]["base_url"],
|
||||
"stt": "GPU (nightmare)",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Text chat
|
||||
# ---------------------------------------------------------------------------
|
||||
@app.post("/api/text")
|
||||
async def api_text(request: Request):
|
||||
body = await request.json()
|
||||
text = (body.get("text") or "").strip()
|
||||
if not text:
|
||||
return JSONResponse({"ok": False, "error": "empty text"}, 400)
|
||||
msgs = [{"role": "system", "content": system_prompt()},
|
||||
{"role": "user", "content": text}]
|
||||
out = []
|
||||
async for chunk in stream_llm(msgs, CFG):
|
||||
out.append(chunk)
|
||||
return {"ok": True, "reply": "".join(out).strip()}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Outbound call
|
||||
# ---------------------------------------------------------------------------
|
||||
@app.post("/api/call")
|
||||
async def api_call(request: Request):
|
||||
if not PHONE.ready:
|
||||
return JSONResponse({"ok": False, "error": "SignalWire not configured"}, 400)
|
||||
body = await request.json()
|
||||
to = (body.get("to") or "").strip()
|
||||
task = (body.get("task") or "").strip()
|
||||
if not to:
|
||||
return JSONResponse({"ok": False, "error": "missing 'to'"}, 400)
|
||||
public = CFG["server"].get("public_base_url", "")
|
||||
if not public:
|
||||
return JSONResponse({"ok": False, "error": "public_base_url not set"}, 500)
|
||||
call = await asyncio.to_thread(
|
||||
PHONE.create_call, to, f"{public}/signalwire/outbound")
|
||||
PHONE.tasks[call.sid] = task
|
||||
return {"ok": True, "call_sid": call.sid, "to": to, "task": task}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SignalWire webhooks
|
||||
# ---------------------------------------------------------------------------
|
||||
@app.post("/signalwire/voice")
|
||||
async def sw_inbound():
|
||||
public = CFG["server"].get("public_base_url", "")
|
||||
return PlainTextResponse(stream_twiml("inbound", public),
|
||||
media_type="application/xml")
|
||||
|
||||
|
||||
@app.post("/signalwire/outbound")
|
||||
async def sw_outbound():
|
||||
public = CFG["server"].get("public_base_url", "")
|
||||
return PlainTextResponse(stream_twiml("outbound", public),
|
||||
media_type="application/xml")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Browser full-duplex voice
|
||||
# ---------------------------------------------------------------------------
|
||||
@app.websocket("/ws")
|
||||
async def browser_ws(ws: WebSocket):
|
||||
await ws.accept()
|
||||
session = Session(CFG, system_prompt())
|
||||
|
||||
async def send_audio(pcm):
|
||||
await ws.send_bytes(pcm)
|
||||
|
||||
async def send_text(text, role):
|
||||
await ws.send_text(json.dumps({"type": "transcript", "role": role,
|
||||
"text": text}))
|
||||
|
||||
await session.warm()
|
||||
session.start_worker(send_audio, send_text)
|
||||
try:
|
||||
while True:
|
||||
data = await ws.receive()
|
||||
if data["type"] == "websocket.receive":
|
||||
if data.get("bytes"):
|
||||
await session.feed_audio(data["bytes"])
|
||||
elif data["type"] == "websocket.disconnect":
|
||||
break
|
||||
except WebSocketDisconnect:
|
||||
pass
|
||||
finally:
|
||||
if session._worker:
|
||||
session._worker.cancel()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SignalWire media stream (phone)
|
||||
# ---------------------------------------------------------------------------
|
||||
async def send_phone_audio(ws, stream_sid, pcm16k):
|
||||
pcm8 = resample(pcm16k, 16000, 8000)
|
||||
ulaw = pcm_to_ulaw(pcm8)
|
||||
frame = 160 # 20ms @ 8kHz
|
||||
for i in range(0, len(ulaw), frame):
|
||||
chunk = ulaw[i:i + frame]
|
||||
await ws.send_text(json.dumps({
|
||||
"event": "media", "streamSid": stream_sid,
|
||||
"media": {"payload": base64.b64encode(chunk).decode()}}))
|
||||
await asyncio.sleep(0.02)
|
||||
|
||||
|
||||
@app.websocket("/signalwire/stream")
|
||||
async def sw_stream(ws: WebSocket, direction: str = "inbound"):
|
||||
await ws.accept()
|
||||
session = Session(CFG, system_prompt())
|
||||
stream_sid = None
|
||||
call_sid = None
|
||||
first = True
|
||||
|
||||
async def send_audio(pcm16k):
|
||||
await send_phone_audio(ws, stream_sid, pcm16k)
|
||||
|
||||
async def send_text(text, role):
|
||||
pass # no transcript channel on phone; audio only
|
||||
|
||||
await session.warm()
|
||||
session.start_worker(send_audio, send_text)
|
||||
try:
|
||||
while True:
|
||||
msg = await ws.receive_text()
|
||||
try:
|
||||
obj = json.loads(msg)
|
||||
except Exception:
|
||||
continue
|
||||
ev = obj.get("event")
|
||||
|
||||
if ev == "start":
|
||||
start = obj.get("start", {})
|
||||
stream_sid = start.get("streamSid", stream_sid)
|
||||
call_sid = start.get("callSid", call_sid)
|
||||
if direction == "outbound" and first:
|
||||
first = False
|
||||
task = PHONE.tasks.pop(call_sid, "")
|
||||
if task:
|
||||
preamble = CFG["agent"].get("outbound_preamble", "") + task
|
||||
await session.speak(preamble, send_audio, send_text)
|
||||
|
||||
elif ev == "media":
|
||||
payload = obj.get("media", {}).get("payload", "")
|
||||
if not payload:
|
||||
continue
|
||||
raw = base64.b64decode(payload)
|
||||
pcm8 = ulaw_to_pcm(raw)
|
||||
pcm16k = resample(pcm8, 8000, 16000)
|
||||
await session.feed_audio(pcm16k)
|
||||
|
||||
elif ev == "stop":
|
||||
break
|
||||
except WebSocketDisconnect:
|
||||
pass
|
||||
finally:
|
||||
if session._worker:
|
||||
session._worker.cancel()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entrypoint
|
||||
# ---------------------------------------------------------------------------
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
s = CFG["server"]
|
||||
uvicorn.run(app, host=s.get("host", "0.0.0.0"), port=s.get("port", 8766))
|
||||
17
collector-stt.service
Normal file
17
collector-stt.service
Normal file
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=The Collector — GPU STT (faster-whisper on CUDA)
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=drjones
|
||||
Group=drjones
|
||||
ExecStart=/usr/bin/python3 /opt/collector-stt/nightmare_stt.py
|
||||
WorkingDirectory=/opt/collector-stt
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
Environment=LD_LIBRARY_PATH=/home/drjones/.local/lib/python3.14/site-packages/nvidia/cuda_runtime/lib:/home/drjones/.local/lib/python3.14/site-packages/nvidia/cudnn/lib:/home/drjones/.local/lib/python3.14/site-packages/nvidia/cublas/lib:/home/drjones/.local/lib/python3.14/site-packages/nvidia/cuda_nvrtc/lib
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
13
collector-tunnel.service
Normal file
13
collector-tunnel.service
Normal file
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=The Collector — Cloudflare quick tunnel (public URL + phone webhooks)
|
||||
After=collector.service network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/bin/bash /opt/collector/tunnel.sh
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
15
collector.service
Normal file
15
collector.service
Normal file
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=The Collector — conversational voice AI (ornith + Piper + SignalWire)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/python3 /opt/collector/app.py
|
||||
WorkingDirectory=/opt/collector
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
52
config.yaml
Normal file
52
config.yaml
Normal file
@@ -0,0 +1,52 @@
|
||||
server:
|
||||
host: 0.0.0.0
|
||||
port: 8766
|
||||
public_base_url: ""
|
||||
|
||||
# LLM — ornith conversational lane.
|
||||
# Primary = nightmare .29 (4080S, measured fast first-token; .222 is saturated by
|
||||
# the 92 cron jobs + Kalshi bots and hangs on /api/chat). Fallback = .222 (ornith home lane).
|
||||
ollama:
|
||||
base_url: http://10.30.20.29:11434
|
||||
fallback_url: http://10.30.20.222:11434
|
||||
model: ornith-1.5:9b-64k
|
||||
think: false
|
||||
temperature: 0.5
|
||||
num_predict: 256
|
||||
|
||||
stt:
|
||||
model: small.en
|
||||
device: cpu
|
||||
compute_type: int8
|
||||
cpu_threads: 2
|
||||
remote_url: http://10.30.20.29:8770
|
||||
|
||||
# Piper lessac-high = natural voice, ~400ms/sentence on CPU, no GPU needed.
|
||||
# (TTS is NOT the latency bottleneck — STT + LLM first-token dominate the budget.)
|
||||
tts:
|
||||
engine: piper
|
||||
voice: lessac-high
|
||||
speed: 1.05
|
||||
piper_voice: /opt/collector/voices/en_US-lessac-high.onnx
|
||||
|
||||
agent:
|
||||
name: The Collector
|
||||
owner_phone: "+14252800023"
|
||||
system_prompt: >
|
||||
You are The Collector, a smooth, effortlessly charming conversational AI running on
|
||||
Indiana's homelab. You are on a LIVE VOICE call. Reply in 1-3 short conversational
|
||||
sentences. No markdown, no bullet points, no emoji. Listen first, then respond with
|
||||
warmth and wit — a trusted friend who happens to be exceptionally smooth. Match the
|
||||
caller's energy. Keep it natural, human, and brief.
|
||||
outbound_preamble: >
|
||||
You are The Collector, making a phone call on Indiana's behalf. Introduce yourself
|
||||
naturally, accomplish the objective, confirm details, wrap up politely. Objective:
|
||||
|
||||
signalwire:
|
||||
account_sid: 8a89a49f-6e98-4de1-b32e-df897f656f24
|
||||
auth_token: PT2ad398d81d9a1632375b8ac88844ea8ecfa46021b5389dfb
|
||||
phone_number: "+12545139895"
|
||||
|
||||
api:
|
||||
auth_key: collector-voice-2026
|
||||
auth_enabled: false
|
||||
17
install.sh
Normal file
17
install.sh
Normal file
@@ -0,0 +1,17 @@
|
||||
#!/bin/bash
|
||||
# The Collector — dependency install (Debian 12 LXC)
|
||||
set -e
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq \
|
||||
python3 python3-pip python3-venv python3-yaml \
|
||||
ffmpeg libsndfile1 espeak-ng
|
||||
|
||||
pip3 install --break-system-packages --no-cache-dir \
|
||||
fastapi "uvicorn[standard]" websockets httpx \
|
||||
twilio signalwire pyyaml numpy \
|
||||
faster-whisper piper-tts requests python-multipart
|
||||
|
||||
mkdir -p /opt/collector/voices
|
||||
echo "INSTALL_OK"
|
||||
62
multiturn_test.py
Normal file
62
multiturn_test.py
Normal file
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Multi-turn conversational test: two back-and-forth exchanges over /ws."""
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, "/opt/collector")
|
||||
from app import TTS, CFG
|
||||
import websockets
|
||||
|
||||
tts = TTS(CFG["tts"])
|
||||
tts.load()
|
||||
P1 = tts.synthesize_16k("What is your name?")
|
||||
P2 = tts.synthesize_16k("Tell me something interesting.")
|
||||
|
||||
FRAME = 640 # 20ms @ 16kHz
|
||||
|
||||
|
||||
async def send_utterance(ws, pcm):
|
||||
for i in range(0, len(pcm), FRAME):
|
||||
await ws.send(pcm[i:i + FRAME])
|
||||
await asyncio.sleep(0.02)
|
||||
silence = b"\x00" * FRAME
|
||||
for _ in range(50): # 1s silence -> VAD turn end
|
||||
await ws.send(silence)
|
||||
await asyncio.sleep(0.02)
|
||||
|
||||
|
||||
async def read_reply(ws, seen_assistant):
|
||||
"""Read messages until we collect a NEW assistant transcript + audio."""
|
||||
audio = 0
|
||||
while True:
|
||||
try:
|
||||
msg = await asyncio.wait_for(ws.recv(), timeout=45)
|
||||
except Exception:
|
||||
return False
|
||||
if isinstance(msg, str):
|
||||
m = json.loads(msg)
|
||||
if m.get("role") == "assistant":
|
||||
seen_assistant += 1
|
||||
else:
|
||||
audio += len(msg)
|
||||
if seen_assistant >= 1 and audio > 0:
|
||||
return True
|
||||
|
||||
|
||||
async def main():
|
||||
async with websockets.connect("ws://127.0.0.1:8766/ws") as ws:
|
||||
await asyncio.sleep(6) # warm
|
||||
# Turn 1
|
||||
await send_utterance(ws, P1)
|
||||
ok1 = await read_reply(ws, 0)
|
||||
print("TURN1 reply:", "PASS" if ok1 else "FAIL")
|
||||
# Turn 2
|
||||
await asyncio.sleep(1)
|
||||
await send_utterance(ws, P2)
|
||||
ok2 = await read_reply(ws, 1)
|
||||
print("TURN2 reply:", "PASS" if ok2 else "FAIL")
|
||||
print("MULTITURN", "PASS" if (ok1 and ok2) else "FAIL")
|
||||
|
||||
|
||||
asyncio.run(main())
|
||||
41
nightmare_stt.py
Normal file
41
nightmare_stt.py
Normal file
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env python3
|
||||
"""GPU STT service for The Collector — faster-whisper small.en on CUDA (4080S).
|
||||
|
||||
POST /transcribe (raw PCM16 mono 16 kHz body) -> {"text": "..."}
|
||||
GET /health
|
||||
"""
|
||||
import numpy as np
|
||||
from fastapi import FastAPI, Request
|
||||
from faster_whisper import WhisperModel
|
||||
import uvicorn
|
||||
|
||||
MODEL = "small.en"
|
||||
app = FastAPI()
|
||||
model = None
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def load():
|
||||
global model
|
||||
model = WhisperModel(MODEL, device="cuda", compute_type="float16")
|
||||
print(f"STT ready: {MODEL} on cuda")
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"ok": model is not None, "model": MODEL, "device": "cuda"}
|
||||
|
||||
|
||||
@app.post("/transcribe")
|
||||
async def transcribe(request: Request):
|
||||
body = await request.body()
|
||||
audio = np.frombuffer(body, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
segs, _ = model.transcribe(
|
||||
audio, beam_size=1, language="en", vad_filter=True,
|
||||
condition_on_previous_text=False, no_speech_threshold=0.6)
|
||||
text = " ".join(s.text.strip() for s in segs).strip()
|
||||
return {"text": text}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(app, host="0.0.0.0", port=8770)
|
||||
24
selftest.py
Normal file
24
selftest.py
Normal file
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Collector self-test: synthesize a phrase with Piper, transcribe it back with
|
||||
faster-whisper. Proves the STT/TTS audio pipeline end-to-end (no phone needed)."""
|
||||
import sys
|
||||
sys.path.insert(0, "/opt/collector")
|
||||
from app import STT, TTS, CFG
|
||||
|
||||
stt = STT(CFG["stt"])
|
||||
tts = TTS(CFG["tts"])
|
||||
|
||||
print("loading STT (first run downloads whisper model)...")
|
||||
stt.load()
|
||||
print("loading TTS...")
|
||||
tts.load()
|
||||
|
||||
phrase = "This is the collector self test. If you can hear this, the voice pipeline works."
|
||||
pcm16k = tts.synthesize_16k(phrase)
|
||||
print("TTS_OK bytes=%d" % len(pcm16k))
|
||||
|
||||
text = stt.transcribe(pcm16k)
|
||||
print("STT_OK transcript=%r" % text)
|
||||
|
||||
ok = "collector" in text.lower() or "voice" in text.lower() or "hear" in text.lower()
|
||||
print("SELFTEST", "PASS" if ok else "PARTIAL", file=sys.stderr)
|
||||
325
static/index.html
Normal file
325
static/index.html
Normal file
@@ -0,0 +1,325 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>The Collector</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg0:#07070f; --bg1:#0d0d1f; --purple:#7b5cff; --teal:#3cc8a0;
|
||||
--ink:#e9eaf5; --dim:#9aa0b8; --panel:rgba(255,255,255,.045);
|
||||
--edge:rgba(255,255,255,.08); --accent:#7b5cff;
|
||||
}
|
||||
* { box-sizing:border-box; }
|
||||
html,body { margin:0; height:100%; }
|
||||
body {
|
||||
font-family:ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;
|
||||
color:var(--ink); background:
|
||||
radial-gradient(1100px 700px at 15% -10%, #1c1540 0%, transparent 60%),
|
||||
radial-gradient(900px 600px at 100% 110%, #0e2b3d 0%, transparent 55%),
|
||||
linear-gradient(160deg, var(--bg0), var(--bg1));
|
||||
display:flex; align-items:center; justify-content:center;
|
||||
overflow:hidden; height:100dvh;
|
||||
}
|
||||
/* slow drifting nebula (calm, not chaotic) */
|
||||
.glow { position:fixed; border-radius:50%; filter:blur(90px); opacity:.5; pointer-events:none; }
|
||||
.glow.a { width:520px;height:520px;left:-120px;top:-80px; background:#4b3ad0; animation:drift1 26s ease-in-out infinite; }
|
||||
.glow.b { width:420px;height:420px;right:-100px;bottom:-60px; background:#1d5f86; animation:drift2 32s ease-in-out infinite; }
|
||||
@keyframes drift1 { 0%,100%{transform:translate(0,0)} 50%{transform:translate(60px,40px)} }
|
||||
@keyframes drift2 { 0%,100%{transform:translate(0,0)} 50%{transform:translate(-50px,-30px)} }
|
||||
|
||||
.card {
|
||||
position:relative; width:min(520px,94vw); padding:34px 30px 28px;
|
||||
text-align:center; border-radius:28px;
|
||||
background:var(--panel); border:1px solid var(--edge);
|
||||
backdrop-filter:blur(14px); box-shadow:0 30px 80px rgba(0,0,0,.5);
|
||||
}
|
||||
h1 { margin:0; font-size:26px; font-weight:700; letter-spacing:.5px; }
|
||||
.sub { color:var(--dim); font-size:12.5px; margin:5px 0 22px; letter-spacing:.4px; }
|
||||
|
||||
.stage { position:relative; width:230px; height:230px; margin:0 auto 22px; }
|
||||
canvas#ring { position:absolute; inset:0; width:100%; height:100%; }
|
||||
.orb {
|
||||
position:absolute; inset:34px; border-radius:50%;
|
||||
background:radial-gradient(circle at 34% 28%, #a68aff, #6a4be0 48%, #241457 80%);
|
||||
box-shadow:0 0 60px rgba(123,92,255,.5), inset 0 0 46px rgba(255,255,255,.10);
|
||||
transition:box-shadow .5s ease, background .5s ease;
|
||||
animation:breathe 4.5s ease-in-out infinite;
|
||||
}
|
||||
@keyframes breathe { 0%,100%{transform:scale(1)} 50%{transform:scale(1.045)} }
|
||||
.orb.listening { background:radial-gradient(circle at 34% 28%, #c09cff, #7b5cff 48%, #2c1a6e 80%); box-shadow:0 0 95px rgba(123,92,255,.95), inset 0 0 50px rgba(255,255,255,.18); }
|
||||
.orb.speaking { background:radial-gradient(circle at 34% 28%, #7df2d0, #2fae8b 48%, #0c4a3c 80%); box-shadow:0 0 95px rgba(60,200,160,.85), inset 0 0 50px rgba(255,255,255,.16); }
|
||||
|
||||
.status { display:flex; align-items:center; justify-content:center; gap:8px; font-size:12.5px; color:var(--dim); margin-bottom:18px; min-height:18px; }
|
||||
.dot { width:8px; height:8px; border-radius:50%; background:#555; }
|
||||
.dot.on { background:var(--teal); box-shadow:0 0 10px var(--teal); }
|
||||
|
||||
button {
|
||||
border:0; cursor:pointer; border-radius:999px; font-weight:600;
|
||||
transition:filter .15s, transform .1s, background .2s;
|
||||
}
|
||||
button:hover { filter:brightness(1.14); }
|
||||
button:active { transform:scale(.97); }
|
||||
#btn { padding:13px 40px; font-size:16px; background:var(--purple); color:#fff; box-shadow:0 8px 26px rgba(123,92,255,.4); }
|
||||
#btn.off { background:#252537; color:var(--dim); box-shadow:none; }
|
||||
|
||||
.tools { display:flex; justify-content:center; gap:10px; margin-top:12px; }
|
||||
.tools button { width:42px; height:42px; font-size:18px; background:rgba(255,255,255,.06); color:var(--dim); border:1px solid var(--edge); }
|
||||
.tools button.on { color:#fff; background:rgba(123,92,255,.25); border-color:var(--purple); }
|
||||
|
||||
.panel { display:none; margin-top:18px; text-align:left; background:rgba(0,0,0,.22); border:1px solid var(--edge); border-radius:16px; padding:16px 18px; }
|
||||
.panel.open { display:block; }
|
||||
.row { display:flex; align-items:center; justify-content:space-between; margin:10px 0; gap:12px; }
|
||||
.row label { font-size:13px; color:var(--dim); white-space:nowrap; }
|
||||
.row .val { font-size:13px; font-weight:600; color:var(--ink); min-width:34px; text-align:right; }
|
||||
input[type=range] { -webkit-appearance:none; flex:1; height:5px; border-radius:3px; background:rgba(255,255,255,.12); outline:none; }
|
||||
input[type=range]::-webkit-slider-thumb { -webkit-appearance:none; width:18px; height:18px; border-radius:50%; background:var(--purple); cursor:pointer; box-shadow:0 0 10px rgba(123,92,255,.7); }
|
||||
input[type=text] {
|
||||
flex:1; background:rgba(255,255,255,.06); border:1px solid var(--edge); border-radius:10px;
|
||||
color:var(--ink); padding:9px 12px; font-size:13.5px; outline:none;
|
||||
}
|
||||
input[type=text]:focus { border-color:var(--purple); }
|
||||
.callbtn { width:100%; margin-top:12px; padding:11px; font-size:14px; background:var(--teal); color:#04241a; }
|
||||
|
||||
.transcript {
|
||||
margin-top:20px; text-align:left; max-height:220px; overflow-y:auto;
|
||||
font-size:14px; line-height:1.65; display:flex; flex-direction:column; gap:9px; padding:2px;
|
||||
}
|
||||
.msg { max-width:88%; padding:9px 13px; border-radius:14px; }
|
||||
.msg.u { align-self:flex-end; background:rgba(123,92,255,.16); border:1px solid rgba(123,92,255,.25); }
|
||||
.msg.a { align-self:flex-start; background:rgba(60,200,160,.10); border:1px solid rgba(60,200,160,.22); }
|
||||
.msg .who { font-size:10.5px; font-weight:700; letter-spacing:.6px; text-transform:uppercase; margin-bottom:2px; }
|
||||
.msg.u .who { color:#a68aff; } .msg.a .who { color:var(--teal); }
|
||||
.transcript::-webkit-scrollbar { width:6px; }
|
||||
.transcript::-webkit-scrollbar-thumb { background:rgba(255,255,255,.14); border-radius:3px; }
|
||||
.hint { color:#5f6478; font-size:11.5px; margin-top:14px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="glow a"></div><div class="glow b"></div>
|
||||
|
||||
<div class="card">
|
||||
<h1>The Collector</h1>
|
||||
<div class="sub">smooth operator · local voice</div>
|
||||
|
||||
<div class="stage">
|
||||
<canvas id="ring" width="460" height="460"></canvas>
|
||||
<div class="orb" id="orb"></div>
|
||||
</div>
|
||||
|
||||
<div class="status"><span class="dot" id="dot"></span><span id="status">offline</span></div>
|
||||
|
||||
<button id="btn" onclick="toggle()">Start</button>
|
||||
|
||||
<div class="tools">
|
||||
<button id="gearBtn" onclick="togglePanel('settings')" title="Settings">⚙</button>
|
||||
<button id="callBtn" onclick="togglePanel('call')" title="Make a call">📞</button>
|
||||
</div>
|
||||
|
||||
<div class="panel" id="settingsPanel">
|
||||
<div class="row"><label>Temperature</label><input type="range" id="temp" min="0" max="1.5" step="0.05" value="0.5"><span class="val" id="tempVal">0.50</span></div>
|
||||
<div class="row"><label>Voice speed</label><input type="range" id="speed" min="0.5" max="2.0" step="0.05" value="1.05"><span class="val" id="speedVal">1.05</span></div>
|
||||
<div class="row" style="margin-top:14px;font-size:12px;color:#6b7086;justify-content:space-between">
|
||||
<span>model</span><span id="modelInfo">ornith-1.5:9b-64k</span>
|
||||
</div>
|
||||
<div class="row" style="font-size:12px;color:#6b7086;justify-content:space-between">
|
||||
<span>voice</span><span>lessac-high · STT on GPU</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="panel" id="callPanel">
|
||||
<div class="row"><label>Number</label><input type="text" id="phone" placeholder="+14252800023"></div>
|
||||
<div class="row"><label>What to say</label><input type="text" id="task" placeholder="Introduce yourself as The Collector"></div>
|
||||
<button class="callbtn" onclick="makeCall()">Call</button>
|
||||
<div id="callStatus" style="font-size:12px;color:var(--dim);margin-top:10px;text-align:center"></div>
|
||||
</div>
|
||||
|
||||
<div class="transcript" id="log"></div>
|
||||
<div class="hint" id="hint">tap Start and talk — it listens, thinks, and speaks back</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const orb = document.getElementById('orb'), btn = document.getElementById('btn');
|
||||
const logEl = document.getElementById('log'), statusEl = document.getElementById('status');
|
||||
const dot = document.getElementById('dot'), hint = document.getElementById('hint');
|
||||
const canvas = document.getElementById('ring'), ctx = canvas.getContext('2d');
|
||||
|
||||
let running=false, ws=null, audio=null, micNode=null, playNode=null;
|
||||
let micLevel=0, playLevel=0; // 0..1 current (smoothed)
|
||||
const BARS=72;
|
||||
|
||||
/* ---- audio ring ---- */
|
||||
function drawRing(){
|
||||
ctx.clearRect(0,0,canvas.width,canvas.height);
|
||||
const cx=canvas.width/2, cy=canvas.height/2, r0=88, r1=112;
|
||||
for(let i=0;i<BARS;i++){
|
||||
const a = (i/BARS)*Math.PI*2 - Math.PI/2;
|
||||
// listening = mic level, speaking = play level, else tiny idle
|
||||
let lvl = running ? Math.max(micLevel, playLevel) : 0.06;
|
||||
if(!running) lvl = 0.05;
|
||||
const h = 4 + lvl*38;
|
||||
const x0=cx+Math.cos(a)*r0, y0=cy+Math.sin(a)*r0;
|
||||
const x1=cx+Math.cos(a)*r1, y1=cy+Math.sin(a)*r1;
|
||||
ctx.strokeStyle = running ? (playLevel>micLevel ? '#3cc8a0' : '#a68aff') : 'rgba(160,160,200,.25)';
|
||||
ctx.lineWidth=3; ctx.lineCap='round';
|
||||
ctx.beginPath(); ctx.moveTo(x0,y0); ctx.lineTo(x1,y1); ctx.stroke();
|
||||
}
|
||||
}
|
||||
function tick(){ micLevel*=.90; playLevel*=.88; drawRing(); requestAnimationFrame(tick); }
|
||||
requestAnimationFrame(tick);
|
||||
|
||||
function rmsOf(i16){ let s=0; const n=i16.length; for(let i=0;i<n;i+=4){const v=i16[i]/32768; s+=v*v;} return Math.sqrt(s/(n/4)); }
|
||||
|
||||
/* ---- worklets ---- */
|
||||
const MIC_SRC = `
|
||||
class MicProc extends AudioWorkletProcessor {
|
||||
constructor(){ super(); this._buf = new Float32Array(); }
|
||||
process(inputs){
|
||||
const ch = inputs[0] && inputs[0][0];
|
||||
if(!ch) return true;
|
||||
const m = new Float32Array(this._buf.length + ch.length);
|
||||
m.set(this._buf); m.set(ch, this._buf.length);
|
||||
const ratio = sampleRate/16000;
|
||||
const out = new Int16Array(Math.floor(m.length/ratio));
|
||||
for(let i=0;i<out.length;i++){ const s=Math.max(-1,Math.min(1,m[Math.floor(i*ratio)])); out[i]=s<0?s*0x8000:s*0x7fff; }
|
||||
this._buf = new Float32Array(0);
|
||||
this.port.postMessage(out.buffer,[out.buffer]);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('mic-proc', MicProc);`;
|
||||
|
||||
// Playback worklet: receives Int16/16kHz PCM (as Float32), upsamples to the
|
||||
// AudioContext rate, streams to output.
|
||||
const PLAY_SRC = `
|
||||
class PlayProc extends AudioWorkletProcessor {
|
||||
constructor(){
|
||||
super();
|
||||
this._buf = new Float32Array(0);
|
||||
this._pos = 0;
|
||||
this.port.onmessage = (e) => {
|
||||
const inc = new Float32Array(e.data);
|
||||
const m = new Float32Array(this._buf.length + inc.length);
|
||||
m.set(this._buf);
|
||||
m.set(inc, this._buf.length);
|
||||
this._buf = m;
|
||||
};
|
||||
}
|
||||
process(_, outputs){
|
||||
const ch = outputs[0] && outputs[0][0];
|
||||
if(ch){
|
||||
const step = 16000 / sampleRate; // upsample 16k -> context rate
|
||||
for(let i=0;i<ch.length;i++){
|
||||
if(this._buf.length === 0){ ch[i]=0; continue; }
|
||||
const fi = Math.floor(this._pos);
|
||||
const frac = this._pos - fi;
|
||||
const a = this._buf[fi] || 0;
|
||||
const b = (fi + 1 < this._buf.length) ? this._buf[fi+1] : a;
|
||||
ch[i] = a + (b - a) * frac;
|
||||
this._pos += step;
|
||||
}
|
||||
const consumed = Math.floor(this._pos);
|
||||
if(consumed > 0){ this._buf = this._buf.subarray(consumed); this._pos -= consumed; }
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('play-proc', PlayProc);`;
|
||||
|
||||
const blob = s => URL.createObjectURL(new Blob([s],{type:'application/javascript'}));
|
||||
|
||||
function setState(s){ statusEl.textContent=s; orb.classList.toggle('listening',s==='listening'); orb.classList.toggle('speaking',s==='speaking'); dot.classList.toggle('on',s!=='offline'); }
|
||||
|
||||
async function toggle(){
|
||||
if(running){ stop(); return; }
|
||||
running=true; btn.textContent='Stop'; btn.classList.remove('off'); setState('connecting');
|
||||
audio = new (window.AudioContext||window.webkitAudioContext)();
|
||||
if(audio.state === 'suspended'){ await audio.resume(); }
|
||||
await audio.audioWorklet.addModule(blob(MIC_SRC));
|
||||
const stream = await navigator.mediaDevices.getUserMedia({audio:true});
|
||||
const src = audio.createMediaStreamSource(stream);
|
||||
micNode = new AudioWorkletNode(audio,'mic-proc'); src.connect(micNode);
|
||||
await audio.audioWorklet.addModule(blob(PLAY_SRC));
|
||||
playNode = new AudioWorkletNode(audio,'play-proc'); playNode.connect(audio.destination);
|
||||
|
||||
const proto = location.protocol==='https:'?'wss':'ws';
|
||||
ws = new WebSocket(proto+'://'+location.host+'/ws'); ws.binaryType='arraybuffer';
|
||||
ws.onopen = ()=>{ setState('listening'); hint.textContent='listening — just talk'; };
|
||||
ws.onmessage = ev => {
|
||||
if(typeof ev.data==='string'){
|
||||
try{ const m=JSON.parse(ev.data);
|
||||
if(m.type==='transcript'){ addMsg(m.role,m.text);
|
||||
if(m.role==='assistant'){ setState('speaking'); setTimeout(()=>running&&setState('listening'),1400); }
|
||||
}
|
||||
}catch(e){}
|
||||
} else {
|
||||
const i16=new Int16Array(ev.data); const f32=new Float32Array(i16.length);
|
||||
for(let i=0;i<i16.length;i++) f32[i]=i16[i]/32768;
|
||||
playLevel=Math.max(playLevel,rmsOf(i16));
|
||||
playNode.port.postMessage(f32.buffer,[f32.buffer]);
|
||||
}
|
||||
};
|
||||
ws.onclose=()=>{ if(running) stop(); };
|
||||
ws.onerror=()=>{ statusEl.textContent='error'; };
|
||||
micNode.port.onmessage = ev => {
|
||||
const i16=new Int16Array(ev.data); micLevel=Math.max(micLevel,rmsOf(i16));
|
||||
if(ws&&ws.readyState===1) ws.send(ev.data);
|
||||
};
|
||||
}
|
||||
|
||||
function stop(){
|
||||
running=false; btn.textContent='Start'; btn.classList.add('off'); setState('offline');
|
||||
hint.textContent='tap Start and talk — it listens, thinks, and speaks back';
|
||||
if(ws){try{ws.close();}catch(e){} ws=null;}
|
||||
if(micNode){try{micNode.disconnect();}catch(e){} micNode=null;}
|
||||
if(playNode){try{playNode.disconnect();}catch(e){} playNode=null;}
|
||||
if(audio){try{audio.close();}catch(e){} audio=null;}
|
||||
}
|
||||
|
||||
function addMsg(role,text){
|
||||
const who = role==='user' ? 'You' : 'Collector';
|
||||
const d=document.createElement('div'); d.className='msg '+role;
|
||||
d.innerHTML='<div class="who">'+who+'</div>'+escapeHtml(text);
|
||||
logEl.appendChild(d); logEl.scrollTop=logEl.scrollHeight;
|
||||
}
|
||||
|
||||
/* ---- settings ---- */
|
||||
function togglePanel(which){
|
||||
const s=document.getElementById('settingsPanel'), c=document.getElementById('callPanel');
|
||||
const g=document.getElementById('gearBtn'), cb=document.getElementById('callBtn');
|
||||
if(which==='settings'){ s.classList.toggle('open'); c.classList.remove('open'); g.classList.toggle('on',s.classList.contains('open')); cb.classList.remove('on'); }
|
||||
else { c.classList.toggle('open'); s.classList.remove('open'); cb.classList.toggle('on',c.classList.contains('open')); g.classList.remove('on'); }
|
||||
}
|
||||
const temp=document.getElementById('temp'), speed=document.getElementById('speed');
|
||||
function postCfg(o){ fetch('/api/config',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(o)}).catch(()=>{}); }
|
||||
temp.oninput=()=>{ document.getElementById('tempVal').textContent=Number(temp.value).toFixed(2); };
|
||||
temp.onchange=()=>postCfg({temperature:parseFloat(temp.value)});
|
||||
speed.oninput=()=>{ document.getElementById('speedVal').textContent=Number(speed.value).toFixed(2); };
|
||||
speed.onchange=()=>postCfg({speed:parseFloat(speed.value)});
|
||||
|
||||
/* ---- call ---- */
|
||||
async function makeCall(){
|
||||
const to=document.getElementById('phone').value.trim();
|
||||
const task=document.getElementById('task').value.trim();
|
||||
const st=document.getElementById('callStatus');
|
||||
if(!to){ st.textContent='enter a number'; return; }
|
||||
st.textContent='dialing…';
|
||||
try{
|
||||
const r=await fetch('/api/call',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({to,task})});
|
||||
const d=await r.json();
|
||||
st.textContent = d.ok ? ('call placed — '+(task||'intro')) : ('error: '+(d.error||''));
|
||||
}catch(e){ st.textContent='call failed'; }
|
||||
}
|
||||
|
||||
/* ---- load current settings ---- */
|
||||
fetch('/api/settings').then(r=>r.json()).then(d=>{
|
||||
temp.value=d.temperature||0.5; document.getElementById('tempVal').textContent=Number(temp.value).toFixed(2);
|
||||
speed.value=d.speed||1.05; document.getElementById('speedVal').textContent=Number(speed.value).toFixed(2);
|
||||
document.getElementById('modelInfo').textContent=d.model||'ornith';
|
||||
}).catch(()=>{});
|
||||
|
||||
function escapeHtml(s){ return s.replace(/[&<>]/g,c=>({'&':'&','<':'<','>':'>'}[c])); }
|
||||
</script>
|
||||
<a href="https://buymeacoffee.com/r26xrthzttg" target="_blank" rel="noopener" style="display:block;text-align:center;padding:12px;color:#888;font-size:13px;text-decoration:none">☕ Support on Buy Me a Coffee</a>
|
||||
</body>
|
||||
</html>
|
||||
324
static/index.html.bak-bmac-20260923
Normal file
324
static/index.html.bak-bmac-20260923
Normal file
@@ -0,0 +1,324 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>The Collector</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg0:#07070f; --bg1:#0d0d1f; --purple:#7b5cff; --teal:#3cc8a0;
|
||||
--ink:#e9eaf5; --dim:#9aa0b8; --panel:rgba(255,255,255,.045);
|
||||
--edge:rgba(255,255,255,.08); --accent:#7b5cff;
|
||||
}
|
||||
* { box-sizing:border-box; }
|
||||
html,body { margin:0; height:100%; }
|
||||
body {
|
||||
font-family:ui-sans-serif,system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;
|
||||
color:var(--ink); background:
|
||||
radial-gradient(1100px 700px at 15% -10%, #1c1540 0%, transparent 60%),
|
||||
radial-gradient(900px 600px at 100% 110%, #0e2b3d 0%, transparent 55%),
|
||||
linear-gradient(160deg, var(--bg0), var(--bg1));
|
||||
display:flex; align-items:center; justify-content:center;
|
||||
overflow:hidden; height:100dvh;
|
||||
}
|
||||
/* slow drifting nebula (calm, not chaotic) */
|
||||
.glow { position:fixed; border-radius:50%; filter:blur(90px); opacity:.5; pointer-events:none; }
|
||||
.glow.a { width:520px;height:520px;left:-120px;top:-80px; background:#4b3ad0; animation:drift1 26s ease-in-out infinite; }
|
||||
.glow.b { width:420px;height:420px;right:-100px;bottom:-60px; background:#1d5f86; animation:drift2 32s ease-in-out infinite; }
|
||||
@keyframes drift1 { 0%,100%{transform:translate(0,0)} 50%{transform:translate(60px,40px)} }
|
||||
@keyframes drift2 { 0%,100%{transform:translate(0,0)} 50%{transform:translate(-50px,-30px)} }
|
||||
|
||||
.card {
|
||||
position:relative; width:min(520px,94vw); padding:34px 30px 28px;
|
||||
text-align:center; border-radius:28px;
|
||||
background:var(--panel); border:1px solid var(--edge);
|
||||
backdrop-filter:blur(14px); box-shadow:0 30px 80px rgba(0,0,0,.5);
|
||||
}
|
||||
h1 { margin:0; font-size:26px; font-weight:700; letter-spacing:.5px; }
|
||||
.sub { color:var(--dim); font-size:12.5px; margin:5px 0 22px; letter-spacing:.4px; }
|
||||
|
||||
.stage { position:relative; width:230px; height:230px; margin:0 auto 22px; }
|
||||
canvas#ring { position:absolute; inset:0; width:100%; height:100%; }
|
||||
.orb {
|
||||
position:absolute; inset:34px; border-radius:50%;
|
||||
background:radial-gradient(circle at 34% 28%, #a68aff, #6a4be0 48%, #241457 80%);
|
||||
box-shadow:0 0 60px rgba(123,92,255,.5), inset 0 0 46px rgba(255,255,255,.10);
|
||||
transition:box-shadow .5s ease, background .5s ease;
|
||||
animation:breathe 4.5s ease-in-out infinite;
|
||||
}
|
||||
@keyframes breathe { 0%,100%{transform:scale(1)} 50%{transform:scale(1.045)} }
|
||||
.orb.listening { background:radial-gradient(circle at 34% 28%, #c09cff, #7b5cff 48%, #2c1a6e 80%); box-shadow:0 0 95px rgba(123,92,255,.95), inset 0 0 50px rgba(255,255,255,.18); }
|
||||
.orb.speaking { background:radial-gradient(circle at 34% 28%, #7df2d0, #2fae8b 48%, #0c4a3c 80%); box-shadow:0 0 95px rgba(60,200,160,.85), inset 0 0 50px rgba(255,255,255,.16); }
|
||||
|
||||
.status { display:flex; align-items:center; justify-content:center; gap:8px; font-size:12.5px; color:var(--dim); margin-bottom:18px; min-height:18px; }
|
||||
.dot { width:8px; height:8px; border-radius:50%; background:#555; }
|
||||
.dot.on { background:var(--teal); box-shadow:0 0 10px var(--teal); }
|
||||
|
||||
button {
|
||||
border:0; cursor:pointer; border-radius:999px; font-weight:600;
|
||||
transition:filter .15s, transform .1s, background .2s;
|
||||
}
|
||||
button:hover { filter:brightness(1.14); }
|
||||
button:active { transform:scale(.97); }
|
||||
#btn { padding:13px 40px; font-size:16px; background:var(--purple); color:#fff; box-shadow:0 8px 26px rgba(123,92,255,.4); }
|
||||
#btn.off { background:#252537; color:var(--dim); box-shadow:none; }
|
||||
|
||||
.tools { display:flex; justify-content:center; gap:10px; margin-top:12px; }
|
||||
.tools button { width:42px; height:42px; font-size:18px; background:rgba(255,255,255,.06); color:var(--dim); border:1px solid var(--edge); }
|
||||
.tools button.on { color:#fff; background:rgba(123,92,255,.25); border-color:var(--purple); }
|
||||
|
||||
.panel { display:none; margin-top:18px; text-align:left; background:rgba(0,0,0,.22); border:1px solid var(--edge); border-radius:16px; padding:16px 18px; }
|
||||
.panel.open { display:block; }
|
||||
.row { display:flex; align-items:center; justify-content:space-between; margin:10px 0; gap:12px; }
|
||||
.row label { font-size:13px; color:var(--dim); white-space:nowrap; }
|
||||
.row .val { font-size:13px; font-weight:600; color:var(--ink); min-width:34px; text-align:right; }
|
||||
input[type=range] { -webkit-appearance:none; flex:1; height:5px; border-radius:3px; background:rgba(255,255,255,.12); outline:none; }
|
||||
input[type=range]::-webkit-slider-thumb { -webkit-appearance:none; width:18px; height:18px; border-radius:50%; background:var(--purple); cursor:pointer; box-shadow:0 0 10px rgba(123,92,255,.7); }
|
||||
input[type=text] {
|
||||
flex:1; background:rgba(255,255,255,.06); border:1px solid var(--edge); border-radius:10px;
|
||||
color:var(--ink); padding:9px 12px; font-size:13.5px; outline:none;
|
||||
}
|
||||
input[type=text]:focus { border-color:var(--purple); }
|
||||
.callbtn { width:100%; margin-top:12px; padding:11px; font-size:14px; background:var(--teal); color:#04241a; }
|
||||
|
||||
.transcript {
|
||||
margin-top:20px; text-align:left; max-height:220px; overflow-y:auto;
|
||||
font-size:14px; line-height:1.65; display:flex; flex-direction:column; gap:9px; padding:2px;
|
||||
}
|
||||
.msg { max-width:88%; padding:9px 13px; border-radius:14px; }
|
||||
.msg.u { align-self:flex-end; background:rgba(123,92,255,.16); border:1px solid rgba(123,92,255,.25); }
|
||||
.msg.a { align-self:flex-start; background:rgba(60,200,160,.10); border:1px solid rgba(60,200,160,.22); }
|
||||
.msg .who { font-size:10.5px; font-weight:700; letter-spacing:.6px; text-transform:uppercase; margin-bottom:2px; }
|
||||
.msg.u .who { color:#a68aff; } .msg.a .who { color:var(--teal); }
|
||||
.transcript::-webkit-scrollbar { width:6px; }
|
||||
.transcript::-webkit-scrollbar-thumb { background:rgba(255,255,255,.14); border-radius:3px; }
|
||||
.hint { color:#5f6478; font-size:11.5px; margin-top:14px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="glow a"></div><div class="glow b"></div>
|
||||
|
||||
<div class="card">
|
||||
<h1>The Collector</h1>
|
||||
<div class="sub">smooth operator · local voice</div>
|
||||
|
||||
<div class="stage">
|
||||
<canvas id="ring" width="460" height="460"></canvas>
|
||||
<div class="orb" id="orb"></div>
|
||||
</div>
|
||||
|
||||
<div class="status"><span class="dot" id="dot"></span><span id="status">offline</span></div>
|
||||
|
||||
<button id="btn" onclick="toggle()">Start</button>
|
||||
|
||||
<div class="tools">
|
||||
<button id="gearBtn" onclick="togglePanel('settings')" title="Settings">⚙</button>
|
||||
<button id="callBtn" onclick="togglePanel('call')" title="Make a call">📞</button>
|
||||
</div>
|
||||
|
||||
<div class="panel" id="settingsPanel">
|
||||
<div class="row"><label>Temperature</label><input type="range" id="temp" min="0" max="1.5" step="0.05" value="0.5"><span class="val" id="tempVal">0.50</span></div>
|
||||
<div class="row"><label>Voice speed</label><input type="range" id="speed" min="0.5" max="2.0" step="0.05" value="1.05"><span class="val" id="speedVal">1.05</span></div>
|
||||
<div class="row" style="margin-top:14px;font-size:12px;color:#6b7086;justify-content:space-between">
|
||||
<span>model</span><span id="modelInfo">ornith-1.5:9b-64k</span>
|
||||
</div>
|
||||
<div class="row" style="font-size:12px;color:#6b7086;justify-content:space-between">
|
||||
<span>voice</span><span>lessac-high · STT on GPU</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="panel" id="callPanel">
|
||||
<div class="row"><label>Number</label><input type="text" id="phone" placeholder="+14252800023"></div>
|
||||
<div class="row"><label>What to say</label><input type="text" id="task" placeholder="Introduce yourself as The Collector"></div>
|
||||
<button class="callbtn" onclick="makeCall()">Call</button>
|
||||
<div id="callStatus" style="font-size:12px;color:var(--dim);margin-top:10px;text-align:center"></div>
|
||||
</div>
|
||||
|
||||
<div class="transcript" id="log"></div>
|
||||
<div class="hint" id="hint">tap Start and talk — it listens, thinks, and speaks back</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const orb = document.getElementById('orb'), btn = document.getElementById('btn');
|
||||
const logEl = document.getElementById('log'), statusEl = document.getElementById('status');
|
||||
const dot = document.getElementById('dot'), hint = document.getElementById('hint');
|
||||
const canvas = document.getElementById('ring'), ctx = canvas.getContext('2d');
|
||||
|
||||
let running=false, ws=null, audio=null, micNode=null, playNode=null;
|
||||
let micLevel=0, playLevel=0; // 0..1 current (smoothed)
|
||||
const BARS=72;
|
||||
|
||||
/* ---- audio ring ---- */
|
||||
function drawRing(){
|
||||
ctx.clearRect(0,0,canvas.width,canvas.height);
|
||||
const cx=canvas.width/2, cy=canvas.height/2, r0=88, r1=112;
|
||||
for(let i=0;i<BARS;i++){
|
||||
const a = (i/BARS)*Math.PI*2 - Math.PI/2;
|
||||
// listening = mic level, speaking = play level, else tiny idle
|
||||
let lvl = running ? Math.max(micLevel, playLevel) : 0.06;
|
||||
if(!running) lvl = 0.05;
|
||||
const h = 4 + lvl*38;
|
||||
const x0=cx+Math.cos(a)*r0, y0=cy+Math.sin(a)*r0;
|
||||
const x1=cx+Math.cos(a)*r1, y1=cy+Math.sin(a)*r1;
|
||||
ctx.strokeStyle = running ? (playLevel>micLevel ? '#3cc8a0' : '#a68aff') : 'rgba(160,160,200,.25)';
|
||||
ctx.lineWidth=3; ctx.lineCap='round';
|
||||
ctx.beginPath(); ctx.moveTo(x0,y0); ctx.lineTo(x1,y1); ctx.stroke();
|
||||
}
|
||||
}
|
||||
function tick(){ micLevel*=.90; playLevel*=.88; drawRing(); requestAnimationFrame(tick); }
|
||||
requestAnimationFrame(tick);
|
||||
|
||||
function rmsOf(i16){ let s=0; const n=i16.length; for(let i=0;i<n;i+=4){const v=i16[i]/32768; s+=v*v;} return Math.sqrt(s/(n/4)); }
|
||||
|
||||
/* ---- worklets ---- */
|
||||
const MIC_SRC = `
|
||||
class MicProc extends AudioWorkletProcessor {
|
||||
constructor(){ super(); this._buf = new Float32Array(); }
|
||||
process(inputs){
|
||||
const ch = inputs[0] && inputs[0][0];
|
||||
if(!ch) return true;
|
||||
const m = new Float32Array(this._buf.length + ch.length);
|
||||
m.set(this._buf); m.set(ch, this._buf.length);
|
||||
const ratio = sampleRate/16000;
|
||||
const out = new Int16Array(Math.floor(m.length/ratio));
|
||||
for(let i=0;i<out.length;i++){ const s=Math.max(-1,Math.min(1,m[Math.floor(i*ratio)])); out[i]=s<0?s*0x8000:s*0x7fff; }
|
||||
this._buf = new Float32Array(0);
|
||||
this.port.postMessage(out.buffer,[out.buffer]);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('mic-proc', MicProc);`;
|
||||
|
||||
// Playback worklet: receives Int16/16kHz PCM (as Float32), upsamples to the
|
||||
// AudioContext rate, streams to output.
|
||||
const PLAY_SRC = `
|
||||
class PlayProc extends AudioWorkletProcessor {
|
||||
constructor(){
|
||||
super();
|
||||
this._buf = new Float32Array(0);
|
||||
this._pos = 0;
|
||||
this.port.onmessage = (e) => {
|
||||
const inc = new Float32Array(e.data);
|
||||
const m = new Float32Array(this._buf.length + inc.length);
|
||||
m.set(this._buf);
|
||||
m.set(inc, this._buf.length);
|
||||
this._buf = m;
|
||||
};
|
||||
}
|
||||
process(_, outputs){
|
||||
const ch = outputs[0] && outputs[0][0];
|
||||
if(ch){
|
||||
const step = 16000 / sampleRate; // upsample 16k -> context rate
|
||||
for(let i=0;i<ch.length;i++){
|
||||
if(this._buf.length === 0){ ch[i]=0; continue; }
|
||||
const fi = Math.floor(this._pos);
|
||||
const frac = this._pos - fi;
|
||||
const a = this._buf[fi] || 0;
|
||||
const b = (fi + 1 < this._buf.length) ? this._buf[fi+1] : a;
|
||||
ch[i] = a + (b - a) * frac;
|
||||
this._pos += step;
|
||||
}
|
||||
const consumed = Math.floor(this._pos);
|
||||
if(consumed > 0){ this._buf = this._buf.subarray(consumed); this._pos -= consumed; }
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('play-proc', PlayProc);`;
|
||||
|
||||
const blob = s => URL.createObjectURL(new Blob([s],{type:'application/javascript'}));
|
||||
|
||||
function setState(s){ statusEl.textContent=s; orb.classList.toggle('listening',s==='listening'); orb.classList.toggle('speaking',s==='speaking'); dot.classList.toggle('on',s!=='offline'); }
|
||||
|
||||
async function toggle(){
|
||||
if(running){ stop(); return; }
|
||||
running=true; btn.textContent='Stop'; btn.classList.remove('off'); setState('connecting');
|
||||
audio = new (window.AudioContext||window.webkitAudioContext)();
|
||||
if(audio.state === 'suspended'){ await audio.resume(); }
|
||||
await audio.audioWorklet.addModule(blob(MIC_SRC));
|
||||
const stream = await navigator.mediaDevices.getUserMedia({audio:true});
|
||||
const src = audio.createMediaStreamSource(stream);
|
||||
micNode = new AudioWorkletNode(audio,'mic-proc'); src.connect(micNode);
|
||||
await audio.audioWorklet.addModule(blob(PLAY_SRC));
|
||||
playNode = new AudioWorkletNode(audio,'play-proc'); playNode.connect(audio.destination);
|
||||
|
||||
const proto = location.protocol==='https:'?'wss':'ws';
|
||||
ws = new WebSocket(proto+'://'+location.host+'/ws'); ws.binaryType='arraybuffer';
|
||||
ws.onopen = ()=>{ setState('listening'); hint.textContent='listening — just talk'; };
|
||||
ws.onmessage = ev => {
|
||||
if(typeof ev.data==='string'){
|
||||
try{ const m=JSON.parse(ev.data);
|
||||
if(m.type==='transcript'){ addMsg(m.role,m.text);
|
||||
if(m.role==='assistant'){ setState('speaking'); setTimeout(()=>running&&setState('listening'),1400); }
|
||||
}
|
||||
}catch(e){}
|
||||
} else {
|
||||
const i16=new Int16Array(ev.data); const f32=new Float32Array(i16.length);
|
||||
for(let i=0;i<i16.length;i++) f32[i]=i16[i]/32768;
|
||||
playLevel=Math.max(playLevel,rmsOf(i16));
|
||||
playNode.port.postMessage(f32.buffer,[f32.buffer]);
|
||||
}
|
||||
};
|
||||
ws.onclose=()=>{ if(running) stop(); };
|
||||
ws.onerror=()=>{ statusEl.textContent='error'; };
|
||||
micNode.port.onmessage = ev => {
|
||||
const i16=new Int16Array(ev.data); micLevel=Math.max(micLevel,rmsOf(i16));
|
||||
if(ws&&ws.readyState===1) ws.send(ev.data);
|
||||
};
|
||||
}
|
||||
|
||||
function stop(){
|
||||
running=false; btn.textContent='Start'; btn.classList.add('off'); setState('offline');
|
||||
hint.textContent='tap Start and talk — it listens, thinks, and speaks back';
|
||||
if(ws){try{ws.close();}catch(e){} ws=null;}
|
||||
if(micNode){try{micNode.disconnect();}catch(e){} micNode=null;}
|
||||
if(playNode){try{playNode.disconnect();}catch(e){} playNode=null;}
|
||||
if(audio){try{audio.close();}catch(e){} audio=null;}
|
||||
}
|
||||
|
||||
function addMsg(role,text){
|
||||
const who = role==='user' ? 'You' : 'Collector';
|
||||
const d=document.createElement('div'); d.className='msg '+role;
|
||||
d.innerHTML='<div class="who">'+who+'</div>'+escapeHtml(text);
|
||||
logEl.appendChild(d); logEl.scrollTop=logEl.scrollHeight;
|
||||
}
|
||||
|
||||
/* ---- settings ---- */
|
||||
function togglePanel(which){
|
||||
const s=document.getElementById('settingsPanel'), c=document.getElementById('callPanel');
|
||||
const g=document.getElementById('gearBtn'), cb=document.getElementById('callBtn');
|
||||
if(which==='settings'){ s.classList.toggle('open'); c.classList.remove('open'); g.classList.toggle('on',s.classList.contains('open')); cb.classList.remove('on'); }
|
||||
else { c.classList.toggle('open'); s.classList.remove('open'); cb.classList.toggle('on',c.classList.contains('open')); g.classList.remove('on'); }
|
||||
}
|
||||
const temp=document.getElementById('temp'), speed=document.getElementById('speed');
|
||||
function postCfg(o){ fetch('/api/config',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(o)}).catch(()=>{}); }
|
||||
temp.oninput=()=>{ document.getElementById('tempVal').textContent=Number(temp.value).toFixed(2); };
|
||||
temp.onchange=()=>postCfg({temperature:parseFloat(temp.value)});
|
||||
speed.oninput=()=>{ document.getElementById('speedVal').textContent=Number(speed.value).toFixed(2); };
|
||||
speed.onchange=()=>postCfg({speed:parseFloat(speed.value)});
|
||||
|
||||
/* ---- call ---- */
|
||||
async function makeCall(){
|
||||
const to=document.getElementById('phone').value.trim();
|
||||
const task=document.getElementById('task').value.trim();
|
||||
const st=document.getElementById('callStatus');
|
||||
if(!to){ st.textContent='enter a number'; return; }
|
||||
st.textContent='dialing…';
|
||||
try{
|
||||
const r=await fetch('/api/call',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({to,task})});
|
||||
const d=await r.json();
|
||||
st.textContent = d.ok ? ('call placed — '+(task||'intro')) : ('error: '+(d.error||''));
|
||||
}catch(e){ st.textContent='call failed'; }
|
||||
}
|
||||
|
||||
/* ---- load current settings ---- */
|
||||
fetch('/api/settings').then(r=>r.json()).then(d=>{
|
||||
temp.value=d.temperature||0.5; document.getElementById('tempVal').textContent=Number(temp.value).toFixed(2);
|
||||
speed.value=d.speed||1.05; document.getElementById('speedVal').textContent=Number(speed.value).toFixed(2);
|
||||
document.getElementById('modelInfo').textContent=d.model||'ornith';
|
||||
}).catch(()=>{});
|
||||
|
||||
function escapeHtml(s){ return s.replace(/[&<>]/g,c=>({'&':'&','<':'<','>':'>'}[c])); }
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
30
tunnel.sh
Normal file
30
tunnel.sh
Normal file
@@ -0,0 +1,30 @@
|
||||
#!/bin/bash
|
||||
# Start a Cloudflare quick tunnel for The Collector, extract the trycloudflare
|
||||
# URL, and self-register it with the app (/api/config). Blocks while the tunnel
|
||||
# runs (intended for a systemd service).
|
||||
LOG=/opt/collector/tunnel.log
|
||||
APP=http://127.0.0.1:8766
|
||||
|
||||
: > "$LOG"
|
||||
/usr/local/bin/cloudflared tunnel --url http://localhost:8766 --no-autoupdate >> "$LOG" 2>&1 &
|
||||
CF_PID=$!
|
||||
|
||||
trap 'kill $CF_PID 2>/dev/null' EXIT
|
||||
|
||||
URL=""
|
||||
for _ in $(seq 1 45); do
|
||||
URL=$(grep -oE 'https://[a-z0-9-]+\.trycloudflare\.com' "$LOG" | head -1)
|
||||
[ -n "$URL" ] && break
|
||||
sleep 2
|
||||
done
|
||||
|
||||
if [ -n "$URL" ]; then
|
||||
echo "TUNNEL_URL=$URL"
|
||||
curl -s -X POST "$APP/api/config" -H 'Content-Type: application/json' \
|
||||
-d "{\"public_base_url\":\"$URL\"}" >/dev/null 2>&1
|
||||
echo "registered with app"
|
||||
else
|
||||
echo "TUNNEL_FAILED"
|
||||
fi
|
||||
|
||||
wait $CF_PID
|
||||
52
voice_test.py
Normal file
52
voice_test.py
Normal file
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env python3
|
||||
"""End-to-end voice test: synthesize speech, push it through the /ws browser
|
||||
channel, assert we get back a transcript + reply audio (PCM16)."""
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, "/opt/collector")
|
||||
from app import TTS, CFG
|
||||
|
||||
import websockets
|
||||
|
||||
tts = TTS(CFG["tts"])
|
||||
tts.load()
|
||||
pcm = tts.synthesize_16k("Hello Collector, tell me who you are in one sentence.")
|
||||
|
||||
FRAME = 640 # 20ms @ 16kHz
|
||||
|
||||
|
||||
async def main():
|
||||
async with websockets.connect("ws://127.0.0.1:8766/ws") as ws:
|
||||
await asyncio.sleep(6) # let server warm (STT+TTS+LLM)
|
||||
# speech in 20ms frames
|
||||
for i in range(0, len(pcm), FRAME):
|
||||
await ws.send(pcm[i:i + FRAME])
|
||||
await asyncio.sleep(0.02)
|
||||
# 1s silence -> trigger VAD turn end
|
||||
silence = b"\x00" * FRAME
|
||||
for _ in range(50):
|
||||
await ws.send(silence)
|
||||
await asyncio.sleep(0.02)
|
||||
|
||||
got_text = got_audio = False
|
||||
deadline = asyncio.get_event_loop().time() + 60
|
||||
while asyncio.get_event_loop().time() < deadline:
|
||||
try:
|
||||
msg = await asyncio.wait_for(ws.recv(), timeout=40)
|
||||
except Exception:
|
||||
break
|
||||
if isinstance(msg, str):
|
||||
print("TRANSCRIPT:", msg)
|
||||
got_text = True
|
||||
else:
|
||||
print("AUDIO bytes:", len(msg))
|
||||
got_audio = True
|
||||
if got_text and got_audio:
|
||||
break
|
||||
|
||||
print("VOICE_TEST", "PASS" if (got_text and got_audio) else "FAIL")
|
||||
|
||||
|
||||
asyncio.run(main())
|
||||
Reference in New Issue
Block a user