diff --git a/docs/superpowers/specs/2026-07-23-esp32-sensor-node-design.md b/docs/superpowers/specs/2026-07-23-esp32-sensor-node-design.md new file mode 100644 index 0000000..1ac839c --- /dev/null +++ b/docs/superpowers/specs/2026-07-23-esp32-sensor-node-design.md @@ -0,0 +1,281 @@ +# ESP32-P4 Sensor Node — pairing, ingestion, live dashboard, firmware + +Third sub-project of the broader arc, and the first hardware/firmware work +in this repo. The physical device (an ESP32-P4, referenced in the README's +Reliquary section as part of the future "Ultimate Quantum Box" line) joins +the seeker's home WiFi and streams sensor readings to their own account on +the live site, rendered in real time. Built as parallel workstreams against +the contract below. + +## Honesty-policy note (binding on every workstream) + +This app's whole ethos is "real signal processing on real data, and it says +so when something is unverified" (see Spirit Radio's `HARDWARE PASS +REQUIRED` marking). Nobody here has physical ESP32-P4 hardware to flash and +test against. Firmware workstreams must write real, structurally correct +ESP-IDF code and mark it clearly as **unverified against real hardware** in +both a header comment and the final report — exactly the same honesty +convention `frontend/src/lib/sdr.ts` already uses. Do not claim something +works on-device when it has only compiled/been reasoned about. + +## Contract (binding for all workstreams) + +**Device identity & pairing.** A `Device` belongs to exactly one `User`. +Pairing flow: the seeker generates a device from their account (name + +optional sensor-type hint), the backend returns a **raw pairing token once** +(never stored or retrievable again — identical convention to +`generate_session_token()`/`hash_token()` in +`backend/app/models/auth_session.py`: `secrets.token_urlsafe(32)` raw, +`hashlib.sha256(raw).hexdigest()` stored). The seeker flashes/configures +that raw token (plus their WiFi credentials) into the device firmware. The +device authenticates every request with `Authorization: Bearer `; the backend looks up by the token's hash, never the raw value. + +**`Device` model** (new table `devices`): `id (uuid)`, `user_id` FK, +`name: str`, `token_hash: str` (unique, indexed, same shape as +`AuthSession.token_hash`), `created_at`, `last_seen_at: datetime | None`. + +**Ingestion endpoint** — `POST /api/device/telemetry`, authenticated via the +device bearer token (NOT the user's session cookie — this is a headless +client). Body: + +```json +{ + "readings": [ + {"sensor_type": "presence", "value": 1, "unit": "bool", "metadata": {}}, + {"sensor_type": "temperature", "value": 21.4, "unit": "c", "metadata": {}}, + {"sensor_type": "humidity", "value": 47.2, "unit": "pct", "metadata": {}}, + {"sensor_type": "pressure", "value": 1013.2, "unit": "hpa", "metadata": {}} + ] +} +``` + +`sensor_type` is a free-form string, not an enum — this is the whole point +per the project's brief ("add all sorts of sensors... anything you can +think of"). The backend does not validate against a fixed sensor list, only +against shape (string type, numeric or array value, string unit, dict +metadata) and sane bounds (cap `readings` array length, e.g. 64 per +request, and reject bodies over a reasonable size, e.g. 16KB, to prevent a +misbehaving or malicious device from flooding the endpoint — reuse +`app.rate_limit.RateLimiter` for a per-device rate cap too, e.g. 1 +request/second sustained is generous for sensor telemetry). + +**This is the actual business model, not a side dashboard — say it plainly +so nobody undersells the scope: paired hardware must be able to summon +spirits, the same as the browser-based modes already do.** A live dashboard +of numbers is not the point; feeding the séance's real anomaly-detection +pipeline is. Concretely: hardware sensor readings become a **sixth anomaly +source** in the existing `/ws/session` protocol (alongside `wire`, `evp`, +`radio`, `emf`), not a separate system. Per `(user, device_id, +sensor_type)`, maintain a rolling baseline and flag anomalies — reuse +`detect_wire_spike`'s statistical shape (min sample count, an absolute +floor, 3σ + relative threshold) for continuous sensors (temperature, +humidity, pressure, any future numeric sensor), and a simple +state-transition rule for discrete/boolean sensors (presence going +false→true is the anomaly, not a statistical spike). When a user has an +active séance session open with a paired device sending data, detected +hardware anomalies get pushed into that session's `state.anomalies` via +the same `{"type": "anomaly", "source": ..., "frequency": ..., +"magnitude": ...}` shape the frontend already sends for the other four +modes — `source` should be the device's `sensor_type` (e.g. `"presence"`, +`"temperature"`) so it fingerprints and Codex-displays distinctly, same as +`"wire"`/`"evp"`/etc. do today. This means summoning via hardware reuses +100% of the existing signature/mint/Codex pipeline — no new mint logic +anywhere. Requires locating the user's live `SeanceState` (if any) from the ingestion +handler — `backend/app/ws.py` does not currently track active sessions by +user anywhere (each `session_socket` connection only holds its own local +`state` variable), so this workstream needs to add a minimal in-process +registry: a module-level `dict[uuid.UUID, SeanceState]` (or a list, if a +user could plausibly have multiple simultaneous séance tabs open — your +call, a single most-recent-session mapping is a reasonable v1), populated +when `session_socket` establishes a session and removed in its `finally`/ +disconnect cleanup. Keep it a plain in-process dict — no external state +store needed at this scale, same reasoning as the ingestion→dashboard +pub/sub above. + +**No permanent storage of every reading.** This mirrors how Wire Ghost +telemetry isn't persisted either (only anomaly *events* are). On receipt, +the backend updates `Device.last_seen_at` and **broadcasts the readings +live** to the owning user if they have an active dashboard WS connection +(see below) — it does not write each reading to Postgres. This keeps the +database bounded regardless of how chatty a device is. + +**Live dashboard WS** — new endpoint `/ws/device-feed`, authenticated by the +normal `qm_session` cookie (this one IS a browser client, unlike the +ingestion endpoint). On connect, the server sends the user's device list +(`{"type": "devices", "devices": [{"id", "name", "last_seen_at"}, ...]}`). +As telemetry arrives from any of that user's paired devices via the +ingestion endpoint, relay it: `{"type": "reading", "device_id": str, +"sensor_type": str, "value": ..., "unit": str, "metadata": {}, "at": +iso8601}`. This is a fan-out problem (ingestion endpoint receives, dashboard +WS receives) — implement via an in-process pub/sub keyed by `user_id` (a +simple `dict[uuid, list[WebSocket]]` registry is fine at this scale, no +external message broker needed; check how `backend/app/ws.py`'s existing +`SeanceState`/sender-task pattern works and follow the same +single-sender-task-per-socket convention so concurrent producers never +interleave, exactly like the existing séance WS already guards against). + +**`POST /api/device` / `GET /api/device`** — REST endpoints (authenticated +by session cookie) to create a device (returns the one-time raw token) and +list the user's devices (name, id, last_seen_at — never the token or its +hash). + +## Workstream G (backend) — pairing, ingestion, live broadcast + +New `backend/app/models/device.py` (the `Device` model per the contract), +new `backend/app/routes/device.py` (the REST pairing endpoints: create +device / list devices), new ingestion handling and the `/ws/device-feed` +WS endpoint (extend `backend/app/ws.py` or add a new router — your call on +file organization, follow this codebase's existing convention of one +router per concern in `backend/app/routes/`). Idempotent migration: since +`devices` is a brand-new table, `Base.metadata.create_all` in `lifespan` +handles it automatically — no ALTER TABLE needed. Rate-limit the ingestion +endpoint per-device (reuse `RateLimiter`). Reject malformed/oversized +payloads with clear 4xx errors, never a 500. **Scope boundary: you own +pairing, ingestion auth/validation, and the dashboard pub/sub broadcast. +You do NOT own feeding readings into the séance/summon pipeline — that's +Workstream K, running in parallel, which depends on your ingestion +endpoint's shape but not your code.** To make that possible without a +sequencing dependency, structure your ingestion handler so the actual +per-reading processing is a clearly separated, small internal function +(e.g. `_process_reading(device, reading)`) that Workstream K's merge can +extend/call into — document this handoff point in your report. Tests: +token generation/hashing round-trip, auth rejection on bad/missing token, +payload validation (size caps, shape validation, rejecting garbage +sensor_type/value types gracefully), the pub/sub fan-out (a reading posted +while a dashboard WS is connected for that user arrives on the socket; a +reading posted for a device with no connected owner doesn't error), +`last_seen_at` updates. + +## Workstream K (backend) — hardware anomalies feed the summon pipeline + +This is the actual point of the whole project (see the contract's +"business model" note above) — paired hardware must be able to summon +spirits like the browser-based modes already do. Depends conceptually on +Workstream G's ingestion endpoint shape (per the contract, not on +Workstream G's actual code — build against the documented `POST +/api/device/telemetry` request shape and assume it exists). Implement: + +1. Per-`(user_id, device_id, sensor_type)` rolling-baseline anomaly + detection — adapt `detect_wire_spike`'s shape (`backend/app/telemetry.py`: + min sample count, an absolute floor, 3σ + relative threshold) for + continuous sensors; for discrete/boolean sensors (e.g. `presence`), the + anomaly is a false→true state transition, not a statistical spike. + New `backend/app/device_anomaly.py` (pure-ish functions, a small + per-key rolling-history cache) is a reasonable home for this. +2. A minimal in-process active-session registry: `backend/app/ws.py` + currently has no way to look up a live `SeanceState` by `user_id` (each + `session_socket` connection only holds its own local `state` variable) + — add a module-level `dict[uuid.UUID, SeanceState]`, populated when a + session starts and cleaned up in `session_socket`'s disconnect/`finally` + handling. +3. Wire the two together: when an ingested reading is flagged anomalous + and the owning user has an active séance session, push + `{"type": "anomaly", "source": , "frequency": ..., + "magnitude": ...}` into that session's `state.anomalies` via the same + path the existing four modes already use (check `_handle_anomaly` in + `backend/app/ws.py`) — `source` should be the sensor_type string itself + (e.g. `"presence"`, `"temperature"`) so it fingerprints/displays + distinctly in the Codex, same as `"wire"`/`"evp"`/etc. do today. Pick + reasonable `frequency`/`magnitude` mappings per sensor_type (e.g. for a + numeric sensor, magnitude could be the deviation-from-baseline in the + sensor's own units; frequency can be a stable per-sensor-type constant + or derived from the reading's value — document your choices). +4. Since you can't see Workstream G's actual ingestion code, implement + your detection+push logic as a self-contained function + (`process_device_reading_for_summon(user_id, device_id, sensor_type, + value, ...) -> None`) that the controller will wire into Workstream G's + `_process_reading` handoff point during the merge — don't try to edit + `backend/app/routes/device.py` or the ingestion handler yourself, since + it won't exist yet in your worktree. + +Tests: the statistical detector (reuse `test_telemetry.py`'s test shapes +for `detect_wire_spike` as a model — min-samples guard, floor guard, +threshold guard, all independently verified), the boolean transition +detector, and an integration-style test that constructs a fake +`SeanceState` registered in your registry and confirms a flagged anomaly +correctly appends to `state.anomalies` in the right shape. + +## Workstream H (frontend) — pairing UI + live dashboard + +New `frontend/src/pages/DevicesPage.tsx` (or extend the existing +`/inventory` area if that reads as more consistent — your call, check +current nav structure) — a "pair a new device" flow (name input, calls +`POST /api/device`, shows the raw token **exactly once** with a clear "copy +this now, it cannot be shown again" warning, matching how real API-key UIs +handle one-time secrets) and a live dashboard subscribing to +`/ws/device-feed`, rendering each connected device's most recent reading +per `sensor_type` it has seen, updating in place as new readings arrive. +Since `sensor_type` is open-ended, render generically (a card per device, +a row per distinct sensor_type seen so far, value + unit) rather than +hardcoding presence/temperature/humidity/pressure as fixed fields — reuse +`TelemetryReadout.tsx`'s visual conventions as a starting point but make it +dynamic. "Hacker witch" styling per the established art direction: this +reads naturally as the most "hacker" of all the surfaces built so far +(it's literally a live instrument panel) — lean into that side harder than +the occult side here specifically, terminal/telemetry aesthetics, but keep +the existing dark/violet palette. Tests: pairing flow (token shown once, +warning present), dashboard rendering with mock WS frames for multiple +devices/sensor types, generic rendering for an unrecognized sensor_type +(must not crash — unknown types render with their raw value/unit, no +special-casing required). + +## Workstream I (firmware, ESP-IDF C) — core sensor node + +New top-level `firmware/esp32p4-sensor-node/` directory (sibling to +`backend/`/`frontend/`/`deploy/`) — a real ESP-IDF project skeleton +(`CMakeLists.txt`, `sdkconfig.defaults`, `main/` component) implementing: +WiFi station-mode connection (SSID/password from a `main/device_config.h` +the seeker fills in before flashing — a full provisioning UI is out of +scope, document the manual-config step clearly in a README), an HTTP client +task that POSTs to `/api/device/telemetry` with the bearer token from that +same config header, and I2C drivers for a **BME280** (temperature/humidity/ +pressure — a common, well-documented sensor, pick this as the concrete +default since no specific part number was given) and a simple presence +sensor (an mmWave module like the LD2410 over UART, or a basic PIR over +GPIO — your call, document which and why; LD2410 is recommended since it +gives distance/motion data beyond a boolean, richer for this app's +"believable" ethos). Structure the reporting loop so adding a new sensor +type is genuinely easy — a small internal registry/interface (e.g. a +`sensor_driver_t` struct with an init + read function pointer) that the +main loop iterates over to build the `readings` array, not one-off hardcoded +POST calls per sensor. Mark the whole directory's top-level README with the +honesty-policy note from above. No unit tests in the traditional sense (this +is embedded C, not covered by this repo's pytest/vitest suites) — instead, +write a `firmware/esp32p4-sensor-node/README.md` documenting build steps +(`idf.py build`), exactly what's been verified (compiles cleanly against +ESP-IDF, structurally sound) versus not (anything requiring physical +hardware), and wiring/pinout assumptions for the BME280 (I2C) and presence +sensor (UART or GPIO) so a real owner can adapt it to their actual wiring. + +## Workstream J (firmware, ESP-IDF C) — RTL-SDR experimental module + +A clearly-separated **experimental, stretch-goal** module in the same +firmware project (`firmware/esp32p4-sensor-node/main/rtlsdr_experimental.c` +or a dedicated `components/` subdir) exploring USB-host communication with +an RTL2832U-based dongle via ESP32-P4's USB-OTG host capability. Be +honest in the README about the real constraint: this is genuinely at the +edge of feasibility for an MCU (wideband IQ sample rates and FFT processing +are demanding relative to ESP32-P4's compute, even with its AI +accelerator) — the realistic architecture is likely "device pulls raw IQ +samples off the dongle via USB host and streams them upstream for the +backend to FFT," not on-device spectrum analysis, and even that needs +verifying against real USB throughput. Implement what's structurally +reasonable (USB host init, basic RTL2832U vendor-command sequence adapted +from the existing `frontend/src/lib/sdr.ts` WebUSB implementation's +documented init sequence — read that file for the protocol details already +researched for this project) but mark this entire module as the least +certain part of the whole build, both in code comments and the final +report. If a full implementation isn't reasonably achievable in scope, +a well-documented partial attempt plus a clear write-up of exactly where +it stops and why is a legitimate, honest outcome — better than a +fabricated "it works" claim. + +## Explicitly out of scope here + +Thermal camera support, a full BLE/WiFi-AP provisioning UX (manual config +header is the v1 approach), on-device spectrum analysis/FFT (Workstream J +targets raw IQ passthrough at most), persisting historical readings for +charts/graphs over time (live-only for this spec) — each a reasonable +later spec once the core pairing+ingestion+dashboard+firmware skeleton +exists and (ideally) someone has tested against real hardware.