qtalker---/docs/superpowers/specs/2026-07-23-esp32-sensor-node-design.md

ESP32-P4 Sensor Node — pairing, ingestion, live dashboard, firmware

Third sub-project of the broader arc, and the first hardware/firmware work in this repo. The physical device (an ESP32-P4, referenced in the README's Reliquary section as part of the future "Ultimate Quantum Box" line) joins the seeker's home WiFi and streams sensor readings to their own account on the live site, rendered in real time. Built as parallel workstreams against the contract below.

Honesty-policy note (binding on every workstream)

This app's whole ethos is "real signal processing on real data, and it says so when something is unverified" (see Spirit Radio's HARDWARE PASS REQUIRED marking). Nobody here has physical ESP32-P4 hardware to flash and test against. Firmware workstreams must write real, structurally correct ESP-IDF code and mark it clearly as unverified against real hardware in both a header comment and the final report — exactly the same honesty convention frontend/src/lib/sdr.ts already uses. Do not claim something works on-device when it has only compiled/been reasoned about.

Contract (binding for all workstreams)

Device identity & pairing. A Device belongs to exactly one User. Pairing flow: the seeker generates a device from their account (name + optional sensor-type hint), the backend returns a raw pairing token once (never stored or retrievable again — identical convention to generate_session_token()/hash_token() in backend/app/models/auth_session.py: secrets.token_urlsafe(32) raw, hashlib.sha256(raw).hexdigest() stored). The seeker flashes/configures that raw token (plus their WiFi credentials) into the device firmware. The device authenticates every request with Authorization: Bearer <raw token>; the backend looks up by the token's hash, never the raw value.

Device model (new table devices): id (uuid), user_id FK, name: str, token_hash: str (unique, indexed, same shape as AuthSession.token_hash), created_at, last_seen_at: datetime | None.

Ingestion endpoint — POST /api/device/telemetry, authenticated via the device bearer token (NOT the user's session cookie — this is a headless client). Body:

{
  "readings": [
    {"sensor_type": "presence", "value": 1, "unit": "bool", "metadata": {}},
    {"sensor_type": "temperature", "value": 21.4, "unit": "c", "metadata": {}},
    {"sensor_type": "humidity", "value": 47.2, "unit": "pct", "metadata": {}},
    {"sensor_type": "pressure", "value": 1013.2, "unit": "hpa", "metadata": {}}
  ]
}

sensor_type is a free-form string, not an enum — this is the whole point per the project's brief ("add all sorts of sensors... anything you can think of"). The backend does not validate against a fixed sensor list, only against shape (string type, numeric or array value, string unit, dict metadata) and sane bounds (cap readings array length, e.g. 64 per request, and reject bodies over a reasonable size, e.g. 16KB, to prevent a misbehaving or malicious device from flooding the endpoint — reuse app.rate_limit.RateLimiter for a per-device rate cap too, e.g. 1 request/second sustained is generous for sensor telemetry).

This is the actual business model, not a side dashboard — say it plainly so nobody undersells the scope: paired hardware must be able to summon spirits, the same as the browser-based modes already do. A live dashboard of numbers is not the point; feeding the séance's real anomaly-detection pipeline is. Concretely: hardware sensor readings become a sixth anomaly source in the existing /ws/session protocol (alongside wire, evp, radio, emf), not a separate system. Per (user, device_id, sensor_type), maintain a rolling baseline and flag anomalies — reuse detect_wire_spike's statistical shape (min sample count, an absolute floor, 3σ + relative threshold) for continuous sensors (temperature, humidity, pressure, any future numeric sensor), and a simple state-transition rule for discrete/boolean sensors (presence going false→true is the anomaly, not a statistical spike). When a user has an active séance session open with a paired device sending data, detected hardware anomalies get pushed into that session's state.anomalies via the same {"type": "anomaly", "source": ..., "frequency": ..., "magnitude": ...} shape the frontend already sends for the other four modes — source should be the device's sensor_type (e.g. "presence", "temperature") so it fingerprints and Codex-displays distinctly, same as "wire"/"evp"/etc. do today. This means summoning via hardware reuses 100% of the existing signature/mint/Codex pipeline — no new mint logic anywhere. Requires locating the user's live SeanceState (if any) from the ingestion handler — backend/app/ws.py does not currently track active sessions by user anywhere (each session_socket connection only holds its own local state variable), so this workstream needs to add a minimal in-process registry: a module-level dict[uuid.UUID, SeanceState] (or a list, if a user could plausibly have multiple simultaneous séance tabs open — your call, a single most-recent-session mapping is a reasonable v1), populated when session_socket establishes a session and removed in its finally/ disconnect cleanup. Keep it a plain in-process dict — no external state store needed at this scale, same reasoning as the ingestion→dashboard pub/sub above.

No permanent storage of every reading. This mirrors how Wire Ghost telemetry isn't persisted either (only anomaly events are). On receipt, the backend updates Device.last_seen_at and broadcasts the readings live to the owning user if they have an active dashboard WS connection (see below) — it does not write each reading to Postgres. This keeps the database bounded regardless of how chatty a device is.

Live dashboard WS — new endpoint /ws/device-feed, authenticated by the normal qm_session cookie (this one IS a browser client, unlike the ingestion endpoint). On connect, the server sends the user's device list ({"type": "devices", "devices": [{"id", "name", "last_seen_at"}, ...]}). As telemetry arrives from any of that user's paired devices via the ingestion endpoint, relay it: {"type": "reading", "device_id": str, "sensor_type": str, "value": ..., "unit": str, "metadata": {}, "at": iso8601}. This is a fan-out problem (ingestion endpoint receives, dashboard WS receives) — implement via an in-process pub/sub keyed by user_id (a simple dict[uuid, list[WebSocket]] registry is fine at this scale, no external message broker needed; check how backend/app/ws.py's existing SeanceState/sender-task pattern works and follow the same single-sender-task-per-socket convention so concurrent producers never interleave, exactly like the existing séance WS already guards against).

POST /api/device / GET /api/device — REST endpoints (authenticated by session cookie) to create a device (returns the one-time raw token) and list the user's devices (name, id, last_seen_at — never the token or its hash).

Workstream G (backend) — pairing, ingestion, live broadcast

New backend/app/models/device.py (the Device model per the contract), new backend/app/routes/device.py (the REST pairing endpoints: create device / list devices), new ingestion handling and the /ws/device-feed WS endpoint (extend backend/app/ws.py or add a new router — your call on file organization, follow this codebase's existing convention of one router per concern in backend/app/routes/). Idempotent migration: since devices is a brand-new table, Base.metadata.create_all in lifespan handles it automatically — no ALTER TABLE needed. Rate-limit the ingestion endpoint per-device (reuse RateLimiter). Reject malformed/oversized payloads with clear 4xx errors, never a 500. Scope boundary: you own pairing, ingestion auth/validation, and the dashboard pub/sub broadcast. You do NOT own feeding readings into the séance/summon pipeline — that's Workstream K, running in parallel, which depends on your ingestion endpoint's shape but not your code. To make that possible without a sequencing dependency, structure your ingestion handler so the actual per-reading processing is a clearly separated, small internal function (e.g. _process_reading(device, reading)) that Workstream K's merge can extend/call into — document this handoff point in your report. Tests: token generation/hashing round-trip, auth rejection on bad/missing token, payload validation (size caps, shape validation, rejecting garbage sensor_type/value types gracefully), the pub/sub fan-out (a reading posted while a dashboard WS is connected for that user arrives on the socket; a reading posted for a device with no connected owner doesn't error), last_seen_at updates.

Workstream K (backend) — hardware anomalies feed the summon pipeline

This is the actual point of the whole project (see the contract's "business model" note above) — paired hardware must be able to summon spirits like the browser-based modes already do. Depends conceptually on Workstream G's ingestion endpoint shape (per the contract, not on Workstream G's actual code — build against the documented POST /api/device/telemetry request shape and assume it exists). Implement:

1. Per-(user_id, device_id, sensor_type) rolling-baseline anomaly detection — adapt detect_wire_spike's shape (backend/app/telemetry.py: min sample count, an absolute floor, 3σ + relative threshold) for continuous sensors; for discrete/boolean sensors (e.g. presence), the anomaly is a false→true state transition, not a statistical spike. New backend/app/device_anomaly.py (pure-ish functions, a small per-key rolling-history cache) is a reasonable home for this.
2. A minimal in-process active-session registry: backend/app/ws.py currently has no way to look up a live SeanceState by user_id (each session_socket connection only holds its own local state variable) — add a module-level dict[uuid.UUID, SeanceState], populated when a session starts and cleaned up in session_socket's disconnect/finally handling.
3. Wire the two together: when an ingested reading is flagged anomalous and the owning user has an active séance session, push {"type": "anomaly", "source": <sensor_type>, "frequency": ..., "magnitude": ...} into that session's state.anomalies via the same path the existing four modes already use (check _handle_anomaly in backend/app/ws.py) — source should be the sensor_type string itself (e.g. "presence", "temperature") so it fingerprints/displays distinctly in the Codex, same as "wire"/"evp"/etc. do today. Pick reasonable frequency/magnitude mappings per sensor_type (e.g. for a numeric sensor, magnitude could be the deviation-from-baseline in the sensor's own units; frequency can be a stable per-sensor-type constant or derived from the reading's value — document your choices).
4. Since you can't see Workstream G's actual ingestion code, implement your detection+push logic as a self-contained function (process_device_reading_for_summon(user_id, device_id, sensor_type, value, ...) -> None) that the controller will wire into Workstream G's _process_reading handoff point during the merge — don't try to edit backend/app/routes/device.py or the ingestion handler yourself, since it won't exist yet in your worktree.

Tests: the statistical detector (reuse test_telemetry.py's test shapes for detect_wire_spike as a model — min-samples guard, floor guard, threshold guard, all independently verified), the boolean transition detector, and an integration-style test that constructs a fake SeanceState registered in your registry and confirms a flagged anomaly correctly appends to state.anomalies in the right shape.

Workstream H (frontend) — pairing UI + live dashboard

New frontend/src/pages/DevicesPage.tsx (or extend the existing /inventory area if that reads as more consistent — your call, check current nav structure) — a "pair a new device" flow (name input, calls POST /api/device, shows the raw token exactly once with a clear "copy this now, it cannot be shown again" warning, matching how real API-key UIs handle one-time secrets) and a live dashboard subscribing to /ws/device-feed, rendering each connected device's most recent reading per sensor_type it has seen, updating in place as new readings arrive. Since sensor_type is open-ended, render generically (a card per device, a row per distinct sensor_type seen so far, value + unit) rather than hardcoding presence/temperature/humidity/pressure as fixed fields — reuse TelemetryReadout.tsx's visual conventions as a starting point but make it dynamic. "Hacker witch" styling per the established art direction: this reads naturally as the most "hacker" of all the surfaces built so far (it's literally a live instrument panel) — lean into that side harder than the occult side here specifically, terminal/telemetry aesthetics, but keep the existing dark/violet palette. Tests: pairing flow (token shown once, warning present), dashboard rendering with mock WS frames for multiple devices/sensor types, generic rendering for an unrecognized sensor_type (must not crash — unknown types render with their raw value/unit, no special-casing required).

Workstream I (firmware, ESP-IDF C) — core sensor node

New top-level firmware/esp32p4-sensor-node/ directory (sibling to backend//frontend//deploy/) — a real ESP-IDF project skeleton (CMakeLists.txt, sdkconfig.defaults, main/ component) implementing: WiFi station-mode connection (SSID/password from a main/device_config.h the seeker fills in before flashing — a full provisioning UI is out of scope, document the manual-config step clearly in a README), an HTTP client task that POSTs to /api/device/telemetry with the bearer token from that same config header, and I2C drivers for a BME280 (temperature/humidity/ pressure — a common, well-documented sensor, pick this as the concrete default since no specific part number was given) and a simple presence sensor (an mmWave module like the LD2410 over UART, or a basic PIR over GPIO — your call, document which and why; LD2410 is recommended since it gives distance/motion data beyond a boolean, richer for this app's "believable" ethos). Structure the reporting loop so adding a new sensor type is genuinely easy — a small internal registry/interface (e.g. a sensor_driver_t struct with an init + read function pointer) that the main loop iterates over to build the readings array, not one-off hardcoded POST calls per sensor. Mark the whole directory's top-level README with the honesty-policy note from above. No unit tests in the traditional sense (this is embedded C, not covered by this repo's pytest/vitest suites) — instead, write a firmware/esp32p4-sensor-node/README.md documenting build steps (idf.py build), exactly what's been verified (compiles cleanly against ESP-IDF, structurally sound) versus not (anything requiring physical hardware), and wiring/pinout assumptions for the BME280 (I2C) and presence sensor (UART or GPIO) so a real owner can adapt it to their actual wiring.

Workstream J (firmware, ESP-IDF C) — RTL-SDR experimental module

A clearly-separated experimental, stretch-goal module in the same firmware project (firmware/esp32p4-sensor-node/main/rtlsdr_experimental.c or a dedicated components/ subdir) exploring USB-host communication with an RTL2832U-based dongle via ESP32-P4's USB-OTG host capability. Be honest in the README about the real constraint: this is genuinely at the edge of feasibility for an MCU (wideband IQ sample rates and FFT processing are demanding relative to ESP32-P4's compute, even with its AI accelerator) — the realistic architecture is likely "device pulls raw IQ samples off the dongle via USB host and streams them upstream for the backend to FFT," not on-device spectrum analysis, and even that needs verifying against real USB throughput. Implement what's structurally reasonable (USB host init, basic RTL2832U vendor-command sequence adapted from the existing frontend/src/lib/sdr.ts WebUSB implementation's documented init sequence — read that file for the protocol details already researched for this project) but mark this entire module as the least certain part of the whole build, both in code comments and the final report. If a full implementation isn't reasonably achievable in scope, a well-documented partial attempt plus a clear write-up of exactly where it stops and why is a legitimate, honest outcome — better than a fabricated "it works" claim.

Explicitly out of scope here

Thermal camera support, a full BLE/WiFi-AP provisioning UX (manual config header is the v1 approach), on-device spectrum analysis/FFT (Workstream J targets raw IQ passthrough at most), persisting historical readings for charts/graphs over time (live-only for this spec) — each a reasonable later spec once the core pairing+ingestion+dashboard+firmware skeleton exists and (ideally) someone has tested against real hardware.