Add ESP32-P4 Sensor Node design spec
Third sub-project: physical hardware (ESP32-P4, presence + BME280 env sensors, experimental RTL-SDR USB-host module) pairs with a user's account and streams telemetry that feeds the SAME anomaly/summon pipeline the browser-based modes already use — not a passive dashboard, the actual business model (selling devices that summon spirits). Defines device pairing/auth (reusing the existing session-token hash convention), a generic/extensible sensor-reading shape, and the live-broadcast + anomaly-detection contract split across 5 workstreams (G/K backend, H frontend, I/J firmware).
This commit is contained in:
Indiana
281
docs/superpowers/specs/2026-07-23-esp32-sensor-node-design.md
Normal file
281
docs/superpowers/specs/2026-07-23-esp32-sensor-node-design.md
Normal file
@@ -0,0 +1,281 @@
|
|||||||
|
# ESP32-P4 Sensor Node — pairing, ingestion, live dashboard, firmware
|
||||||
|
|
||||||
|
Third sub-project of the broader arc, and the first hardware/firmware work
|
||||||
|
in this repo. The physical device (an ESP32-P4, referenced in the README's
|
||||||
|
Reliquary section as part of the future "Ultimate Quantum Box" line) joins
|
||||||
|
the seeker's home WiFi and streams sensor readings to their own account on
|
||||||
|
the live site, rendered in real time. Built as parallel workstreams against
|
||||||
|
the contract below.
|
||||||
|
|
||||||
|
## Honesty-policy note (binding on every workstream)
|
||||||
|
|
||||||
|
This app's whole ethos is "real signal processing on real data, and it says
|
||||||
|
so when something is unverified" (see Spirit Radio's `HARDWARE PASS
|
||||||
|
REQUIRED` marking). Nobody here has physical ESP32-P4 hardware to flash and
|
||||||
|
test against. Firmware workstreams must write real, structurally correct
|
||||||
|
ESP-IDF code and mark it clearly as **unverified against real hardware** in
|
||||||
|
both a header comment and the final report — exactly the same honesty
|
||||||
|
convention `frontend/src/lib/sdr.ts` already uses. Do not claim something
|
||||||
|
works on-device when it has only compiled/been reasoned about.
|
||||||
|
|
||||||
|
## Contract (binding for all workstreams)
|
||||||
|
|
||||||
|
**Device identity & pairing.** A `Device` belongs to exactly one `User`.
|
||||||
|
Pairing flow: the seeker generates a device from their account (name +
|
||||||
|
optional sensor-type hint), the backend returns a **raw pairing token once**
|
||||||
|
(never stored or retrievable again — identical convention to
|
||||||
|
`generate_session_token()`/`hash_token()` in
|
||||||
|
`backend/app/models/auth_session.py`: `secrets.token_urlsafe(32)` raw,
|
||||||
|
`hashlib.sha256(raw).hexdigest()` stored). The seeker flashes/configures
|
||||||
|
that raw token (plus their WiFi credentials) into the device firmware. The
|
||||||
|
device authenticates every request with `Authorization: Bearer <raw
|
||||||
|
token>`; the backend looks up by the token's hash, never the raw value.
|
||||||
|
|
||||||
|
**`Device` model** (new table `devices`): `id (uuid)`, `user_id` FK,
|
||||||
|
`name: str`, `token_hash: str` (unique, indexed, same shape as
|
||||||
|
`AuthSession.token_hash`), `created_at`, `last_seen_at: datetime | None`.
|
||||||
|
|
||||||
|
**Ingestion endpoint** — `POST /api/device/telemetry`, authenticated via the
|
||||||
|
device bearer token (NOT the user's session cookie — this is a headless
|
||||||
|
client). Body:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"readings": [
|
||||||
|
{"sensor_type": "presence", "value": 1, "unit": "bool", "metadata": {}},
|
||||||
|
{"sensor_type": "temperature", "value": 21.4, "unit": "c", "metadata": {}},
|
||||||
|
{"sensor_type": "humidity", "value": 47.2, "unit": "pct", "metadata": {}},
|
||||||
|
{"sensor_type": "pressure", "value": 1013.2, "unit": "hpa", "metadata": {}}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`sensor_type` is a free-form string, not an enum — this is the whole point
|
||||||
|
per the project's brief ("add all sorts of sensors... anything you can
|
||||||
|
think of"). The backend does not validate against a fixed sensor list, only
|
||||||
|
against shape (string type, numeric or array value, string unit, dict
|
||||||
|
metadata) and sane bounds (cap `readings` array length, e.g. 64 per
|
||||||
|
request, and reject bodies over a reasonable size, e.g. 16KB, to prevent a
|
||||||
|
misbehaving or malicious device from flooding the endpoint — reuse
|
||||||
|
`app.rate_limit.RateLimiter` for a per-device rate cap too, e.g. 1
|
||||||
|
request/second sustained is generous for sensor telemetry).
|
||||||
|
|
||||||
|
**This is the actual business model, not a side dashboard — say it plainly
|
||||||
|
so nobody undersells the scope: paired hardware must be able to summon
|
||||||
|
spirits, the same as the browser-based modes already do.** A live dashboard
|
||||||
|
of numbers is not the point; feeding the séance's real anomaly-detection
|
||||||
|
pipeline is. Concretely: hardware sensor readings become a **sixth anomaly
|
||||||
|
source** in the existing `/ws/session` protocol (alongside `wire`, `evp`,
|
||||||
|
`radio`, `emf`), not a separate system. Per `(user, device_id,
|
||||||
|
sensor_type)`, maintain a rolling baseline and flag anomalies — reuse
|
||||||
|
`detect_wire_spike`'s statistical shape (min sample count, an absolute
|
||||||
|
floor, 3σ + relative threshold) for continuous sensors (temperature,
|
||||||
|
humidity, pressure, any future numeric sensor), and a simple
|
||||||
|
state-transition rule for discrete/boolean sensors (presence going
|
||||||
|
false→true is the anomaly, not a statistical spike). When a user has an
|
||||||
|
active séance session open with a paired device sending data, detected
|
||||||
|
hardware anomalies get pushed into that session's `state.anomalies` via
|
||||||
|
the same `{"type": "anomaly", "source": ..., "frequency": ...,
|
||||||
|
"magnitude": ...}` shape the frontend already sends for the other four
|
||||||
|
modes — `source` should be the device's `sensor_type` (e.g. `"presence"`,
|
||||||
|
`"temperature"`) so it fingerprints and Codex-displays distinctly, same as
|
||||||
|
`"wire"`/`"evp"`/etc. do today. This means summoning via hardware reuses
|
||||||
|
100% of the existing signature/mint/Codex pipeline — no new mint logic
|
||||||
|
anywhere. Requires locating the user's live `SeanceState` (if any) from the ingestion
|
||||||
|
handler — `backend/app/ws.py` does not currently track active sessions by
|
||||||
|
user anywhere (each `session_socket` connection only holds its own local
|
||||||
|
`state` variable), so this workstream needs to add a minimal in-process
|
||||||
|
registry: a module-level `dict[uuid.UUID, SeanceState]` (or a list, if a
|
||||||
|
user could plausibly have multiple simultaneous séance tabs open — your
|
||||||
|
call, a single most-recent-session mapping is a reasonable v1), populated
|
||||||
|
when `session_socket` establishes a session and removed in its `finally`/
|
||||||
|
disconnect cleanup. Keep it a plain in-process dict — no external state
|
||||||
|
store needed at this scale, same reasoning as the ingestion→dashboard
|
||||||
|
pub/sub above.
|
||||||
|
|
||||||
|
**No permanent storage of every reading.** This mirrors how Wire Ghost
|
||||||
|
telemetry isn't persisted either (only anomaly *events* are). On receipt,
|
||||||
|
the backend updates `Device.last_seen_at` and **broadcasts the readings
|
||||||
|
live** to the owning user if they have an active dashboard WS connection
|
||||||
|
(see below) — it does not write each reading to Postgres. This keeps the
|
||||||
|
database bounded regardless of how chatty a device is.
|
||||||
|
|
||||||
|
**Live dashboard WS** — new endpoint `/ws/device-feed`, authenticated by the
|
||||||
|
normal `qm_session` cookie (this one IS a browser client, unlike the
|
||||||
|
ingestion endpoint). On connect, the server sends the user's device list
|
||||||
|
(`{"type": "devices", "devices": [{"id", "name", "last_seen_at"}, ...]}`).
|
||||||
|
As telemetry arrives from any of that user's paired devices via the
|
||||||
|
ingestion endpoint, relay it: `{"type": "reading", "device_id": str,
|
||||||
|
"sensor_type": str, "value": ..., "unit": str, "metadata": {}, "at":
|
||||||
|
iso8601}`. This is a fan-out problem (ingestion endpoint receives, dashboard
|
||||||
|
WS receives) — implement via an in-process pub/sub keyed by `user_id` (a
|
||||||
|
simple `dict[uuid, list[WebSocket]]` registry is fine at this scale, no
|
||||||
|
external message broker needed; check how `backend/app/ws.py`'s existing
|
||||||
|
`SeanceState`/sender-task pattern works and follow the same
|
||||||
|
single-sender-task-per-socket convention so concurrent producers never
|
||||||
|
interleave, exactly like the existing séance WS already guards against).
|
||||||
|
|
||||||
|
**`POST /api/device` / `GET /api/device`** — REST endpoints (authenticated
|
||||||
|
by session cookie) to create a device (returns the one-time raw token) and
|
||||||
|
list the user's devices (name, id, last_seen_at — never the token or its
|
||||||
|
hash).
|
||||||
|
|
||||||
|
## Workstream G (backend) — pairing, ingestion, live broadcast
|
||||||
|
|
||||||
|
New `backend/app/models/device.py` (the `Device` model per the contract),
|
||||||
|
new `backend/app/routes/device.py` (the REST pairing endpoints: create
|
||||||
|
device / list devices), new ingestion handling and the `/ws/device-feed`
|
||||||
|
WS endpoint (extend `backend/app/ws.py` or add a new router — your call on
|
||||||
|
file organization, follow this codebase's existing convention of one
|
||||||
|
router per concern in `backend/app/routes/`). Idempotent migration: since
|
||||||
|
`devices` is a brand-new table, `Base.metadata.create_all` in `lifespan`
|
||||||
|
handles it automatically — no ALTER TABLE needed. Rate-limit the ingestion
|
||||||
|
endpoint per-device (reuse `RateLimiter`). Reject malformed/oversized
|
||||||
|
payloads with clear 4xx errors, never a 500. **Scope boundary: you own
|
||||||
|
pairing, ingestion auth/validation, and the dashboard pub/sub broadcast.
|
||||||
|
You do NOT own feeding readings into the séance/summon pipeline — that's
|
||||||
|
Workstream K, running in parallel, which depends on your ingestion
|
||||||
|
endpoint's shape but not your code.** To make that possible without a
|
||||||
|
sequencing dependency, structure your ingestion handler so the actual
|
||||||
|
per-reading processing is a clearly separated, small internal function
|
||||||
|
(e.g. `_process_reading(device, reading)`) that Workstream K's merge can
|
||||||
|
extend/call into — document this handoff point in your report. Tests:
|
||||||
|
token generation/hashing round-trip, auth rejection on bad/missing token,
|
||||||
|
payload validation (size caps, shape validation, rejecting garbage
|
||||||
|
sensor_type/value types gracefully), the pub/sub fan-out (a reading posted
|
||||||
|
while a dashboard WS is connected for that user arrives on the socket; a
|
||||||
|
reading posted for a device with no connected owner doesn't error),
|
||||||
|
`last_seen_at` updates.
|
||||||
|
|
||||||
|
## Workstream K (backend) — hardware anomalies feed the summon pipeline
|
||||||
|
|
||||||
|
This is the actual point of the whole project (see the contract's
|
||||||
|
"business model" note above) — paired hardware must be able to summon
|
||||||
|
spirits like the browser-based modes already do. Depends conceptually on
|
||||||
|
Workstream G's ingestion endpoint shape (per the contract, not on
|
||||||
|
Workstream G's actual code — build against the documented `POST
|
||||||
|
/api/device/telemetry` request shape and assume it exists). Implement:
|
||||||
|
|
||||||
|
1. Per-`(user_id, device_id, sensor_type)` rolling-baseline anomaly
|
||||||
|
detection — adapt `detect_wire_spike`'s shape (`backend/app/telemetry.py`:
|
||||||
|
min sample count, an absolute floor, 3σ + relative threshold) for
|
||||||
|
continuous sensors; for discrete/boolean sensors (e.g. `presence`), the
|
||||||
|
anomaly is a false→true state transition, not a statistical spike.
|
||||||
|
New `backend/app/device_anomaly.py` (pure-ish functions, a small
|
||||||
|
per-key rolling-history cache) is a reasonable home for this.
|
||||||
|
2. A minimal in-process active-session registry: `backend/app/ws.py`
|
||||||
|
currently has no way to look up a live `SeanceState` by `user_id` (each
|
||||||
|
`session_socket` connection only holds its own local `state` variable)
|
||||||
|
— add a module-level `dict[uuid.UUID, SeanceState]`, populated when a
|
||||||
|
session starts and cleaned up in `session_socket`'s disconnect/`finally`
|
||||||
|
handling.
|
||||||
|
3. Wire the two together: when an ingested reading is flagged anomalous
|
||||||
|
and the owning user has an active séance session, push
|
||||||
|
`{"type": "anomaly", "source": <sensor_type>, "frequency": ...,
|
||||||
|
"magnitude": ...}` into that session's `state.anomalies` via the same
|
||||||
|
path the existing four modes already use (check `_handle_anomaly` in
|
||||||
|
`backend/app/ws.py`) — `source` should be the sensor_type string itself
|
||||||
|
(e.g. `"presence"`, `"temperature"`) so it fingerprints/displays
|
||||||
|
distinctly in the Codex, same as `"wire"`/`"evp"`/etc. do today. Pick
|
||||||
|
reasonable `frequency`/`magnitude` mappings per sensor_type (e.g. for a
|
||||||
|
numeric sensor, magnitude could be the deviation-from-baseline in the
|
||||||
|
sensor's own units; frequency can be a stable per-sensor-type constant
|
||||||
|
or derived from the reading's value — document your choices).
|
||||||
|
4. Since you can't see Workstream G's actual ingestion code, implement
|
||||||
|
your detection+push logic as a self-contained function
|
||||||
|
(`process_device_reading_for_summon(user_id, device_id, sensor_type,
|
||||||
|
value, ...) -> None`) that the controller will wire into Workstream G's
|
||||||
|
`_process_reading` handoff point during the merge — don't try to edit
|
||||||
|
`backend/app/routes/device.py` or the ingestion handler yourself, since
|
||||||
|
it won't exist yet in your worktree.
|
||||||
|
|
||||||
|
Tests: the statistical detector (reuse `test_telemetry.py`'s test shapes
|
||||||
|
for `detect_wire_spike` as a model — min-samples guard, floor guard,
|
||||||
|
threshold guard, all independently verified), the boolean transition
|
||||||
|
detector, and an integration-style test that constructs a fake
|
||||||
|
`SeanceState` registered in your registry and confirms a flagged anomaly
|
||||||
|
correctly appends to `state.anomalies` in the right shape.
|
||||||
|
|
||||||
|
## Workstream H (frontend) — pairing UI + live dashboard
|
||||||
|
|
||||||
|
New `frontend/src/pages/DevicesPage.tsx` (or extend the existing
|
||||||
|
`/inventory` area if that reads as more consistent — your call, check
|
||||||
|
current nav structure) — a "pair a new device" flow (name input, calls
|
||||||
|
`POST /api/device`, shows the raw token **exactly once** with a clear "copy
|
||||||
|
this now, it cannot be shown again" warning, matching how real API-key UIs
|
||||||
|
handle one-time secrets) and a live dashboard subscribing to
|
||||||
|
`/ws/device-feed`, rendering each connected device's most recent reading
|
||||||
|
per `sensor_type` it has seen, updating in place as new readings arrive.
|
||||||
|
Since `sensor_type` is open-ended, render generically (a card per device,
|
||||||
|
a row per distinct sensor_type seen so far, value + unit) rather than
|
||||||
|
hardcoding presence/temperature/humidity/pressure as fixed fields — reuse
|
||||||
|
`TelemetryReadout.tsx`'s visual conventions as a starting point but make it
|
||||||
|
dynamic. "Hacker witch" styling per the established art direction: this
|
||||||
|
reads naturally as the most "hacker" of all the surfaces built so far
|
||||||
|
(it's literally a live instrument panel) — lean into that side harder than
|
||||||
|
the occult side here specifically, terminal/telemetry aesthetics, but keep
|
||||||
|
the existing dark/violet palette. Tests: pairing flow (token shown once,
|
||||||
|
warning present), dashboard rendering with mock WS frames for multiple
|
||||||
|
devices/sensor types, generic rendering for an unrecognized sensor_type
|
||||||
|
(must not crash — unknown types render with their raw value/unit, no
|
||||||
|
special-casing required).
|
||||||
|
|
||||||
|
## Workstream I (firmware, ESP-IDF C) — core sensor node
|
||||||
|
|
||||||
|
New top-level `firmware/esp32p4-sensor-node/` directory (sibling to
|
||||||
|
`backend/`/`frontend/`/`deploy/`) — a real ESP-IDF project skeleton
|
||||||
|
(`CMakeLists.txt`, `sdkconfig.defaults`, `main/` component) implementing:
|
||||||
|
WiFi station-mode connection (SSID/password from a `main/device_config.h`
|
||||||
|
the seeker fills in before flashing — a full provisioning UI is out of
|
||||||
|
scope, document the manual-config step clearly in a README), an HTTP client
|
||||||
|
task that POSTs to `/api/device/telemetry` with the bearer token from that
|
||||||
|
same config header, and I2C drivers for a **BME280** (temperature/humidity/
|
||||||
|
pressure — a common, well-documented sensor, pick this as the concrete
|
||||||
|
default since no specific part number was given) and a simple presence
|
||||||
|
sensor (an mmWave module like the LD2410 over UART, or a basic PIR over
|
||||||
|
GPIO — your call, document which and why; LD2410 is recommended since it
|
||||||
|
gives distance/motion data beyond a boolean, richer for this app's
|
||||||
|
"believable" ethos). Structure the reporting loop so adding a new sensor
|
||||||
|
type is genuinely easy — a small internal registry/interface (e.g. a
|
||||||
|
`sensor_driver_t` struct with an init + read function pointer) that the
|
||||||
|
main loop iterates over to build the `readings` array, not one-off hardcoded
|
||||||
|
POST calls per sensor. Mark the whole directory's top-level README with the
|
||||||
|
honesty-policy note from above. No unit tests in the traditional sense (this
|
||||||
|
is embedded C, not covered by this repo's pytest/vitest suites) — instead,
|
||||||
|
write a `firmware/esp32p4-sensor-node/README.md` documenting build steps
|
||||||
|
(`idf.py build`), exactly what's been verified (compiles cleanly against
|
||||||
|
ESP-IDF, structurally sound) versus not (anything requiring physical
|
||||||
|
hardware), and wiring/pinout assumptions for the BME280 (I2C) and presence
|
||||||
|
sensor (UART or GPIO) so a real owner can adapt it to their actual wiring.
|
||||||
|
|
||||||
|
## Workstream J (firmware, ESP-IDF C) — RTL-SDR experimental module
|
||||||
|
|
||||||
|
A clearly-separated **experimental, stretch-goal** module in the same
|
||||||
|
firmware project (`firmware/esp32p4-sensor-node/main/rtlsdr_experimental.c`
|
||||||
|
or a dedicated `components/` subdir) exploring USB-host communication with
|
||||||
|
an RTL2832U-based dongle via ESP32-P4's USB-OTG host capability. Be
|
||||||
|
honest in the README about the real constraint: this is genuinely at the
|
||||||
|
edge of feasibility for an MCU (wideband IQ sample rates and FFT processing
|
||||||
|
are demanding relative to ESP32-P4's compute, even with its AI
|
||||||
|
accelerator) — the realistic architecture is likely "device pulls raw IQ
|
||||||
|
samples off the dongle via USB host and streams them upstream for the
|
||||||
|
backend to FFT," not on-device spectrum analysis, and even that needs
|
||||||
|
verifying against real USB throughput. Implement what's structurally
|
||||||
|
reasonable (USB host init, basic RTL2832U vendor-command sequence adapted
|
||||||
|
from the existing `frontend/src/lib/sdr.ts` WebUSB implementation's
|
||||||
|
documented init sequence — read that file for the protocol details already
|
||||||
|
researched for this project) but mark this entire module as the least
|
||||||
|
certain part of the whole build, both in code comments and the final
|
||||||
|
report. If a full implementation isn't reasonably achievable in scope,
|
||||||
|
a well-documented partial attempt plus a clear write-up of exactly where
|
||||||
|
it stops and why is a legitimate, honest outcome — better than a
|
||||||
|
fabricated "it works" claim.
|
||||||
|
|
||||||
|
## Explicitly out of scope here
|
||||||
|
|
||||||
|
Thermal camera support, a full BLE/WiFi-AP provisioning UX (manual config
|
||||||
|
header is the v1 approach), on-device spectrum analysis/FFT (Workstream J
|
||||||
|
targets raw IQ passthrough at most), persisting historical readings for
|
||||||
|
charts/graphs over time (live-only for this spec) — each a reasonable
|
||||||
|
later spec once the core pairing+ingestion+dashboard+firmware skeleton
|
||||||
|
exists and (ideally) someone has tested against real hardware.
|
||||||
Reference in New Issue
Block a user